The agile boundary

The agile boundary demarcates the parts of an organization that are using agile practices from the parts that are not. Understanding the boundary, its placement, and how it can be strategically evolved over time is essential to improved organizational performance via effective agile transformation and hybrid agile practices.

Table of contents

• Introduction
• Defining the agile boundary
  • Hybrid development is ubiquitous
  • There is always an agile boundary
  • Agile beginnings
• Creating a strategy and roadmap for agile boundary expansion
  • Organizations are complex adaptive systems
  • Other options for expanding the agile boundary
• Working across the agile boundary
  • Short release cycles and small batches
  • High-level up-front planning with just-in-time detailed planning
  • High-level up-front requirements with just-in-time detailed requirements
  • Emergent design
  • Continuous, automated testing integrated into development
  • Frequent structured collaboration and empirical, responsive, improvement-oriented environment
• Conclusions

Introduction

Organizations naturally contain boundaries of different types, such as team membership, functional organization, and physical sites. These boundaries can evolve over time in response to events and actions, such as reorganizations, mergers and acquisitions, and changes to business strategy and practices.

We have found that organizations struggle to connect disparate groups and work effectively across the boundaries among them. “Silos” are frequently criticized, but even silos offer benefits as well as drawbacks.

This article addresses one specific organizational boundary that is related to agile adoption. The agile boundary divides the parts of an organization that are using agile approaches from the parts of the organization that are not. The agile boundary is dynamic, and the percentage of the organization inside the agile boundary typically expands over time as more people and groups adapt to working in agile ways. However, this expansion is often haphazard and relatively unplanned, which can lead to uneven progress, frustration, lack of perceived benefits, and efforts that “die on the vine.”

To help organizations avoid such problems, we cover the following topics:

• Defining the agile boundary
• Creating a strategy and roadmap for agile boundary expansion
• Working across the agile boundary

Defining the agile boundary

Hybrid development is ubiquitous

One school of thought over the last two decades is that an organization must be 100% agile to thrive. The alternative to agile is often described as a specious scenario consisting of a 100% sequential, or “waterfall” approach.

In reality, the 100% agile and 100% sequential approaches are both unrealistic. More than a half-century ago, Winston Royce wrote about the nonviability of a purely sequential approach, saying it is “risky and invites failure.”[1] Even in 1970, Royce recognized the need for flow back and forth between adjacent activities, as well as combinations of activities such as system test and requirements that tend to be further apart.

Organizations that attempt to be 100% agile struggle to work coherently at longer time scales, meet regulatory or safety requirements, or manage hardware and software co-development. Business requires both predictability and the ability to respond to change.

A business must be able to match the rate of change in its environment, based on factors such as new technology, changing customer needs, increasing complexity, actions of competitors, etc. These factors vary in intensity across business domains, so the necessary responses by organizations will also vary. If an organization cannot keep pace with change, it inevitably trends towards irrelevance and loses to competitors who are able to keep pace with change.

Agile is not an all-or-nothing proposition. A mix of agile and sequential practices—an approach know as hybrid agile—is often the best solution.

There is always an agile boundary

Once an organization has begun to use agile, there is always an agile boundary. The agile boundary is most often contained within the organization itself. But even when an entire organization has adopted agile, it is very likely that a few of its customers, suppliers, or partners within its ecosystem will not have. So, the questions become where exactly to locate the agile boundary, how to work across it effectively, and how to evolve it over time.

Agile beginnings

At the start of agile adoption, the agile boundary will encompass just a small portion of an organization. Because agile practices have been motivated and driven primarily by software development over the past 20 years, the first foray into agile practice will probably be within the software development portion of the organization. At this early stage, it is uncommon to see other business functions inside the agile boundary.

Agile often enters an organization through a small group of highly motivated champions. In this sense, agile is an innovation within the organization and its champions are from the Innovator and Early Adopter populations described within the discipline of Diffusion of Innovations.[2] These Innovators and Early Adopters are dedicated, daring, influential, and intrinsically motivated to see the new agile practices succeed. The agile boundary appears soon after they begin their work, defined by their interpersonal and organizational networks, the new vocabulary they use, and the novel concepts and practices they espouse. Innovations are, by nature, different from what existed before. It is these differences that characterize the boundary and the populations on either side.

At the beginning, the agile boundary tends to serve a protective function, creating a space in which new practices can be learned and practiced in relative isolation from the rest of the organization. This seems sensible, but it results in immediate friction between the software development team inside the agile boundary and business units and existing business practices that are outside the agile boundary. Numerous mismatches are possible when:

• Senior management perceives a lack of predictability and transparency
• Product management continues to use large batch sizes for new work
• The organization makes premature promises to customers
• The organization cannot shorten its long release cycles even after software development has shortened development cycles

These conflicts are not sustainable, so it is paramount to define the agile boundary in a way that avoids the collapse of the fledgling agile effort.

If agile adoption has already progressed beyond this scope in your organization, extend your search for the current boundary. Use these tactics to find it:

• Follow both formal (org chart) and informal networks, especially those used by influencers within your organization. Ask known agile practitioners for help in locating the other practitioners in their networks.
• Look for social networking channels devoted to agile, brown-bag sessions, intranet websites, book clubs, and communities of practice.
• Publicize your efforts via push (email, newsletter, etc.) and pull (website) sources. Invite all agile practitioners into a new forum or community.
• Analyze existing process descriptions, flow diagrams, and value stream maps for functions and groups inside and outside the boundary. These functions and groups may be suppliers, sources, recipients, or consumers.

These techniques and others will help you find the portions of the organization within or about to be within the agile boundary.

Knowing the extent of the existing agile boundary is key for creating a strategy for where and when to expand the boundary. Organic adoption is fine at the beginning, but delaying the establishment of consistent practices for agile adoption is counterproductive.

Creating a strategy and roadmap for agile boundary expansion

After determining the location of the current agile boundary, expand it strategically. Doing so has numerous benefits compared to a hands-off approach. However, stepwise, linear planning is not suitable to agile boundary expansion.

Organizations are complex adaptive systems

Organizations and their people form a complex adaptive system that is not deterministic and fully analyzable no matter how much time and effort you devote to the task. Use an approach that seeks to understand what is happening now and what is feasible in the immediate future.

A three-year Gantt chart describing each step for becoming Agile will fail. This does not mean that all planning is useless—it means that different planning is required. Use agile to create agile. Agile boundary expansion is a combination of planned and opportunistic work. The key is determining the current state of the organization and feasible next steps.

Because agile usually first appears within the software development organization, the first step is often broader agile adoption in that organization. It then becomes necessary to move the agile boundary beyond software development.

Software test and QA are an early expansion area if those functions were not part of the initial agile adoption. Cross-functionality and team autonomy are greatly enhanced by test and QA helping to create releasable increments according to a robust Definition of Done (DoD).

Next, look upstream and downstream from software development. On the upstream side, include product management so that the inputs to the software development teams are sized and prioritized in a way to enable agile development. On the downstream side, focus on DevOps to improve integration and release capabilities and accelerate business value delivery. This is a common pattern. What is right for your organization might be different.

Other options for expanding the agile boundary

Looking at the activities that are upstream and downstream from the existing agile boundary is a feasible first step, but analyzing your organization’s complete value stream is best for strategic boundary expansion. The most effective place to expand the agile boundary might be distant from software development. Contiguous expansion outward from software development is not always the best approach.

Another approach is to expand according to product or product line rather than by organizational function: transforming an organization “vertically” rather than “horizontally.” Horizontal transformation—function by function across the entire organization—can create islands of new practice within a sea of legacy practice, which can wash away the new islands before change can take hold.

Organizations can couple this vertical form of agile boundary expansion with environment and architecture migration, especially cloud migration. All the necessary functions shift to an agile approach at once but in numbers limited by the product(s) chosen for migration. The application environment and architecture shift at the same time.

Capture your strategy as a high-level roadmap to guide your expansion efforts, accepting that changes will be necessary and unforeseen opportunities will arise. Make the roadmap more detailed and specific in the near term (one quarter) and decreasingly detailed out to a year. Opportunities beyond a year are rarely stable enough, even as aspirational goals. The organization is complex, and every action you take will have both unintended and intended side effects. As one charismatic senior executive has put it, “The plan is the plan—until we change the plan.”

Working across the agile boundary

As described earlier, the location of the agile boundary will change with time but there will always be a boundary somewhere. You must be able to work across the agile boundary, wherever it happens to be.

Boundaries can contain, exclude, and filter. To work across the agile boundary, determine what is permitted to flow across it (in one or both directions), how often it flows, and by what channel. Also determine what is prevented from crossing the boundary.

For example, software architecture is often housed in a single organization that serves all the software development teams. The availability and turnaround time needed to support Scrum or another agile approach might be difficult for the architecture team to support because they are working according to an older model with longer lead times and a sequential approach to architecture definition.

The boundary between architecture and software development requires careful definition to satisfy the needs of both groups. The situation requires a protocol for how to request service, a way to supply the necessary background information and data for the request, a protocol for how the architecture team prioritizes and resources requests, etc. Must the same architect service a particular project or team across time? On which side of the boundary does the architecture specification live? What working agreements are necessary, and what connections across the boundary do they imply?

If these elements are difficult to get right, that might indicate that the portion of the boundary in question is a candidate for being changed sooner rather than later. In this example, pulling some of the architecture team inside the agile boundary might make the most sense.

Agile software development has several characteristics that help determine the nature of the agile boundary and how to work across it effectively. These characteristics define what is different about an agile approach in comparison to a more (but not completely) sequential approach.[3]

These characteristics have many implications for what information must flow across the agile boundary, how often, and via what channels.

Short release cycles and small batches

Short release cycles and small batch sizes inside the agile boundary mean that the inflow and outflows across the boundary will occur with greater frequency but with smaller scope than before. Existing working agreements should be adapted to the increased frequency. Also, check for a mismatch between the small, more frequent outflows and standing meetings and processes for reviewing progress. Downstream processes and practices for release can be substantially affected by more frequent code deliveries, whether or not those deliveries translate into releases.

High-level up-front planning with just-in-time detailed planning

Planning with reduced up-front detail challenges existing norms for investment decisions and milestone reviews. A more incremental decision model creates new opportunities for flexibility in product management but might be seen as risky in some organizations. To counter these problems, relax or adapt milestone exit criteria and move to a more incremental funding model—there’s no need to fully fund a program on fractional data. You can also create a parallel life cycle and funding mechanism for “agile projects” vs. “traditional projects,” but even better is tailoring the funding mechanism to the nature of the system being built.

Even though the “iron triangle” is unrealistic in any nontrivial setting, fixed-bid contracts that dictate scope, date, and investment constrain organizations. Iron triangle thinking can create serious issues even when an entire organization has adopted agile internally. The best course of action is to change contract deliverables from outputs to outcomes and to switch from large monolithic contracts to smaller incremental contracts. Bringing customers inside the agile boundary might prove challenging but can afford the greatest benefits of all.

High-level up-front requirements with just-in-time detailed requirements

Many teams create too much requirements detail initially. Much of that detail is concentrated in areas where the knowledge is best—such detail is readily available and easy to write. However, capturing this known detail does little, if anything, to reduce project risk. If an organization correctly identifies which requirements can and should be written up front vs. left for later, it will improve time to market and reduce the waste of writing a lot of detail in the hope it will be useful someday.

On nearly every project, some requirements must be specified early and others cannot be known and specified early without incurring too much risk of later change. Agile reduces speculation, leaving specification of anything that can be postponed until near the time it will be used in construction.

Adapting life cycle milestone exit criteria is a part of addressing requirements work across the agile boundary, but communication links across the boundary is also important. When product management is outside the agile boundary, communication between the development team and product management must be adapted to increased frequency and smaller batch size. Without this capability, requirements validation will be slow to nonexistent— the project will suffer from delays, wrong requirements, missed requirements, incorrect assumptions, and substantial rework.

Emergent design

Emphasis on emergent design rather than detailed, up-front design creates issues across the agile boundary even though the design function is normally contained within the agile development teams.

In organizations where a centralized architecture function exists outside the agile boundary, architects operate in an episodic, service-provider fashion with teams, with a single architect assigned to service a given team. Agile teams will require numerous small interactions rather than a single large interaction at the beginning of the project. This creates resource loading and scheduling challenges for the architecture organization. Architecture organizations will benefit from adopting Kanban to clarify work queues, limit work in progress (WIP), and help prioritize their time. Kanban is also an effective way to communicate status and WIP across the agile boundary.

A similar issue occurs when a User Experience (UX) team is centralized and located outside the agile boundary. Practices in Lean UX are becoming more common, but many UX groups operate similar to centralized architecture groups, helping teams in a “service provider” fashion. Again, Kanban adoption makes sense, but in both cases, the question becomes why those functions should be centralized versus made part of the agile development teams to increase development team autonomy.

A third design challenge occurs when the product management organization is outside the agile boundary. If agile development teams must work with product management to validate design decisions internally and with external stakeholders, once again scheduling and resourcing will be issues. Long feedback loops will delay decision making during development and lead to rework. Organizations can establish product councils, management review committees (MRCs), Customer Circles, and similar structures to expedite these reviews and prevent long feedback cycles. The groups can convene on a regular cadence or on demand.

Continuous, automated testing integrated into development

Testing in a continuous, automated way that is integrated with the development environment is challenging if the test function is partially or completely located outside the agile boundary. Communication of what is coming when prevents delays and ensures testing readiness. Agile development teams that reprioritize on a Sprint-by-Sprint basis using Scrum, or even more frequently when using Kanban, can create thrash for the test organization as it plans, resources, and instruments its tests and testing frameworks.

The same sort of transactional service model that challenges architecture and UX is even more challenging for test. For this reason, test is among the most important functions to pull in across the agile boundary. If embedding test within development is not possible, organizations should increase communications across the boundary. Test personnel will listen in on planning activities and daily stand-ups done by the development teams and review Kanban boards to see up-to-date information on the state of the work in progress.

Generally, a test organization outside the agile boundary must step up to work on smaller batches with greater frequency—this is not a simple task because of the lead time that is endemic to test development. Automation helps, especially for regression testing. Shifting some testing to developers can also be effective, using, for example, Acceptance Test-Driven Development (ATDD). Also, organizations realize benefits by improving their use of DevOps and the tool chain to reduce delays in the interval from pull request to production.

Frequent structured collaboration and empirical, responsive, improvement-oriented environment

The factors in these final two areas depend on cultural foundations—the organization’s beliefs, values, and history influence whether issues across the agile boundary exist in these areas and how large those issues are.

In an organization that values individual expertise and achievement, the agile boundary provides one more reason for an individual to work in isolation. If the organization reinforces this behavior pattern in annual reviews by rewarding individual contributors over teams, no shift towards increased collaboration and responsiveness will occur.

Organizations that have learned to bury failure or simply blame other teams cannot expect success in transitioning to an improvement-oriented environment without addressing the root cultural issues. The agile boundary provides a convenient enabler for such blaming and ignoring, because people on each side of the boundary think and act differently from one another based on different perceptions of what is correct and appropriate. Any boundary runs the risk of dividing a population into “them” and “us”—this is not peculiar to the agile boundary. But this contrast illustrates what might be the most difficult, and most important, aspect of agile transformation and agile boundary expansion.

Conclusions

• The agile boundary appears as soon as an organization begins using agile practices, whether that is driven by grass-roots adoption or by a planned agile transformation.
• Once it exists, there is always an agile boundary, even if the entire organization adopts agile.
• Organizations benefit greatly from taking a strategic approach to expanding the area inside the agile boundary over time.
• The nature of the differences between traditional sequential development and agile development characterizes the populations inside and outside the agile boundary.
• To increase overall effectiveness, organizations can use a combination of bringing more people and functions inside the agile boundary and enabling the flow of information and work products across the boundary.
• The challenges presented by the agile boundary relate to agile’s smaller batch sizes, more frequent release cycles, and a just-in-time emphasis in planning, requirements, and design. Deeper cultural elements also play an important role.

References

[1] Winston Royce, Managing the Development of Large Software Systems, IEEE Wescon, August 1970.

[2] Everett Rogers, Diffusion of Innovations, Free Press, 2003.

[3] Steve McConnell, More Effective Agile: A Roadmap for Software Leaders, Construx Press, 2019. See especially Chapter 2, What’s Really Different About Agile? The table shown here is from that chapter.