 | Level: Introductory Philippe KruchtenUniversity of British Columbia
13 Aug 2004 
from The Rational Edge: This article describes techniques for applying agile software development methods to a large project that normally would be considered beyond the scope of agility. The author explains the communications capability that a software architecture team can offer coding teams focused on architectural chunks, even when those teams are disconnected by geography, culture, and specialization.
From The Rational Edge.
If the mountain will not go to Mahomet,
let Mahomet go to the mountain. (Proverb)
I often hear the phrase "scaling up agile processes" to tackle "large" projects,
and I also hear varied opinions about whether it can be done or not. In this
paper, I view this problem slightly upside down. If agile processes work in
some well-identified conditions, is there a way to transform large projects
such that most of the work is done under the optimal conditions of agility,
and therefore using all the good practices associated with those conditions?
What are the consequences, the cost, and the constraints on the organization
and the individuals of doing so?
The "sweet spot" for an agile project
What is the ideal context for an agile project? Under which conditions are
agile methods most likely to be successful?
-
Small group. When no more than ten to fifteen people are on the
project team, it is possible to maintain high-bandwidth, one-to-one communication.
But this starts to become difficult when the team size exceeds twenty people.
With smaller teams, no specialization of roles (analyst, designer, coder,
tester), with handover of artifacts, is necessary.
-
Co-location. Again, this allows high-bandwidth communication and
synchronous communication between people.
-
Customer availability. When the project team has direct access to
the customer, it becomes easier for course corrections to be made periodically,
thus rendering a better product at the end of the project. Better yet, your
team has direct and continuous access to a good customer representative empowered
to make decisions about the needs to be satisfied by the software under development.
-
Business application. Agile development works better for business
software projects in which interactive types of applications are being built.
It works less well for other kinds of projects: for example, an embedded real-time
system.
-
New development. New software projects, often called "green
field" projects, are better suited to agile methods than maintenance
projects.
-
RAD programming environment. Agility requires a rapid application development
environment that allows fast turnaround from construction of code to testing
and collective code ownership; ideally, development teams do daily builds
and monthly delivery.
-
Short lifecycle. Projects should not exceed a few months and ideally
have a rapid internal loop (one day).
-
Common culture. Vocabulary and an implicit process are shared, and
individual developer practices rapidly align with each other.
There is no doubt that in this sweet spot for agile development projects, there
are few elements that require consigning information to some intermediary artifact,
such as change management tools or sophisticated modeling tools. Communication
is direct, including communication with the most interested stakeholder, the
customer. Note that I am not saying that agile development would not work outside
of the sweet spot, but simply that the conditions listed above describe the
perfect case.
The "bitter spot" for an agile project
Conversely, we can describe the "bitter spot" for an agile project.
Here are the conditions that cast agile projects into jeopardy:
-
Large group. When one-to-one communication is possible, but it is
hard to identify the right person to communicate with, agile methods tend
to break down, especially when some team members become swamped by communication
requests. Communication starts to be asynchronous, and teams begin to rely
on documents of various forms, including specific channels and storage mechanisms,
for managing requests and project information.
-
Distributed team(s). This again makes impromptu, rapid communication
more difficult; various electronic media can be used (phone, Internet conferencing
tools, instant messaging systems), but often organizations will opt to produce
documents for asynchronous communication and rely on travel for resolving
critical issues.
-
No empowered customer representative. Under this condition, requirements -- and
the clarification, negotiation, and feedback about requirements -- tend
to evolve over long and inefficient routes, via documents, occasional meetings,
and decisions that are dragged up and down multiple levels of management and
of organization boundaries.
-
Long cycle. When first delivery occurs two years or more after project
inception, agile methods don't hold up. Inevitable personnel turnover leads
to "loss of memory" within the project; it gets difficult to find meaningful
intermediate milestones or short-term objectives for assessing progress or
simply for motivating the team.
-
Inefficient programming environment. Lack of automated tools usually
means long turnaround and no collective code ownership; handover of artifacts
proceeds slowly from designer to coder, from coder to tester, etc.
-
Different development cultures. This is often found when multiple
organizations are involved, which often includes subcontractors and outsource
partners. To remedy this, the process has to be made much more explicit, but
that means the process itself can become the dominant driver, the monster
to satisfy: "We have to take this step because the process demands it."
We should note that in the extreme, large projects have traditionally relied
solely on extensive documentation for all stages of development, with
project managers insisting on written documentation rather than on people to
span the gaps that occur in a distributed environment, multiple organizations,
long project cycles, and the disruptions of personnel turnover.
Large projects
"I never want to work in a large project like that again," someone
told me recently. Well, let's face it: large projects are not going to
go away because of the emergence of agile processes. "Size is some kind
of grit, if not the grit." wrote H. Erdogmus,
1
and this grit is causing much friction, slowing projects to a crawling speed.
I have spent most of my professional life involved in large projects:
telecommunications, command and control, programming tool development, and so
on. For the sake of this discussion, let me characterize a very large project
to avoid questions about marginal conditions, such as "But how about twenty-five
people instead of twenty?" or "What if teams are distributed over
just two sites?"
My large project has these characteristics:
- 200 people
- Distributed across multiple time zones
- Workers report to three different companies
- The lifecycle is 2.5 years to first delivery and 6 months for subsequent
updates
There are some conditions that can be achieved equally in small and large projects:
- Good programming environment
- Skilled people
- Access to the customer
Let's assume all three of these optimal characteristics to make it easy
to compare my large project to a small one.
However, in large projects, certain testing capabilities may not be readily
available to all developers -- e.g., there are only so many telephone switches
or submarine simulators you can make available to any given programmer in the
lab. This deficiency is usually not the case with small projects.
Introducing the large, but agile, project
Now that we've considered the optimal and suboptimal conditions for an
agile project, and we've described the basic scope of a large project,
the question becomes: Can this large project be managed via agile methods? My
answer is, yes. It is possible (though not easy) to organize a large software
development project so that a good proportion of the software practitioners
will work in an environment close to the agile "sweet spot." The
rest of this discussion will describe how software development teams can go
about this.
Setting up an ideal large project
Of the 200 people involved in this project, suppose that I organize ten teams
of fifteen people each, and use the remaining fifty as the superstructure, the
glue that makes it work. In each of the fifteen small teams, members are co-located,
they have their own development playground, and they share a common culture.
The key problems are: Who is the customer? Is this customer empowered? How do
I communicate with other people outside the boundary of my team? A large system
is not just a loosely coupled set of small systems. How do we start on day one?
As the project manager for this monster project, I will need to set up the
project from various perspectives:
-
Organization. I need to define project structure, the various teams,
their roles, responsibilities, and objective.
-
Schedule and rhythm. I need to define the milestones and their associated
objective alongside a timeline.
-
Communication. I need to decide how the various teams will communicate.
Most of the answers lie in the software architecture, which is produced
by the architecture team, and the function of the Elaboration
phase (as it is known in IBM Rational Unified Process®
2
and MBASE
3
process frameworks).
Gradual organization: Inception and Elaboration
The development work can start under one team of seven to twelve developers
(the architects), working in an agile setting that is co-located, has a common
culture, and so on, with direct access to the customer, and using an iterative
work flow. The objective of the architecture team is slightly different from
a classic agile project, however, since the immediate goal is not to deliver
a functioning system, but to identify enough of the significant user requirements
(user stories, use cases, nonfunctional requirements) to set up an architectural
prototype, that will facilitate:
- Organizing use cases (and other aspects of requirements)
- Identifying underlying mechanisms needed to support the implementation of
these use cases, as well as the key nonfunctional requirements (scalability,
performance, availability, etc.)
- Developing an architectural prototype to validate the design choices made,
the technologies selected
- Organizing the code in subsystems
- And, as necessary, setting up guidelines and tools for further development:
e.g., design and code standards, code management, change request and trouble
report, and so on
This initial architecture team proceeds iteratively, and the team gradually
grows in size. When this team reaches fifteen members or more, they split into:
- The architecture team, mostly customer involved, and making key architectural
choices
- The prototyping team, gradually building up and testing the architectural
prototype, refactoring it, and pushing back up any issues detected, thus closing
the loop
The architecture team (seven to twelve people) is the customer for two or three
prototyping teams (consisting of thirty-five or more people).
Staffing the project: Construction and Transition
Towards the end of the elaboration phase, when the structure of the code starts
to take shape and stabilize, additional teams are created, each one "owning"
a chunk of the architecture. The implementation view (according to the "4+1
view model"
4
) serves as the basis for
the partition of the code and the allocation of ownership to a set of teams.
Each chunk of code now belongs to one and only one team.
Ideally, each of the newly created teams is seeded with one or two members
of the original architecture and prototyping teams to bring some of the project
knowledge and emerging culture. Typically, each of these seed members is the
new team's "technical lead." This person will play the role
of the local architect and serve as the communication conduit to the core architecture
team that remains, and, in some cases, with the customer.
There are two types of teams involved in the development of code:
-
Feature teams. Develop code directly related to user functionality.
-
Infrastructure teams. Develop common elements and services used across
the feature teams, or support nonfunctional requirements (e.g., middleware,
services, domain-specific framework, and reusable assets).
For each team, we will strive to create the ideal conditions of agility.
The customer for the feature teams is the real customer(s). But the real customer(s)
cannot spread themselves too thinly across multiple teams and locations. The
customer for the infrastructure team is... all the other teams!, which may have
multiple and conflicting needs.
This is where select members of the core architectural team start playing the
role of customers. They mediate between the real customer and the various teams,
via the tech leads, and identify common needs across the team to be realized
by the infrastructure teams. They become the customer representative for the
infrastructure teams.
Note that this does not prevent direct access by the feature teams to
the real customer, but that contact should occur only when specialty, or subdomain,
issues arise. For example, an air traffic control system has several subdomains:
flight management, radar data, clearance, aeronautical information, conflict
detection and resolution, and so on. As long as the customer involved is competent
and empowered to make decisions about these subdomains, the project is in good
shape.
The architecture and infrastructure teams will continue to fine-tune, refine
(refactor), and expand the architecture of the system throughout the Construction
phase, and ideally this will be conducted with minimal disturbance to the other
teams.
A note on system integration
As we tackle this large project using agile methods, we need to be realistic
in our approach to integrating the complete system. We not only need to assemble
the pieces built by the various teams, but at the same time we need to provide
each team the right "environment" to be able to perform their own builds and
tests. Each team cannot do this on its own, as this may require resources beyond
its capacity. This is the role of a system integration team, which is at the
receiving end of all the software subsystems and commands access to the testing
facility for systems-level tests. This system integration team is derived from
the initial prototyping team. It may also be responsible for certain types of
test such as performance, load, availability, usability, and so on, or when
special equipment is needed.
What about those agile "sweet spot" conditions?
At this point in our project, we have completed the Inception and Elaboration
phases, we have assembled the Feature and Infrastructure teams, and are about
to begin the Construction/Transition phases of the project. Figure 1 shows the
evolution of team structure over the course of Inception, Elaboration, and Construction/Transition
phases. The "blue" teams are the ones that could operate close to the agile
"sweet spot." The "green" teams are the ones that provide the necessary communication
and glue to make it happen.
During Inception and Elaboration a system architecture is prototyped, based
on customer needs, and this serves as the blueprint for a team structure; it
shows where small, cross-functional teams are set up and a how superstructure
is put around them to allow them to efficiently operate.
Figure 1: Evolution of team structure over time
Problems
I said this was not going to be easy. Several dangers lie ahead in this set
up. I'll cover a few of them.
Dependencies between teams
We have created a problem that does not typically occur in our ideal (sweet
spot) agile projects. As I have indicated, any given team, such as a feature
team, is the customer of another team, and is dependent on that team delivering
some functionality. To reduce the impact of this dependency on its ability to
proceed building and testing its code, the team will not only need to build
tests associated with its "user story," but also to build a stub, proxy, or
mock-up that will be passed along in its role as customer to the infrastructure
team.
Stove pipes
Each feature team becomes a single project in itself; large portions of functionality
end up being replicated across the team. This is exacerbated by the geographical
distribution of the team members in this large project and by the different
organizations at work, which do not share a common culture.
The remedy for this? The architecture team must provide some communication
glue, participate regularly in design reviews, and bring in people from other
teams. It can identify potential for common code or functions to be developed
by infrastructure teams, and it can indicate which elements should be reused
from one team to another. It can also suggest rotation of staff when possible
across teams.
Figure 2: Reducing
the "stovepipe" effect. The improvement from side A to side B represents identifying
and developing more common code across various teams and having an organization
to match.
Imbalance
Because the workload is uneven between teams, overloaded teams can potentially
harm the whole project by delays or poor quality. Sometimes the imbalance is
more in the mix of expertise among team members. The solutions to this include:
change of personnel allocation (ratio) of subsystems to team; rotation of staff
to spread knowledge and expertise; and temporary reallocation of staff from
underloaded teams to overloaded teams.
Communication breakdown
When communication between teams becomes inefficient, delays, frustration,
and work blockage can result. The architects must step in and establish and
facilitate direct communications between the involved parties.
Too much staff too early
In general, it is not a good idea to have too many people involved in the
Inception-Elaboration phases. But if your only alternative is idling a portion
of your staff during these phases, it may be possible to start using some of
the staff in small agile projects to develop parts of the system that are obviously
necessary from day one.
Loss of a common goal, lack of focus
Again, although small teams operate with their own objectives, they must never
lose sight that the real success is in delivering the complete system, not just
their chunk. They are not in competition with the other teams. And they must
be open to communication, suggest improvements, and always be ready to rebalance
the project workload.
Dual rhythms (two nested beats)
When the lifecycle is long, measured in years, for example, it may be necessary
to introduce two levels in the project timing, in the project "beat." Figure
3 shows development teams working with a monthly iteration cycle (an internal
release), with a daily "scrum."
5
But this beat
is inside a slower system-level beat, with a system major increment of six months,
and a system-level scrum of a week or two.
Figure 3: Two beats
(scrum of scrum)
Organizing documentation
Some tell me that as soon as I use the unfortunate word -- documentation -- everything
collapses: the project is not agile anymore. But this shouldn't be the case.
Project managers should only introduce permanently maintained artifacts (documents,
models etc.) where they are needed: that is, when they resolve a key problem
and allow the complete project to be globally more efficient.
6
Architecture. The architects will need at some point to document the
architecture, if only to avoid spreading themselves too thin when the overall
team grows and having to explain the structure, the rationales, the metaphors
again and again to newcomers. A well-documented architecture provides a single
visible reference for all team members. Other non-architectural aspects of the
design may never be documented explicitly in a separate document, but only in
comments attached to the code, from which tools like JavaDoc can extract it
when necessary.
Requirements. Requirements are kept very fluid at first, but as the
various feature teams progress in the implementation, use cases are collected,
together with any nonfunctional requirements and organized to facilitate tracing
and testing, and to serve as the starting point for user documentation and training.
Risks, issues, and backlogs
Risk issues and product backlogs should be treated at the level of each team;
only the risk issues and elements of backlog that cannot be rapidly resolved
at the level of a single team should be escalated to the project level. Technical
issues are typically handled by the architecture team, and organization issues
(staffing, resources, etc.) by the project management team. Often issues arise
where the "owning" team is not clear or the problem straddles two teams. Architects
should arbitrate to resolve these problems. A common project-wide tool to manage
local and global issues is a useful part of the development environment (for
example, ClearQuest).
Process
Description of practices, standards, guidelines, and templates is best made
explicit in order to increase uniformity of practice across the whole organization.
A clear description of the process should support the teams in their efforts
to succeed and should not be used in a coercive, administrative fashion.
Concrete examples
Do I have any evidence that large projects can be managed, at least in part,
by agile techniques? Or did I pull this out of my hat? Not quite. Most aspects
of this strategy have been applied to large, iterative development projects
I was personally involved with. Here are a few examples:
1. Ship System 2000 (SS2000), a family of ship control systems developed by
Philips/Bofors/NobelTech/CelsiusTech in Sweden
7
2. The Canadian Automated Air Traffic System (CAATS) developed by Hughes Electronics/Raytheon
in Canada
8
Both these projects had the large-project characteristics I indicated at the
beginning of the paper. Ideas about matching the team structure to architecture -- i.e.,
that the architecture team, integration team, and feature and infrastructure
teams should match the static view of the software architecture, as shown in
Figure 4 -- grew out of the SS2000 project. These ideas were applied and refined
in the CAATS project, with the introduction of use cases. Additionally, we were
fortunately able to convince the customer to give us access to the entire team
of end users on site. The architecture team took over the communication role,
and it was able to resolve issues as they arose across the various teams.
9
Figure 4: On SS2000
and CAATS, the team structure exactly matched the implementation view of the architecture
(i.e., the code view). For details, see P. Kruchten and C. J. Thompson, "An Object-Oriented,
Distributed Architecture for Large Scale Systems," in Proc. of Tri-Ada'94,
Baltimore, November 1994: ACM, 1994.
Another large project was of a different kind:
- The Rational Suite, developed by Rational Software Corporation,
with 700 developers and a delivery beat of six months
The product group that produced the Rational Suite had a dozen teams on eight
sites around the globe, with a central product definition (i.e., the customer
representative) in Lexington, Massachusetts, a distributed architecture team,
and a central integration team. The feature teams were each co-located on the
various sites. Each of these teams was more or less agile depending on its own
origin, history, and culture. Each team has its own internal beat, matched to
align with the beat imposed by the central product teams. My team in Vancouver,
British Columbia, was very agile, accommodating lots of tactical changes, with
a daily build and a more major release every other week or so.
A few years ago, my colleagues at Rational Software reported many other iterative
projects where these ideas have been implemented, with some variants, and with
success.
Summary
I have described the gradual organization of a large project as a team of
teams, where an objective is to set the ideal conditions of agility for a subset
of the teams -- the ones producing the quasi totality of the code -- and
with a few ancillary teams whose main objective is to ensure efficient communications
between teams and to implement the external interface that each team would expect
to have in a smaller project.
For very large projects this is done through:
- Communication inside and outside the team, using the architecture team,
the management team, or the integration team as nodes of communication.
- Scrums and other facilitation and resolution practices at the team level
and inter-team
level.
- Builds and releases at the team level and inter-team level.
- Issues and risks managed at the team level and globally.
- It is preferable that all teams operate according to the same overall rhythm.
Several ingredients are key to doing this successfully:
- The ancillary teams (architecture and integration) must remain at the service
of the other teams and not the reverse. Their role is to ensure that the development
teams are as efficient as possible, and therefore the ancillary teams must
address the needs of the development teams as rapidly as possible through
external communication or resources. For example, through communication with
the customer or with other teams.
- The team involved in code development must keep in mind the ultimate global
objective, the final system, which has priority over its own team objectives,
performance, schedule, and so on. Teams involved in infrastructure must be
very responsive to the needs of their customers, who are the other teams.
- Documentation and an increase in formality (sometimes called the "degree
of ceremony") must be introduced gradually and only when and where it
allows the organization to be more efficient.
- Architecture is the key to setting up the team structure in a way that minimizes
the need for cross-team communications and dependencies.
The project as a whole needs ready access to a competent and empowered customer
representative, who can make decisions about the system. The project needs powerful
tools to manage quick turnaround of code, tests, and test suites, and to support
change management. And, as in any project, skilled and motivated people are
the most important ingredient.
Acknowledgements
This paper benefited greatly from discussions with the participants to the
Canadian Agile Workshop in Banff, February 2003, the USC affiliates research
review in March 2003 and March 2004, and internal reviews by Gary Pollice, Walker
Royce, and Joe Marasco from Rational Software Corporation, now IBM Rational,
as well as Hakan Erdogmus from NRC.
References
H. Erdogmus, "Let's Scale Agile Up." Agile Times, vol. 2, pp. 6-7,
2003.
P. B. Kruchten, The Rational Unified Process: An Introduction, 2nd
ed. Addison-Wesley, 2000.
A. Egyed, B. W. Boehm, D. Port and, M. Abi-Antoun, "Guidelines for the Life
Cycle Objectives (LCO) and the Life Cycle Architecture (LCA) Deliverables for
Model-Based Architecting and Software Engineering (MBASE)." USC, Los Angeles,
USC Technical Report USC-CSE-98-519, 1999.
P. Kruchten, "The 4+1 View Model of Architecture." IEEE Software, vol.
6, pp. 45-50, 1995.
M. Beedle and K. Schwaber, Agile Software Development with SCRUM. Prentice-Hall,
2002.
L. Brownsword and P. Clements, "A Case Study in Successful Product Line Development."
Software Engineering Institute, Technical report CMU/SEI-96-TR-035, 1996.
K. Toth, P. Kruchten, and T. Paine, "Modernizing Air Traffic Control Through
Modern Software Methods." 38th Annual Air Traffic Control Association Conference.
Nashville, Tennessee: ATCA, 1993.
P. Kruchten, "Iterative Software Development for Large Ada Programs." Proc.
Ada-Europe conference, Montreux, Switzerland, vol. LNCS 1088, A. Strohmeier,
Ed.: Springer-Verlag, 1996, pp. 101-110.
P. Kruchten and C. J. Thompson, "An Object-Oriented, Distributed Architecture
for Large Scale Systems." Proc. of Tri-Ada'94, Baltimore, November 1994:
ACM, 1994.
P. Kruchten, "The Software Architect, and the Software Architecture Team."
Software Architecture, P. Donohue, Ed. Kluwer Academic Publishers, 1999,
pp. 565-583.
Notes
1 H. Erdogmus, "Let's Scale Agile Up." Agile Times, vol.
2, pp. 6-7, 2003.
2 P. B. Kruchten, The Rational Unified Process: An Introduction,
3rd ed. Addison-Wesley, 2003.
3 A. Egyed, B. W. Boehm, D. Port and M.Abi-Antoun, "Guidelines for
the Life Cycle Objectives (LCO) and the Life Cycle Architecture (LCA) Deliverables
for Model-Based Architecting and Software Engineering (MBASE)." USC, Los Angeles,
USC Technical Report USC-CSE-98-519, 1999.
4 See P. Kruchten, "The 4+1 View Model of Architecture." IEEE
Software, vol. 6, pp. 45-50, 1995.
5 See M. Beedle and K. Schwaber, Agile Software Development with
SCRUM. Prentice-Hall, 2002.
6 Let's ignore for now the documents that must be produced for contractual
reasons, or for compliance to standards and certification organisms -- FAA,
FDA, etc. -- since these forms of required documents don't necessarily benefit
project organization. In any case, they constitute another topic.
7 See L. Brownsword and P. Clements, "A Case Study in Successful
Product Line Development." Software Engineering Institute, Technical report
CMU/SEI-96-TR-035, 1996.
8 See K. Toth, P. Kruchten, and T. Paine, "Modernizing Air Traffic
Control through Modern Software Methods." 38th Annual Air Traffic Control
Association Conference. Nashville, Tennessee: ATCA, 1993. Also P. Kruchten,
"Iterative Software Development for Large Ada Programs," in Proc. Ada-Europe
conference, Montreux, Switzerland, vol. LNCS 1088, A. Strohmeier, Ed.: Springer-Verlag,
1996, pp. 101-110. Also P. Kruchten and C. J. Thompson, "An Object-Oriented,
Distributed Architecture for Large Scale Systems." Proc. of Tri-Ada'94,
Baltimore, November 1994: ACM, 1994.
9 I describe the role of the architects and the architecture team
in P. Kruchten, "The Software Architect, and the Software Architecture Team,"
in Software Architecture, P. Donohue, Ed. Kluwer Academic Publishers,
1999, pp. 565-583.
Resources - A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
-
Global Rational User Group Community
About the author | | | Philippe Kruchten is former director and general manager of the IBM Rational Software
Process Business Unit, in charge of the Rational Unified Process (RUP). He worked with Rational for thirteen years, in various functions and places: France, Sweden,
the US, and Vancouver, Canada.
Philippe's main interests right now, besides software architecture and design,
are software engineering and the development process. He is campaigning, in
Canada, for the concept of state-licensed professional software engineers.
|
Rate this page
| |
