Level: Introductory Gary Pollice, Professor of Practice, Worcester Polytechnic Institute
15 Aug 2006 from The Rational Edge: The value of following an agreed-upon process in software development is well-established. But what sort of process? How complex? For what size teams? Professor Gary Pollice answers these questions by considering the variety of process needs within an enterprise.
 I started to write this month's Theory and Practice column about using and creating mock objects in testing. That was before I attended Agile 2006, the Agile Alliance's annual conference on Agile methods and techniques. Oh, I can hear you say "Ahhg! Not another article on agility." I assure you that is not my intention. The conference presentations and discussions I had with several people got me thinking about process in general and the role of process in software development.
Of course, what the people at this conference and elsewhere are really interested in is finding a silver bullet for getting their software done. It dawned on me that the conference discussions were centered on a particular level of success within an organization, a level that had to do with a company's size, the size of its software development teams, and the scope of the process used to ensure success. I started trying to sort all this out, and I think I've arrived at some helpful ways to think about the role of process in software development.
For starters, as smaller organizations grow to enterprise organizations, things change and your processes must change as well. The Agile community is quick to point out that one of agility's fundamental principles is reflection upon what you do, with the aim of improving the way you work. But this principle is no different in RUP or any other modern methodology. So the question arises: "Does process matter?"
That question "Does process matter?" may seem like a rhetorical one, especially coming from the former "RUP Curmudgeon," my nickname inside the Rational organization before I left to pursue an academic career. Actually, the question is only somewhat rhetorical. Of course process matters! In some cases, it matters too much -- as when the chosen process overpowers the project team and makes the team unable to satisfy the customer. If we equate satisfying the customer with quality, then process can potentially have a negative effect on quality.
Perhaps a better question to ask is "Does the particular process matter?" In a previous column, I discussed the importance of matching the process with the project,1 and the need to customize and tailor the process to fit a project is now an accepted software development best practice. Moreover, there isn't just one process that can effectively support a project team or organization. There may be many. I can imagine cases where almost any process -- well-defined or not -- will help a team succeed.
I think there are two main factors that influence the number of processes that might apply to a particular situation: 1) the size of the team and 2) the scope of the process in the larger community context, if one exists. I will examine these factors and then look at the need for process interfaces to address the multiple processes that can be found in larger organizations.
A definition
Before we launch into the details, let's make sure we have a common view of the concept of "process." I will turn to my favorite source, the dictionary, and use one of the definitions for process there: "A series of actions, changes, or functions bringing about a result."2 This definition is simple and describes, in my opinion, the essence of a process. The process is the series of steps we take to get our job done. It is not what we write down about how we plan to do our job, nor is it a set of rules and such that we must follow blindly. It is what we do, whether it is written down, defined, or not.
Team size and communication
An effective process is one that supports effective communication, and effective communication is directly related to the size of the project team. This should not be news to anyone who has worked on different-sized project teams.
For those readers who would like more rigorous evidence, just consider the number of communication channels that are possible, based upon the size of the team. A two-person project, persons A and B, has only one communication link.3 If we add person C to the project, we have three communication channels, A-B, B-C, C-A. When person D joins the team we are up to six, and when person E joins, we reach 10. A ten person team will have 45 communication links. With so many possibilities, synchronization and transmittal of vital information is difficult and error-prone.
The formula for determining the number of communication channels is the combinatorial formula for the number of pairs you can select from n items, without regard to order. The formula4 is as follows:
When we are looking for pairs, this can be reduced to
The important characteristic of this is shown in Figure 1, where you can see the growth is not linear. As your team grows, you have to find ways to formalize communication channels by deciding what communications to capture and how to capture them.
Figure 1: The effect of team size on communication: As teams grow larger, the number of communication links grows in a non-linear fashion.
Team size affects more than just the number of communication channels. It affects all parts of the team's process. Two people working on a small project together will adopt some sort of mechanism for assigning work, managing their source code, documenting their work, and so on. This pair really doesn't need a lot of guidance telling them how to do their jobs. They most likely are talking to each other, either in person or electronically, several times a day. Yes, this is communication, but there is very little opportunity for noise from an outside source to interfere with the communication. Since there are only two people, they don't have to commit their communications to a more permanent form, like Unified Modeling Language (UML) diagrams kept in a tool or detailed use-case descriptions. Almost any "light" process will work for these two.5
Communications problems appear when the team size increases. One such problem is work assignment. Two or three people can easily keep track of who is doing what and for what purpose, and they know who to talk to about the different design issues that might arise. Things are quite different on a team of twenty. Somewhere, there needs to be a master to-do list that people can use to direct their efforts. Agile methods like Scrum and eXtreme Programming (XP) have such lists. Scrum calls it the backlog, while XP uses story cards as the main task-capturing mechanism.
If we consider process as a standardizing mechanism, then the need for process goes up as the team size increases. There are just too many things that can go wrong, and lessen the team's efficiency, if each team member decides to perform their tasks differently, especially when the task has an effect upon other team members.
A simple example from a past project I was on illustrates this nicely:
I was working on a product team of about fifteen developers creating a new product. The team was divided into smaller teams, each responsible for specific components. One of the components was a core piece that all others, including mine, depended on. I usually work by making small changes to the code and then testing the new code before I move on.6 On this particular project, a pattern kept repeating. I would make a change, and sometimes my build would fail with compilation errors or linking errors, but not on my code. I would trace the bug to a different module, and when I looked at the check-in log, I could see that Mary7 made a change between my last build and the current, broken one. Without fail, I would ask Mary if she had made any changes, and she would admit to checking in the faulty code. I would then ask if she tested the code. Her answer was always something like, "No. It was a simple change." Then I'd ask if she even compiled the code. Again I was told, "It was only a one-line change. There was no need to do a build."
Clearly, Mary's personal process had an impact on others. My process was quite different from hers. Establishing a simple rule for the team, like no code will be checked in unless it is compiled and tested, would save the team time and effort. You could imagine what the project would look like if we decided that fifteen personal processes were acceptable. Unfortunately, the team decided that we needed to define everything the developers did. This was the other end of the spectrum -- a process that consumed most of our time with unproductive, useless activities.
The lesson from this illustration may seem simple, but a simple lesson can help you make a forceful case for why a process must be agreed to as team size increases: Choose a process that is appropriate for the size of the team. I think Alistair Cockburn describes this nicely in his book, Agile Software Development.8 He describes project size -- both the size of the product produced and the size of the development team -- as one of the parameters that must factor into any process configuration. In my view, team size along with communication issues they bring are directly proportional to product size, so I focus on team size as the leading indicator.
Process scope and purpose
The second main factor affecting the number of process choices is the scope of the process. I'm not talking about whether the process covers development activities from the inception of the project to its retirement. What I mean by scope is the influence the process has on the total organization or community.
There are some scopes that seem to be natural for grouping purposes.
The first, and smallest, is the personal process scope. Each of us has a personal process that we have created over our careers. You can probably think of great developers who prefer to work in their Emacs editor and do everything from there, or those who thrive with lots of tools integrated in a more visual development environment. Some programmers write pseudo-code before they begin writing their program. Others jump right in. The point is that we all have a personal process that works best for us. There are at least as many personal processes as there are developers.
The second process scope is the immediate team process. The immediate team is the smallest unit, greater than a single person, that collaborates to produce a product or discrete part of a product. At this scope, the process not only affects all of the individuals, it process dominates the personal processes of the individual developers. Each individual might have to adjust his or her personal process to accommodate the team's process. A good team process is one that requires minimal changes in the personal processes.
On the third rung of our process scope ladder is the project scope. This is where multiple teams must contribute to a single project, product, or product line. Such processes are common in systems engineering projects, where the software and hardware must be developed in parallel and it must be synchronized. Many times these projects do not have the amount of rapid change that "pure" software projects have -- e.g., a billing system that is not developed in parallel with specialized hardware or that relies upon a specific hardware implementation.
The fourth and largest is the enterprise process scope. A large organization will have many projects, in various stages, at the same time. The organization must be diligent about managing costs, optimizing resources, and ensuring that the projects contribute to the corporate goals. One important business activity in large organizations is portfolio management, which enables enterprises to manage a portfolio of projects as valuable assets. The discipline of project portfolio management arose from the need to effectively manage these assets just like other corporate assets. Portfolio management requires a practical process implementation across all of the projects.
There may be other process scopes that you can think of. These four are sufficient for our purposes, and I will use this hierarchy in the rest of the article.
Now that I've described the two main factors that affect the use of process in an organization -- size of the team and scope of the process -- let's consider the implications for team dynamics up and down the organization.
Process purpose changes with scope
The following observation may be obvious to you, but it wasn't that obvious to me for a long while in my career. As the process scope changes, the reason for having the process changes. I've come to think of this as the difference between tactics and strategy.
An individual process is one that the developer has evolved over time to fit his or her working style. The purpose of the process is to help the developer produce quality code. When I make sure I test all of my code, I do it because it makes me a better developer -- that is, it makes the finished product better. When I learn design patterns and apply them as part of my design process, it makes my code more robust. My personal process is a very proactive process.
A team's process certainly must be designed to help the team provide a quality product. But other things are also expected from the team. Usually, there are status reports (written or otherwise), performance evaluations of the developers, communication mechanisms, and so on.
The project process is more strategic. It will involve more information about the market, corporate strategies, and how the project contributes to them. This type of process has little effect upon the individual processes, and if it does, that effect is usually negative. Let me explain this a little further. At the project level, the process will focus on issues such as determining whether the completed product is ready to ship, whether the documentation for the product is adequate for the users, and so on. It should be getting this information from the different project teams. While the project process may impose some content and formatting constraints on the information, each team should have the ability to determine how it produces the information.
Let's say, however, that the project process decrees that each individual developer provide detailed descriptions of every function they write, just in case the information might be useful to someone. This will certainly have negative effects upon developer productivity and morale. It will also require what we seek to avoid -- mountains of useless documentation. If there is, in fact, a need for such documentation, then the project process should simply tell the teams that the documentation is required,9 then let the team decide how the documentation will be produced.
Finally, the enterprise process can't be concerned with individual details. It is completely strategic. The enterprise process is in place to ensure that the company, as a whole, thrives. There is much more effort placed on planning at the enterprise level than at the individual level, and this is to be expected.
When we look at the hierarchy, and the differences in process purpose, we begin to understand how to select and match the processes.
First, it seems to me that processes have an architecture, like software. Layering different types of processes might be the most effective way to organize them. Using our hierarchy, we can easily imagine a four-level process configuration for a large organization, as shown in Figure 2. Each layer contributes to the process above it. The higher layer defines the (minimal) amount of information a process needs from lower processes in order to fulfill its goal. Notice that each level only communicates with the level above or below. This also implies that the direct effect of each process is aimed directly at the processes at the level immediately below. For example, the enterprise needs information to plan the yearly budgets. It expects each project to provide the budget information. The project needs to get relevant information from the teams and consolidate it into the form the enterprise requires.10 Teams may communicate the need to individuals to help develop their data -- for example, asking about new tools developers want to use, and so on.
Figure 2: Processes are layered.
If the organization has processes layered as shown in Figure 2, it will realize some of the same benefits that we see in layered software architectures. We have the ability to "replace" processes with new ones, without having to revise the whole process system.
Scope and size are related, but they affect the process selection in different ways. Selecting a process to match scope within the organization is usually done by someone, or some group, at a higher level of management in the organization. Selecting a process to match team size is usually done by someone, or some group, in the team who will use it. The two lower layers in Figure 2 represent processes selected by team members for the tactical values they provide -- improved quality, faster development, and so on. The upper two layers represent processes selected by managers and imposed upon the lower levels for strategy and control purposes.
Process layers require appropriate interfaces
If we consider process according to the previous discussion, it leads us to the concept of a process interface. If you look closely at Figure 2, you will see that the higher-level processes enforce constraints on the lower-level processes. This is somewhat analogous to classes in our software designs that call methods in other classes. Because we want to minimize the dependencies, we create an interface between the two. If we create such interfaces between the process layers, we now allow the lower-level processes to evolve and change, as long as they provide the appropriate interface to the upper levels. This may require an adapter to massage or reformat information for the process with the larger scope.
Let's look at a simple, but realistic example. Say that the team process requires that no code can be checked in unless there are unit tests accompanying them. This illustrates an interface between the team (the level at which code is checked in) and the individual (the level at which testing must be performed) processes. Does it matter whether the individual writes tests before implementing the code or after the implementation code is written? Not according to the contract of the process interface. That only requires that code be accompanied by tests. Developers have the freedom to choose what works best for them.
Here's a higher-level example. Imagine that a project process tracks progress based on RUP-oriented use cases, but a given team decides that they will use XP-oriented user stories to measure their progress. When it comes time to report, there needs to be an adapter activity that will match the user stories11 to their use cases and report based upon the use cases. We can certainly argue that this is unnecessary overhead, but if the development team is more efficient when they use user stories, the overhead may be worth it, rather than forcing them to only work with use cases. You can't change the higher process, because it is the client, or customer, for the lower process, and it decides what the reporting requirements are.
Well-designed process interfaces must dictate only a minimal amount of requirements for the lower-level processes. If the requirements are too draconian, the lower processes become bogged down with process rather than performing their tasks efficiently.
Micro-processes must be supportive
In the discussion so far, I've talked about the processes at different levels, as if they were well-defined entities that completely met the needs of entities at the targeted level -- for example, as if the team process is a single process that encompasses all of the work performed by the team. This is an illusion. There is no single process, but many smaller processes that mesh together to form the seemingly unified process at that level. Some of these micro-processes are independent of each other, while many of them interact. When they interact, they must interact smoothly, so the choice of a micro-process depends upon the other micro-processes at that level.
A few examples of micro-processes will illustrate what I mean. Some micro-processes, at different levels, might be:
- Source code management (SCM) practices for the team. For example, no file will be checked in without a second person reviewing and agreeing that the code (and tests) are correct.12
- Defect-tracking processes that set standards, such as status categories, disposition, and tracking are usually project or enterprise processes. Standards for wording the defect responses, and so on, might be delegated to the project or team processes.
- Coding standards and naming conventions are usually determined at the project or enterprise level.
- Work assignment practices will vary from team to team. Each team, depending upon their agreed-upon practices may have different ways of determining who works on what.
All micro-processes must support each other, and they must support the processes at upper levels according to the process interfaces. This relationship between process levels can be illustrated via two levels of processes in an organization, as shown in Figure 3.
Figure 3: Relationship of two process levels, Project and Team, and the micro-processes that must work together within each level
There are two interfaces between micro-processes in Figure 3: the SCM processes and the Status Reporting processes. As long as all of the gears mesh in the team processes and they find a way to make the two inter-level processes mesh, the process works.
This, in my opinion, shows the real art of process engineering -- the ability to give entities working at lower process levels as much freedom to do their job by limiting the amount of required synchronization between levels.
What does all this add up to?
What does the above discussion have to do with Agile 2006 and my opening comments? Now that I have described a way to think about team size and process levels, I can assert that the Agile community at the conference is mainly looking at the team-level process, even though many of the thought leaders claim otherwise. As I noted at the beginning, smaller organizations growing into enterprise organizations must change their processes, with the realization that Agile methods may not suffice at the enterprise or project levels.
What disturbs me, and one of the reasons I don't "teach Agile," is that people tend to develop a narrow view of the world when you give them just one approach. They tend to see only as far as their current environment allows. At the Agile conference, I kept hearing that things would be much better if the managers would just butt out and let developers do their jobs. The impression was that the higher you go up the management chain, the more fouled-up things become and the less competent people are for making the right decisions. The reality is that the people at the upper levels of management are often the ones who either founded the company or came up through the ranks to assume a more strategic role within the company. Their scope is different, and their values are often different from those of the development teams. Developers need to realize that if the company is profitable, these people at the top must be doing something right. There are too many profitable companies for things to be working in spite of poor leadership!
I believe that it's crucial for developers to understand more than just their team processes and start learning about business process in general. Two positive things will result from this. First, they will learn how to better interface their process with the higher-level processes. Second, they will get an appreciation for the more strategic efforts of the company and how their development teams can, and do, contribute to higher goals. This knowledge will make them more valuable to their company. For new hires working within lower-level processes, this understanding will help them quickly discover whether they are compatible with the company. There is no shame in admitting that the way you prefer, or are willing, to work is so incompatible with your personal process that neither you nor the company will benefit with you on board. I would rather know this as soon as possible, rather than having it pointed out to me at my performance review, or worse, at my exit interview.
Process is important, but there is not just one process for all. An enterprise has many processes, at several levels. Understanding how these processes need to work together is a critical awareness, one that is too often ignored. When you understand real process needs, you understand that the specific process you follow is not as important as whether it plays well with the others.
Notes
1 Gary Pollice, "Matching project and process," The Rational Edge, May 2004: http://www.ibm.com/developerworks/rational/library/4720.html
2 Taken from http://www.yourdictionary.com
3 When we talk about communication channels or links, we are only talking about the direct development team members and not the communication links outside of the team. This is just a simplification for the discussion.
4 If you are not familiar with the notation, n! is the product of all integers from 1 to n. So 4! = 4 x 3 x 2 x 1 = 24.
5 I know that I'm getting into a semantically murky area by using the word "light" here. I trust the reader will let me use it in this context without spending time trying to describe it.
6 Today, I usually write the test first, but that's not important to the story.
7 Not Frank's real name.
8 Alistair Cockburn, Agile Software Development. Addison-Wesley 2001, ISBN 0201699699.
9 This might be the case when you are developing software under a contract that will deliver the source code and documentation to the customer. Even this level of documentation might still be considered extreme.
10 Developers will find this analogous to the Adapter design pattern.
11 RUP is the IBM Rational Unified Process, and XP is eXtreme Programming -- both are software development processes, although RUP is capable of scaling up to very large projects.
11 The individual can satisfy this requirement by pair programming or after-the-fact review.
About the author  | 
| | Gary Pollice is a professor of practice at Worcester Polytechnic Institute, in Worcester, MA. He teaches software engineering, design, testing, and other computer science courses, and also directs student projects. Before entering the academic world, he spent more than thirty-five years developing various kinds of software, from business applications to compilers and tools. His last industry job was with IBM Rational software, where he was known as "the RUP Curmudgeon" and was also a member of the original Rational Suite team. He is the primary author of Software Development for Small Teams: A RUP-Centric Approach, published by Addison-Wesley in 2004. He holds a BA in mathematics and an MS in computer science. |
Rate this page
| 
