Preface We are satisfied by doing real work. Software is like a plant that grows: You can’t predict its exact shape, or how big it will grow; you can control its growth only to a limited degree. There are no rules for this kind of thing—it’s never been done before. - Charlie Anderson, Architect, Borland Quattro Pro for Win- dows. You will find no books on the bookshelf here that tell you how to start up a new discipline. Software has been seeking its own way as a relatively young discipline for the past 40 years. Every new discipline struggles to find practices suitable to its survival and growth. Some- times this struggle is incremental. Sometimes disciplines undergo more substantial shifts in process, structure and values that break more with the past to explore new ground. What Charlie Anderson said above about the Borland Quattro Pro for Windows effort in partic- ular applies to the rhythms of software development in general. Get ready for change, for it will come tomorrow. The most exciting advances in science go hand in hand with radical social change. The move from classic physics to quantum physics pre- 1 2 Chapter cipitated from a crisis in physics. We talk about the software crisis, yet no individual crisis in software — let alone the Software Crisis, what- ever that might be — has precipitated the same kinds of change that we associate with great advances in science. Software development has perhaps yet to face its first true crisis that leads to the first true industry-wide systemic change. But that doesn’t mean that software is static. We can identify dif- ferent faces of change in software development over the past five decades. Our interest in this book is what software development has learned about itself from an organizational and social perspective. Software development is perhaps working in its fourth social style of system development. Yet what is really interesting about these social styles is their ties to technical advances in the art. The first style of soft- ware development goes back to the first computers that were pro- grammed manually with console switches. The second style came with the advent of programming languages that allowed scientists to work individually or in small teams, interacting with the machine through a language. In the third style, what we learned from hardware design and manufacturing carried over into software. Formal pro- cesses drove development, management was visible and explicit, and both the system and the organizations that worked on the system were highly hierarchical. Now we are in the fourth style: one that breaks down hierarchy, that features dynamic social structures and communi- cation paths, and that values immediacy. This fourth style often bears the label “agile,” but that is just one of many characterizations of a broad new way of developing software that has emerged over the past decade. Yet human endeavors tend to take the same shape century after cen- tury, and the fact that human endeavors all have the common element of human nature bounds the range of human undertakings. For example, most organizations have leaders, and cliques, and their own little rituals, their own nuances of meaning for terms of the trade, a correspondence between physical space and organizational structure, and hundreds more. The organization of any major human endeavor follows basic laws of efficiency of communication, span of control, xenophobia, specialization, and other sociological forces that drive most large-scale projects to similar practices and structures. Much has been made of the similarity between vernacular housing architecture and the construction of software systems. [CoplienDevos2000]. The 3 same may be true for the organization of any social animals, but we leave verification of that claim to readers better tooled in those disci- plines than we are. Going deeper yet, the systems of nature have common rhythms and trends that underlie their emergent complexity. These properties come from the structure of the organization: the deeply held relationships that define the organization as a social entity. Senge [Senge1990] popu- larized this aspect of social behavior in a discipline he called systems thinking, a discipline that goes beyond our everyday disciplines that are based on a simplistic relationship between cause and effect. Many early attempts at software process improvement relied on this sim- plistic cause-and-effect relationship, particularly as the third paradigm of software development started to take hold in the industry. Most ISO 9000 process improvement efforts were run this way: find what we’re doing wrong, find the place in the process that’s wrong, and make it right. There was rarely any notion that the process as a whole might be wrong and that, for example, a step-by-step process should be replaced by a more reactive, agile process. And it was heresy to conjec- ture that process itself might be the wrong formalism to capture the crucial properties of efficient, effective systems. When we started this work in the early 1990s, we started with docu- mented research that showed serious shortcomings in process-based approaches such as ISO 9000. We asked: if process isn’t the answer, then what is? We chose organizational structure—and, in particular, the structure of relationships between roles—as the basis for system understanding. All systems are about relationships, and most disci- plines that study systems study the relationships in those systems. The idea worked. What’s better, the technique didn’t displace the process- based approaches in existing organizations (it’s always hard to tell an organization to stop doing something they think is helpful, anyhow), but complemented them by adding insight into deep structure that explained behavior at the process level. So if you already have a pro- cess improvement program in place, this book can add an enriching dimension that builds on your own culture and helps develop your peoples’ insights into that culture. Patterns provide a way to capture both the broad, invariant prac- tices of socially built artifacts as well as the specialized practices of individual disciplines, along with an understanding of how those practices build on each other. Long before Alexander started using pat- 4 Chapter terns for the field of architecture, anthropologists were using them to describe human social structures [Kroeber1948]. The pattern lan- guages in this book combine the timeless human structures that tran- scend disciplines with the best practices of contemporary software development. These patterns are all empirical: they capture the major rhythms and structures of successful software-intensive organizations today. Many of the patterns come from our own research, but we also incorporated patterns from other authors working in the same field. This is a collected work and in many ways reflects a community-wide effort. There are two equally valid views of this book: as a guide to organi- zational improvement, and as a record of the “best typical” software development structures of the fourth social paradigm of software development. Most readers will use the book in the first sense. How- ever, it is our hope that this book reflects a well-enough grounded view of contemporary software development to serve as a touchstone that records what life was like in software development organizations in the late twentieth and early twenty-first centuries. Fifty years from now, will this book be a sobering admonishment to the industry? Will we just chuckle at how things used to be? Or will time leave this fourth-generation culmination of software progress largely untouched? In that spirit, we greet the rare reader who finds an archival copy of this tome on a dusty bookshelf in the middle of the twenty-first century, and we salute your efforts to use this under- standing to further improve the lot of the organizations of your time and place. Our sense of history extends in the other direction as well. Christo- pher Alexander’s book includes a picture at the beginning of every pattern [Alexander1977]. Each picture sets a broad tone for the pattern that follows. We wanted that same feeling for this book, and we strove to include a picture for each pattern. We initially felt that the social net- work diagrams would suffice as pictures, but that gave the book an “academic” feel that left a bad taste in our mouths. Paul Bramble turned us on to the prints and photographs division of the Library of Congress (http://lcweb.loc.gov/rr/print/catalog.html), which pro- vided a wealth of vintage photos. Most come from a collection of depression-era photographs sponsored by the Farm Services Adminis- tration. Some of the photos are strikingly poignant or relevant to the patterns; that was a surprise, since they come from an era that predates 5 the software culture that is such a large part of this book. But we feel that the age and human element of the photos lends an overall charm to the book that totally would be lacking in the social network dia- grams. They also give us a feel of the timelessness of the basic human issues facing organizations of any culture, era, or ideology. This is not a book about ideology, but about human nature. Last, the pictures might help make the patterns more memorable for you: pictures are powerful association tools. Many of the pictures have a military theme. Please remember that the purpose of patterns is human quality of life and comfort, and that patterns help us capture as much learning from history’s tragedies as from its moments of peak culture. Also remember that the earliest pat- terns of human organizations (such as [Clavell1989]) have roots in mil- itary organizational structure. Acknowledgments In doing this book we view ourselves as editors and chroniclers of others’ work and ideas. There are literally thousands of people who contributed to this book through their participation in the empirical studies from which we mined the patterns. We don’t have all of their names here, but we appreciate all of them for their time and energy. Especially noteworthy were the Borland QPW Group, coordinated by David Intersimone, a remarkably productive project at AT&T led by Judy Tschirgi, highly effective projects at Schlumberger in Oslo, where we were hosted by Lise Hvatum, and the group managed by Richard Gabriel at ParcPlace Systems that was undergoing a sobering restruc- turing at the time we visited Richard. There are other people who put even more of their own energy into this book by building things for us and doing things with us. Brendan Cain was one of the original members of the Pasteur research effort at Bell Laboratories, and he wrote many of the original analysis tools. Anthropologist Peter B¸rgi was another early member of the research team; he contributed many of the insights on schismogenesis and on other direct parallels between the corporate world and the more “tra- ditional” world of cultural anthropology. Tom Burrows’ early work on the GIL and Romana environments at Bell Labs provided a platform for many of the early tools. We are grateful to Steve North for the dot tool that not only provides good supporting visuals that map out the pattern languages, but which was a key research tool in its own right. 6 Chapter We are particularly indebted to authors who let us reproduce their patterns here. Pieces of the patterns of Alistair Cockburn, Ward Cun- ningham, Bruce Whitenack and Steve Berczuk have all made their way into this collection. Thanks for sharing, folks. Gerard Meszaros wrote several patterns, including ARTIFACT OWNERSHIP, ARCHITECTURE DEFINI- TION TEAM, and ARCHITECTURE ORGANIZATION, whose contents filtered into many patterns in these pattern languages. Other people steered us in the right direction. Diane Grinnell pointed us to the references on organizational incest and its parallels to dysfunctional constellations in family therapy. Tom Stone, then at Addison-Wesley, pointed us to some dynamite references on organiza- tional learning, and in particular the studies that came out of Royal Dutch Shell. Bindu Rama Rao of Lucent pointed us to the work by Kroeber, which gave us strong ties from patterns back to the world of anthropology. Urvashi Kaul of Allstate developed pattern taxonomies that shaped how we organized these patterns into pattern languages. But most of all, we’re grateful to Moody Ahmad, who first gave us a hint back in the mid-1980s that software development research should be investigating not just the technical issues, but the human issues as well. Dozens of people reviewed these patterns in writers workshops at pattern conferences and local pattern groups. Additionally, we enjoyed the feedback of a team of focused and thorough reviewers who weren’t afraid to give us the benefit of their opinion in places where they felt we were misguided, and they were usually right. Gerhard Ackermann at Siemens in Vienna offered a wealth of first-hand insights in organizational growth and repair. Paul Bramble and Ian Graham were our two main manuscript reviewers for the late versions of the manuscript, but they offered a lot of useful advice on the organi- zational of the book. Joshua Kerievsky pioneered the conversion of these patterns to Alexandrian form with his outstanding editorial efforts on SOLO VIRTUOSO, SIZE THE ORGANIZATION, and SELF SELECTING TEAM. And many thanks to our favorite Mercenary Analyst, Betsy Hanes Perry, for her contributions to PUBLIC CHARACTER. Other key early reviewers included Jay Stagnone, ... There are others who worked with us as partners along the way, and whose editorial feedback and suggestions were fundamental to the shape of the book. Martine Devos (then of Argo in Belgium, and currently of Avaya Labs) and Steve Berczuk invested much of them- 7 selves in this work, and we are grateful for their energy and dedica- tion. We enjoyed a good technical and organizational infrastructure to support out work. At Bell Labs, the research organizations managed by Eric Sumner, Mary Zajac, and David Weiss actively supported this work over a decade. That is a long time not only in Internet years but even by research project standards. We honor their vision, patience, and forbearance. David Weiss continued to support this work in a sim- ilar capacity at Avaya Labs. Many thanks to UNIVERSIT‰T Karlsruhe for the Wiki that we used to develop the book manuscript, and in partic- ular to Dr. Helmut Goos. The support came in part under the auspices of the IST 1999-14191 EasyComp joint research project of the European Community, and we are grateful for that support. Research Chair John Roddick and fellow professor Paul Calder sponsored the infrastruc- ture and time for this work while Jim Coplien spent a summer (or was it winter?) at Flinders University in Adelaide, Australia. And of course, where would we be without our editorial support? Alan Apt has always been there in the wings, but not bugging us too much, supporting us in mighty ways. It’s been a joy working with him. This book was generated automatically from a Wiki web site using tools that generated a MIF file for transmission to the publisher. 8 Chapter 9 PART I. History And Introduc- tion This book is about people — people who write software. No, this isn’t a Dilbertesque look at our profession or an analysis of the minds of cult figures in the profession. It is about teams of real people who write real software. You see, over the last ten years or so, we have studied how people work together to create software. And we have seen that these organizations have a lot in common, whether they are writing software for telephone systems, banks, or oil exploration. The people issues shine through whatever application they are developing. And that is comforting, in a way. At the same time, though, it is disconcerting. For while organiza- tions are inanimate, they take on a life of their own. We see that organi- zations grow, learn, and sometimes even get sick! Yet they can heal themselves, and can become healthy again. Of course, we are most interested in the characteristics of the healthy organizations; perhaps other organizations can learn from those experiences. It is this notion of healing, repair and growth that are the founda- tions of Agile development. O.K., we’ll be frank: we chose “Agile” for the title out of marketing concerns. It seems to be the current term of choice for the kinds of things we describe in this book. It is a term that rolls off the tongue more easily that other clever names that clamor for your attention on today’s bookshelves. This manuscript has been 10 Chapter evolving piecemeal for over a decade, and the early pre-publication manuscripts have been a foundation and source of inspiration for many contemporary popular approaches. For example, Jeff Sutherland notes that an early publication on this work, related to one of the major case studies in this book, was one early influence on SCRUM [Sutherland2003]. Ken Schwaber notes that the early background of these pattern languages “were the genesis of some of the agile pro- cesses” [Schwaber2003]. Gabriel’s early article [Gabriel1994] and later book [Gabriel1996] discuss the successful application of these tech- niques at ParcPlace Systems as a key part of a broader effort that trans- formed the organization. And these patterns have the dubious distinction of earning the criticism of one notable software person who was the primary reviewer of the first published version of these pat- terns. He noted that anyone who worked on organizational issues was avoiding doing real work (which by his own admission was limited to anything directly related to Smalltalk programming). He would later go on to be one of the founders of Extreme Programming—a discipline that builds in part on the patterns that have been in this language (such as DEVELOPING IN PAIRS (4.2.28)) for almost a decade. Yet this book is broader than so-called agile development. We are really concerned with effective software development — the ability to produce good software efficiently, time after time. Many of the organi- zations we studied and learned from would not be considered “agile”, but they were highly effective. The 5ESS development in AT&T, for example, would fit nobody’s definition of agile. But year after year, they produced software for a system that was not only one of the largest software systems in the world, but among the world’s most reliable. Yes, many of these patterns contribute to agility, but our chief aim is effectiveness. We have captured the good things organizations do and have written them down as patterns. We hope these patterns will be as interesting and useful to you as they have been to us. Many others have found them useful: at conferences, we find that these patterns are the foundation of improvement programs in many companies world- wide. We have divided this book into four parts: • Part I: History and Introduction, the section you are reading • Part II: The patterns themselves • Part III: Foundations and History