IMPACTS OF SYSTEM OF SYSTEM MANAGEMENT STRATEGIES ON SYSTEM OF SYSTEM CAPABILITY ENGINEERING EFFORT by Jo Ann Lane A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (INDUSTRIAL AND SYSTEMS ENGINEERING) May 2009 Copyright 2009 Jo Ann Lane Acknowledgements This research endeavor is the culmination of a lifelong dream that has been supported and encouraged by many. The realization of this dream started with my soul-mate, Mike, who has always encouraged me to follow my dreams and has continued to lend an ear and challenge me when I needed to explore new concepts. As I pursued the possibility of an advanced degree, I found great enthusiasm and support from my advisor, Dr. Barry W. Boehm. He has guided me through a wonderful adventure, helped me keep my sense of humor as I struggled with the reality of “state-of-the-art” and the availability of data from industry, and kept me out of deep pits and quicksand. The realization of this research effort also exists because of the tremendous support from my other committee members, Dr. F. Stan Settles, Dr. George Friedman, and Dr. Paul Adler, as well as the constructive systems engineering cost model (COSYSMO) research work done by Dr. Ricardo Valerdi. In addition, this research could not have been conducted without support from the University of Southern California Center for Systems and Software Engineering corporate, government, and academic affiliates. Other key supporters include the Department of Defense (DoD) Office of the Secretary of Defense (OSD) Systems Engineering Guide for System of Systems sponsors and researchers (Kristen Baldwin, Dr. Judith Dahmann, Ralph Lowry, and George Rebovich); Rob Flowe; Dr. David Zubrow; Dr. Michael Green and his Naval Postgraduate School students; The Aerospace Corporation (Marilee Wheaton and Richard Adams) and their consultants (Donald Greer and Dr. Laura Black); and Cheryl Jones (US Army). Finally, I would also like to acknowledge my other moral supporters: fellow student ii Indrajeet Dixit, Dr. Mary Anne Herndon, Wendy Becker Hunt, Tony Jordano, and the computer science faculty at San Diego State University, especially Dr. Leland Beck, Dr. Carl Eckberg, Dr. Teresa Larson, and Dr. Marie Roch. And to the nay-sayers who thought systems of systems engineering is no different from the systems engineering that engineers have been doing for decades: you kept me going with your personal observations and questions. It turned out that we were all correct—it just depends upon your point of view of the system of systems. This research has also received support from the following organizations: International Council on Systems Engineering Foundation/Steven PhD Award Committee; Practical Software and Systems Measurement; the Space Systems Cost Analysis Group; and DoD OSD Acquisition, Technology, and Logistics (AT&L) Software Engineering and Systems Assurance (SSA). iii Table of Contents Acknowledgements ii List of Tables vi List of Figures vii Abstract viii Chapter 1: Introduction 1 1.1 Research Overview 1 1.2 Motivation 2 1.3 Proposition and Hypotheses 3 1.4 Intended Research Contribution 4 Chapter 2: Background and Related Work 5 2.1 Overview of Literature Review 5 2.2 What Is a System of Systems? 6 2.3 Related Engineering Disciplines 8 2.4 Traditional SE, SoSE, and Related Industry Standards 9 2.5 Engineering Cost Models 14 2.6 Related Organizational Theory Concepts 18 2.7 Process Models 23 Chapter 3: Methodology 25 3.1 Overview of Research Design 25 3.2 Data Collection Instruments 26 3.3 Data Analysis 29 3.4 Potential Threats to Validity and Limitations 30 Chapter 4: The SoSE Model 36 4.1 SoSE Process Model Assumptions and Constraints 37 4.2 SoSE Model Effort Calculations 41 4.3 SoSE Model Effort Multipliers for Effort Calculations 44 4.4 SoSE Process Model Parameter Variations 51 Chapter 5: Research Results 52 5.1 SoS Size Variation 52 5.2 Scope of SoS Capability 57 5.3 Summary of Model Executions 63 Chapter 6: Conclusions 67 6.1 General Conclusions 67 6.2 Future Work 68 6.3 Summary of Research Contributions 69 References 70 iv Appendix A: DoD SoS Case Study Summaries 75 Appendix B: System Interdependency Survey 77 Appendix C: COSYSMO Parameter Definitions Tailored for SoSE 81 Comparison Model v List of Tables Table 1. SoSE Management Approaches 1 Table 2. Summary of COSYSMO Parameters [Valerdi, 2005] 16 Table 3. Mapping of DoD SoSE Core Elements to COSYSMO Parameters 35 Table 4. SoSE Model Equation Term Definitions 42 Table 5. SoSE EM for SoS Capability Requirements Rationale 45 Table 6. SoSE Oversight Requirements Cost Factor Rationale 47 Table 7. “SE for SoS Requirements, SoSE Support” Cost Factor Rationale 48 Table 8. “SE for SoS Requirements, No SoSE Support” Cost Factor Rationale 49 Table 9. SE for Non-SoS-Requirements Cost Factor Rationale 50 Table 10. SoSE Model Parameters 51 Table 11. Summary of SoSE Model Executions 64 Table A-1. DoD SoS Case Study Summaries 75 Table B-1. Summary of System Interdependency Survey Responses Part I 79 Table B-2. Summary of System Interdependency Survey Responses Part II 80 vi List of Figures Figure 1. Graphical View of SoSE Model 36 Figure 2. SoSE EM for SoS Requirements 45 Figure 3. SoSE EM to Monitor Constituent-system Non-SoS-Requirements 46 Figure 4. SE EM for SoS Requirements with SoSE Support 47 Figure 5. SE EM for SoS Requirements without SoSE Support 48 Figure 6. SE Effort Multiplier for System-Specific Non-SoS-Requirements 50 Figure 7. SoSE Model Case 1 52 Figure 8. SoSE Model Case 2 53 Figure 9. SoSE Model Case 3 54 Figure 10. SoSE Model Case 4 55 Figure 11. SoSE Model Case 5 56 Figure 12. SoSE Model Case 6 57 Figure 13. SoSE Model Case 7a (SoS Size = 10) 58 Figure 14. SoSE Model Case 7b (SoS Size = 100) 58 Figure 15. SoSE Model Case 8 59 Figure 16. SoSE Model Case 9 60 Figure 17. SoSE Model Case 10 61 Figure 18. SoSE Model Case 11 62 Figure 19. SoSE Model Case 12 63 Figure 20. Research Summary 67 vii Abstract Today’s need for more complex, more capable systems in a short timeframe is leading more organizations towards the integration of new and existing systems with commercial-off-the-shelf (COTS) products into network-centric, knowledge-based systems of systems (SoS). With this approach, system development processes to define the new architecture, identify sources to either supply or develop the required components, and eventually integrate and test these high level components are evolving and are being referred to as SoS Engineering (SoSE). In recent years, the systems engineering (SE) community has struggled to decide if SoSE is really different from traditional SE and, if it is different, how does it differ. Recent research and case studies [DoD, 2008] have confirmed that there are indeed key differences and that traditional SE processes are not sufficient for SoSE. However, as with any engineering discipline, how and how much SoSE differs depends on several factors. This research further investigated SoSE through the study of several large-scale SoSE programs and several SE programs that were considered part of one or more SoSs to identify key SoSE strategies and how these strategies differed based on SoS characteristics and constituent-systems. The results of these investigations were then captured in a system dynamics model that allows one to explore SoSE options with respect to engineering effort and return on SoSE investment. Two SoS capability development strategies (with and without an SoSE team to guide capability development) were compared and used to assess the value-added of the SoSE team with respect to total SE effort expended to engineer an SoS capability. It is clear from both the Office of the Secretary of Defense (OSD) pilot studies viii [DoD, 2008] and the system dynamics model analysis conducted as part of this research that there exist conditions under which investments in SoSE have positive and negative returns on investment. This dissertation provides the first quantitative determination of these conditions, and points out directions for future research that would strengthen the results. ix Chapter 1: Introduction 1.1 Research Overview Today’s need for more complex, more capable systems in a short timeframe is leading more organizations towards the integration of new and existing systems with commercial-off-the-shelf (COTS) products into network-centric, knowledge-based system of systems (SoS). With this approach, system development processes to define the new architecture, identify sources to either supply or develop the required components, and eventually integrate and test these high level components are evolving and are being referred to as SoS Engineering (SoSE) [Lane and Valerdi, 2007; Ring and Madni, 2005]. As a result of recent SoS research [Maier, 1998; Dahmann and Baldwin, 2008], four types of SoSE management approaches have been identified: virtual, collaborative, acknowledged, and directed. These categories are primarily based upon the levels of responsibility and authority overseeing the evolution of the SoS. Table 1 describes these SoSE management approaches. Table 1. SoSE Management Approaches Type Description Virtual [Maier, (cid:1) Lacks a central management authority and a clear SoS purpose. 1998] (cid:1) Often ad hoc and may use a service-oriented architecture where the constituent-systems are not necessarily known. Collaborative (cid:1) Constituent-system engineering teams work together more or less [Maier, 1998] voluntarily to fulfill agreed upon central purposes. (cid:1) No SoSE team to guide or manage SoS-related activities of constituent- systems. Acknowledged (cid:1) Have recognized objectives, a designated manager, and resources at the [Dahmann and SoS level (SoSE team), but not complete authority over constituent- Baldwin, 2008] systems. (cid:1) Constituent-systems maintain their independent ownership, objectives, funding, and development approaches. Directed (cid:1) SoS centrally managed by a government, corporate, or Lead System [Maier, 1998] Integrator (LSI) and built to fulfill specific purposes. (cid:1) Constituent-systems maintain ability to operate independently, but evolution predominately controlled by SoS management organization. 1