Improved Affordability in DoD Acquisitions through Strategic Management of Systemic Cost Risk by David Petrucci M.S. Electrical Engineering, Air Force Institute of Technology, 2005 SUBMITTED TO THE SYSTEM DESIGN AND MANAGEMENT PROGRAM IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN ENGINEERING AND MANAGEMENT AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY FEBRUARY 2014 The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or hereafter created. Signature of Author______________________________________________________________ David Petrucci System Design and Management Program November 1, 2013 Certified by____________________________________________________________________ Adam M. Ross Research Scientist, Engineering Systems Lead Research Scientist, Systems Engineering Advancement Research Initiative Thesis Supervisor Accepted by___________________________________________________________________ Patrick Hale Director System Design and Management Program The views expressed in this academic research paper are those of the author and do not reflect the official policy or position of the US government or the Department of Defense. In accordance with Air Force Instruction 51-303, this research paper is not copyrighted but is the property of the United States government. 2 Improved Affordability in DoD Acquisitions through Strategic Management of Systemic Cost Risk by David Petrucci Submitted to the System Design and Management Program on November 1, 2013 in Partial Fulfillment of the Requirements for the Degree of Master of Science in Engineering and Management ABSTRACT For almost 70 years, actual costs of Major Defense Acquisition Programs (MDAPs) in the Department of Defense (DoD) have exceeded on average between 20% and 506% of their life cycle cost estimates, which are official expectations of actual program costs prior to completion. Despite numerous DoD acquisition reform efforts and implementation of sophisticated cost estimation techniques, this cost growth continues to exist. Accurate cost estimates are vital to the capital budgeting process for the DoD since they are used to set the affordability cap for each MDAP and across DoD Component weapon system program portfolios. Affordability is defined as the upper limit a DoD Component can allocate for a program without reducing costs or shifting resources between programs. To improve affordability in the DoD, a method that quantifies and adjusts for the persistent cost growth to enhance the accuracy of cost estimates is needed to promote more responsible and sustainable MDAP capital investment decisions. This thesis presents a simple yet powerful method of quantifying and correcting for systemic cost estimation risk in MDAPs to improve cost estimate accuracy and, consequently, affordability. Cost estimation risk is defined as the difference between estimated and actual MDAP costs (on average a deficit), and it consists of systemic and program-specific components. This dichotomized risk framework is similar to the one used in the Capital Asset Pricing Model (CAPM) in which the growth rate in value of any one of a set of assets comprising a market in equilibrium is proportional to its systemic risk exposure to that market. In the CAPM, systemic risk – aggregated risk from multiple economic factors – pervades the market and is unavoidable, and asset-specific risk is considered unpredictable due to idiosyncratic uncertainties. By analogy, the growth rate in cost estimates for a program belonging to the “market” of MDAPs is assumed proportional to that MDAP’s systemic risk exposure to the market. Like in CAPM, systemic cost estimation risk – aggregated risk from 26 systemic factors identified in this thesis – pervades the market for MDAPs, as evidenced by historical cost overruns, and program-specific cost risk is considered unpredictable and best mitigated by program-dedicated professional cost estimators in the DoD and defense industry. From this analogy, the expected growth-beta relationship of CAPM may be adapted to determine for MDAPs the systemic cost risk-adjusted growth rates for the defense commodity classes of aircraft, electronics and software, missiles, ships, space and satellites, and vehicles. These classes are the same used by the Bureau of Economic Analysis to segment defense commodities into distinct price index “baskets” based upon common inflation risks among commodities within each class. Based on this rationale, each MDAP is assumed to 3 share systemic cost risk within its respective class; this risk is measured by beta in the expected growth-beta relationship. Defense commodity class beta values are calculated by linear regression of historical percentage cost estimate changes of member MDAPs with those of all MDAPs, and then averaging these beta values over the appropriate defense commodity class. Next, the expected cost estimate growth rate for any MDAP may be calculated by first estimating the future expected growth rate in all MDAPs using the arithmetic mean of historical annual cost estimate percentage changes, and then scaling this rate by the particular MDAP’s systemic cost risk exposure – the defense commodity class beta value for which it is a member. Finally, the Systemic Risk Factor (SRF) for each defense commodity class is calculated from these growth rates and the forecast time horizon, adjusted for compounding effects over relatively longer time horizons, and then applied to current MDAP cost estimates to form systemic risk-adjusted cost estimates to improve affordability. This method was applied to an empirical retrospective case study using a set of cost data from six MDAPs, one from each commodity class, as a partial validation of the method. The results of this study show an overall 57% enhancement in estimation accuracy when comparing the initial and SRF-adjusted MDAP cost estimates to the final estimates, indicating quantifying and adjusting for systemic cost risk can improve cost estimation accuracy. To show the effectiveness of this method on improving affordability, these six programs were combined to form a hypothetical acquisition portfolio and assessed for affordability over a five year period. While the unadjusted portfolio was not affordable four out of five years, the SRF-adjusted portfolio was affordable in all but the last year, illustrating the benefit of adjusting cost estimates for systemic risk. However, the benefits of improved cost estimate accuracy and affordability come at the cost of potentially over-budgeting for priority MDAPs thereby leaving less funds available for other, lower priority programs. Additionally, this method is not shown to be optimal in the sense of minimizing cost estimate errors to maximize affordability. Still, empirical results are promising and warrant further research into the idea of using SRFs to adjust life cycle cost estimates and ultimately improve MDAP affordability for the DoD. Thesis Supervisor: Adam M. Ross Title: Lead Research Scientist, Systems Engineering Advancement Research Initiative 4 ACKNOWLEDGEMENTS I would like to convey my appreciation for the assistance received in support of my academic program and thesis. Mr. Pat Hale and Dr. Deborah Nightingale provided valuable curriculum guidance, thesis topic advice, and administrative support, both personally and through the personnel in programs they direct. Mr. Bill Foley helped me navigate the many and varied course scheduling and status update processes. Dr. Donna Rhodes introduced me to my thesis advisor, Dr. Adam Ross, whose comments and critiques identified many areas for improvement in this work. Each of these individuals were very generous in finding for me time in their busy schedules for which I am grateful. Several USAF offices and personnel greatly enhanced my experience. The USAF Fellowship office provided essential mission support and links to the “big blue” Air Force to add relevance to this work. Mr. Richard Lombardi, who sponsored this research and provided its central idea, was instrumental in the successful development of this thesis. The USAF Fellows, both past and present, provided great program and community support. Dustin, Mike, and Marc opened the door for Dave, Scott, and I, and helped us establish a foothold in our MIT programs and the local area. I then had the privilege of returning the favor by introducing the new set of Fellows to MIT. I wish Najeeb, Gerry, and Josh success in their academic endeavors and know each will flourish in their new environment. Finally, I am thankful to my family for their love and understanding in allowing me to spend the necessary time in my studies, regardless of the time or day. 5 TABLE OF CONTENTS 1. INTRODUCTION................................................................................................................. 9 1.1. Epoch Shift ..................................................................................................................................................... 9 1.2. Affordability in DoD Acquisitions ............................................................................................................. 12 1.3. Thesis Development ..................................................................................................................................... 16 1.4. Scope & Research Questions ...................................................................................................................... 17 1.5. Thesis Organization .................................................................................................................................... 18 2. LITERATURE REVIEW .................................................................................................. 20 2.1. DoD Acquisition Program Cost Estimation .............................................................................................. 20 2.2. Cost Growth in DoD Acquisition Programs .............................................................................................. 23 2.2.1. Trends in MDAP Cost Growth ................................................................................................................. 25 2.2.2. Factors Affecting MDAP Cost Growth .................................................................................................... 26 2.2.3. SAR Data Limitations & Adjustments ..................................................................................................... 27 2.3. Capital Asset Pricing Model Applied to DoD Acquisition Programs ..................................................... 28 2.4. Literature Review Summary ...................................................................................................................... 29 3. DOD ACQUISITION PROGRAM COST ESTIMATION ............................................ 31 3.1. DoD Decision Support Systems .................................................................................................................. 31 3.2. Cost Estimates for MDAPs ......................................................................................................................... 32 3.2.1. Relationship between Affordability & Cost Estimates ............................................................................. 33 3.2.2. Cost Estimation Best Practices ................................................................................................................. 35 3.3. Factors Affecting Poor Cost Estimates & MDAP Cost Overruns ........................................................... 47 3.4. Failed Acquisition Reforms Intended to Mitigate MDAP Cost Overruns ............................................. 49 3.5. A New Approach to Mitigate MDAP Cost Overruns ............................................................................... 51 4. THE TIME VALUE OF MONEY & SYSTEMIC RISK-ADJUSTED ASSET VALUATION .............................................................................................................................. 53 4.1. Time Value of Money in the DoD ............................................................................................................... 53 4.1.1. General Inflation Rate .............................................................................................................................. 56 4.1.2. Defense Commodity Price Indices ........................................................................................................... 58 4.1.3. Discount Rates .......................................................................................................................................... 59 4.1.4. Growth Rates ............................................................................................................................................ 60 4.2. Base-Year & Then-Year Dollars ................................................................................................................ 61 6 4.3. Capital Asset Pricing Model ....................................................................................................................... 62 4.3.1. The Expected Growth-Beta Relationship ................................................................................................. 64 4.3.2. Limitations of the CAPM ......................................................................................................................... 66 5. SYSTEMIC RISK-ADJUSTED COST GROWTH RATES FOR DOD ACQUISITION PROGRAMS ............................................................................................................................... 69 5.1. Expected Growth-Beta Expression for MDAP Cost Estimates ............................................................... 69 5.2. Retrieving Cost Estimate Data From SARs .............................................................................................. 71 5.3. Procedure for Calculating Systemic Risk-Adjusted Cost Estimate Growth Rates for MDAPs ........... 72 5.4. Calculation of Beta Values & Systemic Risk-Adjusted Cost Estimate Growth Rates .......................... 76 5.5. Interpretation of Beta Values & Systemic Risk-Adjusted Cost Estimate Growth Rates ...................... 77 5.6. Systemic Risk Factors & Method Validation ............................................................................................ 79 5.7. Application to Affordability in DoD Acquisitions .................................................................................... 82 5.8. Summary ...................................................................................................................................................... 86 6. CONCLUSIONS & RECOMMENDATIONS ................................................................. 87 6.1. Restatement of Motivation & Research Questions ................................................................................... 87 6.2. Findings & Contributions ........................................................................................................................... 88 6.3. Recommendations ....................................................................................................................................... 89 6.4. Limitations ................................................................................................................................................... 90 6.5. Possibilities for Future Research ............................................................................................................... 91 LIST OF REFERENCES ........................................................................................................... 93 APPENDIX A – SUMMARY OF EQUATIONS & METHODS ........................................... 96 APPENDIX B – LIST OF ABBREVIATIONS ........................................................................ 98 APPENDIX C – LIST OF DEFINITIONS ............................................................................... 99 APPENDIX D – MDAP NOMINAL ANNUAL COST CHANGE, 1970-2012 ................... 100 APPENDIX E – BETA CALCULATION SPREADSHEETS .............................................. 101 7 List of Figures Figure 1: A shift in DoD acquisition strategy, in the context of changing national defense strategy & annual MDAP funding uncertainty. ............................................................................ 12 Figure 2: Example DoD Component portfolio Affordability Assessment from [13]. ................. 14 Figure 3: DoD weapon system development life cycle from [10]. .............................................. 15 Figure 4: DoD decision support systems (adapted from [10]). .................................................... 32 Figure 5: Illustrative example of life-cycle costs (adapted from [10]). ....................................... 33 Figure 6: Decision points & milestone reviews requiring LCCEs (adapted from [14]). ............. 33 Figure 7: The cost estimating process contained in [13]. ............................................................ 35 Figure 8: Example WBS for an aircraft. Not all elements shown. Adapted from [13]. ............ 38 Figure 9: Basic primary & secondary sources of cost data (from [13]). ...................................... 40 Figure 10: Three commonly used cost estimating methods in the DoD (from [13]). .................. 42 Figure 11: Example sensitivity analysis showing the point, low, & high cost estimate (from [13])............................................................................................................................................... 43 Figure 12: Monte Carlo simulation of the lower-level WBS elements creates an estimate of the total system cost estimate’s probability distribution (from [13]). Note that the sum of the reference point estimates from the lower-level estimates does not necessarily produce the most likely total system cost estimate. .................................................................................................. 45 Figure 13: An S-curve used to show levels of confidence in total system cost estimates (from [13])............................................................................................................................................... 46 Figure 14: Significant & critical MDAP Nunn-McCurdy breaches between 1997 & 2009 (from [16])............................................................................................................................................... 50 Figure 15: Distribution of cost growth factors in MDAPs from 1968-2003 (from [18]). ........... 51 Figure 16: Appropriation life of various types of DoD funding [30]. ......................................... 54 Figure 17: Inflation (GDP Deflator) & defense commodity price indices over time [32]. ......... 59 Figure 18: Example CAPM linear regression for Yahoo!. .......................................................... 64 Figure 19: Linear regression analysis of the AIM-9X Block I program annual cost estimate percent changes from SARs. ......................................................................................................... 71 Figure 20: Graphical representation of the eight-step procedure for calculating systemic risk- adjusted cost estimate growth rates for MDAPs. “C#” subscripts are used as shorthand for “Commodity #.” ............................................................................................................................ 75 Figure 21: Affordability Assessment of six historical MDAPs in a hypothetical portfolio. ....... 85 List of Tables Table 1: Summary of DoD weapon system acquisition cost growth studies reviewed. .............. 24 Table 2: Beta estimates for each defense commodity type. ......................................................... 76 Table 3: Systemic risk-adjusted cost estimate growth rates for each defense commodity type. . 77 Table 4: SRF adjustment to correct for compounding effects. .................................................... 80 Table 5: SAR cost estimates & adjustment for systemic cost risk improvements for the six historical programs evaluated for validation. ................................................................................ 81 8 1. INTRODUCTION T his work is the culmination of a one-and-one-half year United States Air Force (USAF) Fellowship Program intended to broaden and enrich the experience and education of its members through participation in the Massachusetts Institute of Technology’s (MIT’s) System Design and Management program. Through the pursuit of their fellowship goals, Airmen1 are expected to intellectually venture outside the bounds of their profession to explore novel and relevant ideas to bring back with them upon completion of their fellowship. It is the spirit of this mission that inspired this exploratory research in applying best business practice to strategic management of systemic weapon system acquisition cost risk. As the USAF, along with her sister services, continues to uneasily settle into an epoch of fiscal uncertainty, a new acquisition cost risk strategy is needed. Motivation for this thesis is drawn from this new and challenging environment. 1.1. EPOCH SHIFT In general, the Department of Defense (DoD) acquisition enterprise’s external environment includes exogenous forces which shift contextual factors. These shifts impart great pressures on an enterprise, eventually triggering internal changes [1]. Such effects change over eras and epochs [1]. Eras are defined as the full lifetime of the current instantiation of the enterprise, while epochs are shorter periods of time characterized by fixed contextual factors and value expectations [2]. In the case of the DoD acquisition enterprise, it is currently enjoying the benefits of the sole superpower status of the United States (US), which began after the end of the Cold War in 1991 and will likely continue for decades. From the perspective of defense acquisition strategy, this Sole Superpower Era began with the Post-Cold War Epoch of threat-based acquisition strategies2, 1 The general term for any USAF member signifying the unifying and traditional culture of the USAF’s first mission in the air domain. The USAF has since expanded into the space and cyber domains. 2 Standard terminology in the Department of Defense is threat-based planning acquisition strategy and similarly for capabilities-based planning acquisition strategy. This work omits “planning” to save space in graphics. 9 as formerly Soviet war materiel continued to be the dominant weapons threat even though the nationality of the enemy combatants had changed. After almost a decade, the US entered the Global War on Terrorism Epoch characterized (with respect to defense acquisition strategy) by the idea of capabilities-based acquisition strategy [3]. The 2001 Quadrennial Defense Review finally acknowledged that the new combatants may be unknown and employ weapon systems different from the formerly Soviet arsenal [3]. As such, the focus shifted from specific-threat weapon systems to the possible capabilities of unknown adversaries [3]. Presently the DoD acquisition enterprise finds itself in the Pivot to Asia Epoch and switching to a new affordable capabilities-based acquisition strategy. In his 2012 “Priorities for the 21st Century Defense” strategic guidance, President Barrack Obama called for rebalancing the US military towards the Asia-Pacific region while continuing to provide global security [4]. This change in defense posture has meant a continuation of the capabilities-based acquisition strategy [5]. However, concurrent with this shift in national security posture is a national debt crisis which exerts tremendous forces on the DoD acquisition enterprise, reinvigorating a focus on affordability in acquisition strategy. Known as “Sequestration,” the Budget Control Act requires shedding $487B from defense budgets between 2013 and 2022 [6]. These forced cuts are in addition to $400B of voluntary budget reductions in the DoD negotiated in 2011 over a 10-year period [7]. If budget cuts are planned and known, then the defense acquisition enterprise might devise acquisition strategies to adapt to the new environment. However, future budgets are confounded by large uncertainties in annual budget authorizations and surprise expenses. Secretary of Defense Chuck Hagel warned of cost overruns and the potential for additional budget overages due to the Pivot to Asia during 10