Review of NASA’s Aerospace Technology Enterprise An Assessment of NASA’s Pioneering Revolutionary Technology Program Committee for the Review of NASA’s Pioneering Revolutionary Technology (PRT) Program Aeronautics and Space Engineering Board Division on Engineering and Physical Sciences THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the com- mittee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Contract No. NASW 99037 between the National Academy of Sciences and the National Aeronautics and Space Administration. Any opinions, findings, conclusions, or recom- mendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the agency that provided support for the project. International Standard Book Number 0-309-09080-6 (Book) International Standard Book Number 0-309-52759-7 (PDF) Cover images courtesy of the National Aeronautics and Space Administration. Available in limited supply from: Additional copies are available from: Aeronautics and Space Engineering Board National Academy Press 500 Fifth Street, N.W. Box 285 Washington, DC 20001 500 Fifth Street, N.W. (202) 334-2855 Washington, DC 20055 (800) 624-6242 (202) 334-3313 (in the Washington metropolitan area) http://www.nas.edu Copyright 2003 by the National Academy of Sciences. All rights reserved. Printed in the United States of America The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished schol- ars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and tech- nical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Acad- emy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering pro- grams aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Acad- emy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council. www.national-academies.org COMMITTEE FOR THE REVIEW OF NASA’S PIONEERING REVOLUTIONARY TECHNOLOGY (PRT) PROGRAM RAYMOND S. COLLADAY, Chair, Consultant and President (retired), Lockheed Martin Astronautics, Denver BENJAMIN BUCHBINDER, NASA (retired), Bonaire, Netherlands Antilles LEONARD H. CAVENY, Ballistic Missile Defense Organization (retired), Fort Washington, Maryland SERGIO GUARRO, Aerospace Corporation, El Segundo, California (from June 2002 until April 2003) DAVID J. KASIK, The Boeing Company, Seattle DIMITRI MAVRIS, Georgia Institute of Technology, Atlanta DENNIS K. McBRIDE, Potomac Institute for Policy Studies, Arlington, Virginia TODD J. MOSHER, Utah State University, Logan JAMES ODOM, Science Applications International Corporation, Huntsville, Alabama LEE D. PETERSON, University of Colorado, Boulder JOSEPH B. REAGAN (NAE), Lockheed Martin Missiles and Space (retired), Saratoga, California CYNTHIA R. SAMUELSON, Logistics Management Institute, McLean, Virginia MARC SNIR, University of Illinois, Urbana-Champaign MICHAEL J. ZYDA, Naval Postgraduate School, Monterey, California PANEL ON COMPUTING, INFORMATION, AND COMMUNICATIONS TECHNOLOGIES (CICT) MICHAEL J. ZYDA, Chair, Naval Postgraduate School, Monterey, California WILLIAM COHEN, Consultant, Pittsburgh (from June 2002 until June 2003) DELORES M. ETTER (NAE), United States Naval Academy, Annapolis, Maryland MARY JEAN HARROLD, Georgia Institute of Technology, Atlanta CHANDRIKA KAMATH, Lawrence Livermore National Laboratory, Livermore, California DAVID J. KASIK, The Boeing Company, Seattle ALFRED U. MacRAE (NAE), MacRae Technologies, Berkeley Heights, New Jersey DUANE T. McRUER (NAE), Systems Technology, Inc., Manhattan Beach, California RICHARD MULLER (NAE), University of California, Berkeley CYNTHIA R. SAMUELSON, Logistics Management Institute, McLean, Virginia JUDE SHAVLIK, University of Wisconsin, Madison SANDEEP SINGHAL, ReefEdge, Inc., Fort Lee, New Jersey MARC SNIR, University of Illinois, Urbana-Champaign PANEL ON ENGINEERING FOR COMPLEX SYSTEMS (ECS) DENNIS K. McBRIDE, Chair, Potomac Institute for Policy Studies, Arlington, Virginia TORA K. BIKSON, RAND Corporation, Santa Monica, California BENJAMIN BUCHBINDER, NASA (retired), Bonaire, Netherlands Antilles PHILIP R. COHEN, Oregon Health and Science University, Beaverton SERGIO GUARRO, Aerospace Corporation, El Segundo, California (from June 2002 until April 2003) MYRON HECHT, SoHaR Incorporated, Beverly Hills, California JIM LARUS, Microsoft Research, Redmond, Washington DIMITRI MAVRIS, Georgia Institute of Technology, Atlanta RONALD WESTRUM, Consultant, Ann Arbor, Michigan (from June 2002 until February 2003) F. GORDON WILLIS, Vulcan Works, LLC, Ann Arbor, Michigan iv PANEL ON ENABLING CONCEPTS AND TECHNOLOGIES (ECT) LEE D. PETERSON, Chair, University of Colorado, Boulder CLINTON A. BOYE, Sandia National Laboratories, Albuquerque, New Mexico LEONARD H. CAVENY, Ballistic Missile Defense Organization (retired), Fort Washington, Maryland STANLEY V. GUNN, Rocketdyne (retired), Chatsworth, California ANTHONY K. HYDER, University of Notre Dame, South Bend, Indiana DIMITRIS C. LAGOUDAS, Texas A&M University, College Station TODD J. MOSHER, Utah State University, Logan JAY S. PEARLMAN, The Boeing Company, Seattle JOSEPH B. REAGAN (NAE), Lockheed Martin Missiles and Space (retired), Saratoga, California NANCY R. SOTTOS, University of Illinois, Urbana-Champaign GREGORY G. SPANJERS, Air Force Research Laboratory, Albuquerque, New Mexico MICHAEL J. STALLARD, Aerospace Corporation, Albuquerque, New Mexico COMMITTEE AND PANELS STAFF KAREN E. HARWELL, Study Director DOUGLAS H. BENNETT, Program Officer GEORGE M. LEVIN, Director, Aeronautics and Space Engineering Board BRIDGET R. EDMONDS, Senior Project Assistant JENNIFER D. PINKERMAN, Research Associate ANNA L. FARRAR, Financial Associate v AERONAUTICS AND SPACE ENGINEERING BOARD WILLIAM W. HOOVER, Chair, U.S. Air Force (retired), Williamsburg, Virginia A. DWIGHT ABBOTT, Aerospace Corporation (retired), Palos Verdes Estates, California RUZENA K. BAJCSY (NAE/IOM), University of California, Berkeley JAMES (MICKY) BLACKWELL, Lockheed Martin (retired), Marietta, Georgia ANTHONY J. BRODERICK, Aviation Safety Consultant, Catlett, Virginia SUSAN M. COUGHLIN, Aviation Safety Alliance, Washington, D.C. ROBERT L. CRIPPEN, Thiokol Propulsion (retired), Palm Beach Gardens, Florida DONALD L. CROMER, USAF (retired) and Hughes Space and Communications (retired), Fallbrook, California JOSEPH FULLER, Jr., Futron Corporation, Bethesda, Maryland RICHARD GOLASZEWSKI, GRA Incorporated, Jenkintown, Pennsylvania JAMES M. GUYETTE, Rolls-Royce North America, Chantilly, Virginia JOHN L. JUNKINS (NAE), Texas A&M University, College Station JOHN M. KLINEBERG, Space Systems/Loral (retired), Redwood City, California ILAN M. KROO, Stanford University, Stanford, California JOHN K. LAUBER, Airbus North America, Inc., Washington, D.C. GEORGE K. MUELLNER, The Boeing Company, Seal Beach, California DAVA J. NEWMAN, Massachusetts Institute of Technology, Cambridge JAMES G. O’CONNOR (NAE), Pratt & Whitney (retired), Coventry, Connecticut MALCOLM O’NEILL, Lockheed Martin Corporation, Bethesda, Maryland CYNTHIA R. SAMUELSON, Logistics Management Institute, McLean, Virginia KATHRYN C. THORNTON, University of Virginia, Charlottesville HANSEL E. TOOKES II, Raytheon International (retired), Falls Church, Virginia DIANNE S. (WILEY) PALMER, The Boeing Company, Washington, D.C. THOMAS L. WILLIAMS, Northrop Grumman, Bethpage, New York GEORGE M. LEVIN, Director vi Preface The Committee for the Review of NASA’s Pio- The committee and panels did not assess other pro- neering Revolutionary Technology (PRT) Program of grams within NASA on which the PRT program and its the National Research Council (NRC) and its three sup- portfolio depend or other programs within NASA that porting panels have completed an approximately 20- research similar technology areas. The committee and month-long study evaluating the technical quality of panels did recommend when these programs should be the National Aeronautics and Space Administration’s integrally connected and the PRT portfolios managed (NASA’s) PRT program. The statement of task for this with the global NASA investment in mind. study is given in Appendix A. The study was spon- I wish to take this opportunity to thank the chairs sored by NASA and conducted by a committee and and members of the three supporting panels for their three supporting panels appointed by the NRC (see leadership, detailed assessments, and commitment of Appendix B for biographies of committee and panel time to the review. Their input has been vital to the members). The Office of Management and Budget quality of the entire review. On behalf of the commit- (OMB) requested the review and assisted in the formu- tee and panels, I would also like to thank the various lation of the statement of task. NASA program managers and technical staff for their This report provides a technical assessment of the cooperation in providing the information necessary to quality of the PRT program and its components and complete the review and in hosting our panel members offers recommendations for improving the program. at various site visits and for their open discussion dur- The committee and panels note that they refrained from ing these opportunities. We also thank those who took drawing any conclusions on matters of budget or rec- the time to participate in committee and panel meet- ommending increases in budget levels. While some ings and provide background materials. Finally, this areas may suffer from a lack of critical mass, recom- study and the final report would not have been possible mendations for increased resources to address the prob- without the expert support of the NRC staff. Their dedi- lem are of little value to management and have been cation to keeping the review on track deserves special avoided. The committee and panels also refrained, as recognition and thanks. Thanks go especially to Karen much as possible, from commenting on matters related E. Harwell, study director, for her professional steering to programmatics and program organization unless a of the overall committee effort as well as her support to link could be established between these concerns and the ECT panel, and to Douglas H. Bennett for his sup- technical quality, portfolio management, or interaction port to the CICT and ECS panels. within NASA and with the external technical commu- nity. NASA’s Aerospace Technology Advisory Com- Raymond S. Colladay, Chair mittee (ATAC) and its PRT subcommittee hold an an- Committee for the Review of NASA’s nual relevance and programmatic review for the PRT Pioneering Revolutionary Technology program. (PRT) Program vii Acknowledgment of Reviewers This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s (NRC’s) Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manu- script remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Dwight Abbott, Aerospace Corporation (retired), Douglas Allen, Schafer Corporation, George Apostolakis, Massachusetts Institute of Technology, Daniel Baker, University of Colorado, Vicki Bier, University of Wisconsin, John Evans, COMSAT (retired), Michael Frank, Safety Factor Associates, Inc., Henry Helvajian, Aerospace Corporation, William Howard, Consultant, James McGroddy, IBM Corporation, Phil Papadopoulos, University of California, San Diego, Suraj Rawal, Lockheed Martin Corporation, Walter Robb, Vantage Management, Inc., Richard Schwartz, Purdue University, Norman Sleep, Stanford University, Patrick Stadter, Johns Hopkins University Applied Physics Laboratory, David Waltz, NEC Research Institute, Inc. (retired), and Mary Young, HRL Laboratories. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Alexander H. Flax, Consultant. Appointed by the National Research Council, he was responsible for making certain than an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Respon- sibility for the final content of this report rests entirely with the authoring committee and the institution. viii Contents EXECUTIVE SUMMARY 1 1 INTRODUCTION 11 Background, 11 Approach to Assessment, 11 Report Organization and Development, 12 Reference, 12 2 OVERALL ASSESSMENT OF THE PIONEERING REVOLUTIONARY 13 TECHNOLOGY PROGRAM Overall Assessment, 13 Common Themes, 14 Systems Analysis, 14 Benchmarking and Metrics, 15 External Peer Review and Competition, 15 Stability and Continuity, 17 Research Portfolio Balance, 17 Technology Transition, 19 References, 19 Briefings, 19 3 REPORT OF THE PANEL ON COMPUTING, INFORMATION, AND 20 COMMUNICATIONS TECHNOLOGY Introduction, 20 Review Process, 21 Overall Observations on the CICT Program, 21 General Observations, 22 Research Program Architecture, 23 ix x CONTENTS Service-Oriented Tasks, 23 Final Research Applications, 26 Final Products and Research Benchmarks, 26 Research Portfolio, 27 Detailed Assessment of Research Portfolio, 27 Overlap with Other PRT Programs, 28 Expanding Existing Research Areas, 28 Critical Computing Expertise That May Be Missing, 30 Research Plans and Methodology, 30 Task Deliverables and Their Fit to NASA Goals, 31 Maturing a Technology, 31 Reviewing and Selecting Proposals, 32 Technology Readiness Level, 33 Reorganization of Projects and Management Structure, 33 Technical Community Connections, 33 Awareness of Relevant Research, 35 Use of Talent Inside and Outside NASA, 36 Benchmark Datasets and Problem Sets, 36 Facilities, Personnel, and Equipment, 36 References, 37 Briefings, 37 4 REPORT OF THE PANEL ON ENGINEERING FOR COMPLEX SYSTEMS 38 Introduction, 38 Review Process, 38 General Observations, 39 Programmatic Risk Management, 39 Technical Quality, 40 Challenge Areas, 41 Specific Task Discussions, 41 System Reasoning for Risk Management, 42 Connections to the External Community, 43 Research Portfolio, 44 People and Facilities, 44 Methodology, 45 Quality of Work, 45 Observations on Specific SRRM Tasks, 45 Knowledge Engineering for Safety and Success, 45 Human and Organizational Risk Management, 45 Knowledge Management, 47 Observations on Specific KESS Tasks, 47 Resilient Systems and Operations, 49 Intelligent and Adaptive Operations and Control, 49 Resilient Software Engineering, 49 Observations on Specific RSO Tasks, 51 References, 52 Briefings, 52