Internal NASA Study: NASA’s Protoflight Research Initiative Mary R. Coan PhD1,2, Steven R. Hirshorn1, and Robert Moreland1 1NASA Headquarters Office of the Chief Engineer, Washington D.C., 2NASA Kennedy Space Center, Florida Executive Summary The NASA Protoflight Research Initiative is an internal NASA study conducted within the Office of the Chief Engineer to better understand the use of Protoflight within NASA. Extensive literature reviews and interviews with key NASA members with experience in both robotic and human spaceflight missions has resulted in three main conclusions and two observations. The first conclusion is that NASA’s Protoflight method is not considered to be “prescriptive.” The current policies and guidance allows each Program/Project to tailor the Protoflight approach to better meet their needs, goals and objectives. Second, Risk Management plays a key role in implementation of the Protoflight approach. Any deviations from full qualification will be based on the level of acceptable risk with guidance found in NPR 8705.4. Finally, over the past decade (2004 – 2014) only 6% of NASA’s Protoflight missions and 6% of NASA’s Full qualification missions experienced a publicly disclosed mission failure. In other words, the data indicates that the Protoflight approach, in and of it itself, does not increase the mission risk of in-flight failure. The first observation is that it would be beneficial to document the decision making process on the implementation and use of Protoflight. The second observation is that If a Project/Program chooses to use the Protoflight approach with relevant heritage, it is extremely important that the Program/Project Manager ensures that the current project’s requirements falls within the heritage design, component, instrument and/or subsystem’s requirements for both the planned and operational use, and that the documentation of the relevant heritage is comprehensive, sufficient and the decision well documented. To further benefit/inform this study, a recommendation to perform a deep dive into 30 missions with accessible data on their testing/verification methodology and decision process to research the differences between Protoflight and Full Qualification missions’ Design Requirements and Verification & Validation (V&V) (without any impact or special request directly to the project). Introduction The NASA Protoflight Research Initiative is an internal NASA study conducted within the Office of the Chief Engineer to better understand the use of Protoflight within NASA. NASA has been using the term Protoflight since Goddard Space Flight Center (GSFC) coined the term in the 1960’s to “avoid overrunning budget limitations” along with other factors. The first designated Protoflight spacecraft was launched in November 1965 and was called the Direct Measurements Explorer (DME-A). According to the first General Environmental Test Specification for Spacecraft and Components (GSFC S-320-G-1) published in 1969, a Protoflight spacecraft was defined as “(1) either a spacecraft originally designated as a prototype and subjected to complete or partial design qualification environmental testing, and then designated for flight use or (2) a spacecraft designated in advance to serve both as a prototype and a flight model.” [1] Today’s definition, located in the General Environmental Verification Standard (GEVS) for GSFC Flight Programs and Projects (GSFC-STD-7000A), defines Protoflight as “flight hardware of a new design; it is subject to a qualification test program that combines elements of prototype and flight acceptance verification; that is, the application of design qualification test levels and flight acceptance test durations.” While the definitions of Protoflight have not varied much in the past 50 years, the use of Protoflight has significantly increased. From 1965-1974, a total of 16 Protoflight spacecraft were launched compared to a total of 49 Protoflight missions launched from 2004-2014. The increased use of the Protoflight approach for larger more complex spacecraft, subsystems and instruments causes one to ask the question, does using the Protoflight approach increase the likelihood of mission failures? To answer this question and others, the authors of this paper conducted an extensive literature review encompassing the current policy and standards used within NASA, the number and type of NASA missions flown in the past 20 years and the number of missions that were exclusively Protoflight or that used one or more Protoflight subsystems/instruments in the past 10 years. Given the vast amounts of information regarding this topic and the various deep dives that could take place, the authors restricted the research to three areas each with a stated purpose to properly scope the literature review. The three areas and their purpose are as follows: 1. Conduct a literary review of the current Policy and Standards at the Agency and Center levels for Protoflight programs and projects within NASA. Other US Agencies, foreign space programs, and companies’ policies/procedures/standards regarding Protoflight are not actively sought in this review. The purpose of the literary review is to understand what policies/standards are in place at the Agency and Center levels with a focus on when the Protoflight model began being used and the current guidance on the use of Protoflight models versus a full qualification model. 2. Collect and analyze data on NASA’s flight missions and projects that have flown in the past 10 years using the Protoflight methodology at the payload, subsystem and/or instrument level. The purpose of this data retrieval and analysis is to understand the program/project’s decision to use the Protoflight approach and the rate of failure (partial and/or full) tracing back to a reduced life time expectancy of the payload/subsystem/instrument due to the use of the Protoflight methodology. 3. Characterize the level of Protoflight verification and Risk Management techniques used by NASA’s Protoflight missions/projects that have flown in the past 20 years. The purpose of this characterization is to determine the various implementations of the Protoflight approach used in NASA’s Protoflight missions/projects and the effect of the Protoflight approach used on Risk Management (technical and programmatic). Current Protoflight Guidance An extensive literature review was conducted to locate and obtain NASA’s Agency and Center level documentation regarding Protoflight qualification standards, requirements, policies, and guidance. Public and non-public sources were used to locate documentation. Non-NASA documentation was not actively sought, however if non-NASA documentation was located and obtained it was included in this study. Table 1 shows the various documents obtained including the document number, title, published and expiration date (where applicable) and the responsible center/organization. The documents obtained cover a broad range of information regarding the Protoflight method including design levels, qualification levels for Protoflight, Prototype/Qualification and Acceptance, NASA policy regarding Risk Classification and the use of the Protoflight approach verification test requirements/matrix, see Table 2. Table 1. A list of the obtained NASA’s Agency and Center level Protoflight and/or ProtoQual Qualification Documentation including a Non-NASA sourced Department of Defense (DoD) Handbook including all applicable dates and responsible Center. Document No. Title Published Expires Center General Environmental Verification GSFC-STD-7000 Standard (GEVS) for Flight Programs 4/22/2013 GSFC REV A and Projects Class D Spacecraft Design and APR-8070.2 9/17/2008 9/17/2013 ARC Environmental Test General Environmental Test MSFC-HDBK-670 Guidelines (GETG) for Protoflight 6/1/1991 MSFC instruments and experiments NASA-STD-7002A Payload Test Requirements 9/10/2004 HQ NASA-STD-7003A Pyroshock Test Criteria 12/20/2011 HQ NASA-STD-7001A Payload Vibroacoustic Test Criteria 1/20/2011 HQ Structural Design and Test Factors of NASA-STD-5001B 8/6/2014 HQ Safety for Spaceflight Hardware Risk Classification for NASA NPR 8705.4 6/14/2004 6/14/2018 HQ Payloads (Updated w/change 3) Design, Construction and testing DOD-HDBK-343 requirements for one of a kind Space 2/1/1986 DOD (USAF) Equipment Structural Design Requirements and JSC-65828 REV A Factors for Spaceflight Hardware for 10/1/2011 JSC Human Spaceflight Qualification and Acceptance SSP 41172 REV U Environmental Test Requirements ISS 3/28/2003 JSC Program JPL Rules! DocID Spacecraft System Dynamic and Static 4/15/2013 JPL 55833 Rev 1 Testing, Rev. 1 JPL System Thermal Testing, Rev. 2 5/8/2013 JPL Table 2. A list of the obtained NASA’s Agency and Center level Protoflight Qualification Documentation including a Non-NASA sourced Department of Defense (DoD) Handbook including the subject matter covered in the document. (Yes means the topic was covered in the document, No means the topic was not covered in the document) The majority of the documents obtained give guidance in the form of technical standards, requirements, design criteria and/or test criteria without programmatic guidance. Some of the documents are NASA standards that pertain to specific technical areas such as pyro shocks and vibro acoustics. Others are aligned with the General Environmental Verification Standard (GEVS) for Flight Programs and Projects, GSFC-STD-7000, which covers a broad range of technical information. There is one key difference between the GEVS and other Center’s Protoflight documentation based on the GEVS. The GEVS states “The standard is written in accordance with the current GSFC practice of using a single Protoflight payload for both qualification testing and space flight… The Protoflight verification program, therefore, is given as the nominal test program.” In other words, Full Qualification is the exception and no longer the norm at Goddard Space Flight Center (GSFC). Only one document attempts to give guidance regarding the use of Protoflight through “recommended requirements” based on risk classification, NPR 8705.4 Risk Classification for NASA Payloads. NPR 8705.4 Appendix C - Recommended SMA-Related Program Requirements for NASA Class A-D Payloads, states the level of rigor recommended to qualify flight payloads characterized with a Class A, B, C or D risk classification. Currently GSFC and JPL use the Protoflight verification program as the nominal approach for non-heritage flight payloads with a risk classification of Class B – D, which agrees with the “recommended” requirements in NPR 8705.4 Appendix C. For missions with a risk classification of Class A NPR 8705.4 states “Full formal qualification and acceptance test programs and integrated end-to-end testing at all hardware and software levels.” However, the requirement is only “recommended” and does not require a waiver if a program/project does not comply. For an example, Mars Science Laboratory (MSL) a Category 1, Risk Class A Robotic Mission used the Protoflight approach on specific subsystems/instruments but not the entire payload. While a waiver was not required, the decision to use the Protoflight approach was acknowledged in MSL’s Key Decision Point (KDP) documentation. The Protoflight method within NASA is not considered to be “prescriptive.” The current policies and guidance allows each Program/Project to tailor the Protoflight approach to better meet their needs, goals and objectives. For example, some projects/missions use Engineering Test Units (ETUs) to test various subsystems/instruments prior to building the Protoflight Unit while others do not. Another example of variations between Protoflight projects is the decision to test or to use analysis to verify requirements during the qualification process. The decision process regarding the various implementation options of the Protoflight approach is based on several factors such as the heritage of subsystems/instruments, budget, schedule and risk posture of the project. The decision process is not required to be documented only agreed to by the decision making authorities. Protoflight Mission Failures in the Past 10 and 20 years Over the past 20 years, NASA has launched 215 missions varying from Category 3 Risk Class D missions all of the way to Category 1 Class A missions to Mars and launching humans into Lower Earth Orbit (LEO). Information regarding NASA’s missions over the last 20 years was gathered and reviewed for any links to the use of the Protoflight methodology at the spacecraft, payload, subsystem, and/or instrument level. In some cases, information was not readily available for review. If enough information was not accessible the mission was noted as such. All accessible mission related documentation was reviewed for the following information:  Mission Risk Classification  Qualification Method used (Protoflight, Full Qualification)  Launch Date  Current Phase (Operating, Operating/Extended, Past, Past/Extended  Government Agency and NASA Center involved  Why the Protoflight approach was chosen o Information regarding this question was not available With the following definitions for clarification: Operating Phase: A period of time in which the operational activities are carried out to meet the goals of the mission (Operational Phase) Operating/Extended Phase: A period of time after the planned operational activities have been carried out and additional operational activities are being carried out (Extended Operational Phase) Past Phase: The period of time after the mission has been decommissioned according to plan or because of a significant failure and without the mission entering into an Extended Operational Phase Past/Extended Phase: The period of time after the mission has been decommissioned the mission entering into an Extended Operational Phase The authors categorized a mission’s qualification method as Protoflight only if all of the below Protoflight criteria were met. Protoflight Criteria 1. Mission spacecraft, subsystems and/or instruments were launched and operated in space – For STS missions, the only missions included are those that are external and fixed to the ISS or that are released into space 2. A known Protoflight qualified spacecraft hub, payload, subsystem and/or instrument is on board – If specific spacecraft, subsystem, or instrument information is unavailable but Risk Mission Class is known to be B, C, or D, the mission is automatically assumed to be Protoflight based on NPR 8705.4 Appendix C • Class A missions are automatically assumed to have gone through Full Qualification based on NPR 8705.4 Appendix C 3. The above criteria must have valid references, which include: – NASA personnel who worked the mission communicating via a face to face meeting, telephone call or email – NASA published (internal or external) documentation (NASA website, NASA Case Study, NASA Lessons Learned and/or NASA personnel author of a scientific/engineering journal article) Once a complete list of all of NASA’s missions over the past 20 years was collected and documented, the launch date, current phase, risk classification, qualification method and Government Agency/Center involved was determined through literature reviews of public and private accessible documentation. Documentation regarding the risk classification and qualification method used was not readily accessible for missions launched between 1994 and 2004, resulting in a data set too small for high fidelity analysis. While a large amount of information regarding NASA missions from 1994 – 2004 was inaccessible, the data that was collected is included for the reader’s edification. Information regarding launched NASA missions during the last 10 years, 2004 through 2014, was readily available and allowed for a large enough set of information for analysis. For an example, Table 3 shows the total number of missions launched from 1994 – 2014 as 215 missions with only 110 missions having enough accessible information to determine the qualification method used. In the last decade alone, 2004-2014, 83 out of the 84 NASA missions launched had enough accessible information to determine the qualification method used. Table 4 is an example of the collected mission information. Information on the launched Protoflight missions within the last 20 years (1994-2014) was collected and reviewed for mission failures with and without direct links to the Protoflight qualification method. In this paper and in accordance with NPR 8621.1B NASA Procedural Requirements for Mishap and Close Call Reporting, Investigating, and Recordkeeping, a “NASA mishap is an unplanned event that results in NASA Mission Failure before the scheduled completion of the planned primary mission. Where a NASA Mission Failure is a mishap of whatever intrinsic severity that prevents the achievement of the mission's minimum success criteria or minimum mission objectives as described in the mission operations report or equivalent document. (Note: A mission failure applies only to a NASA program's mission, and not a test or ongoing institutional operation. If a program accomplishes all minimum success criteria but not "full mission objectives," it is not a mission failure (even though in some cases it may appropriately be classified and investigated as a close call).” If a mission failure occurred a Failure Review Board (FRB) and/or Mishap Investigation Board (MIB) was created and a report released describing the failure/mishap and known and/or suspected cause. These reports were used to determine if a mission failure/mishap occurred on a Protoflight mission and the origin of said failure/mishap. If a mishap was found to occur on a Protoflight mission the data was collected and the report reviewed to determine any links to the Protoflight qualification method. When a mission failure/mishap occurred during an “Extended Operations” phase, the mission failure/mishap was not counted and the mission is only considered to be a “Past (extended) Mission.” Table 3 shows that there were 20 mission failures for missions launched from 1994 – 2014 with only 5 of those mission failures on missions launched in the past decade (2004-2014). Table 3. Overview of Mission information collected for NASA Missions launched from 1994 – 2014 and 2004 – 2014 including Protoflight (PF) and Full Qualification Mission information and mission failures. Table 4. Example of Mission information collected for NASA Missions launched from 1994 – 2014 including Protoflight and Full Qualification subsystem information, risk mission classification, launch dates, current phase and responsible government agency and or NASA Center.