Verification and Validation Plan for Flight Performance Requirements on the CEV Parachute Assembly System Aaron L Morris1 and Leah M. Olson2 Engineering Science Contract Group (ESCG), Houston, TX, 77598 The Crew Exploration Vehicle Parachute Assembly System (CPAS) is engaged in a multi-year design and test campaign aimed at qualifying a parachute recovery system for human use on the Orion Spacecraft. Orion has parachute flight performance requirements that will ultimately be verified through the use of Monte Carlo multi-degree of freedom flight simulations. These simulations will be anchored by real world flight test data and iteratively improved to provide a closer approximation to the real physics observed in the inherently chaotic inflation and steady state flight of the CPAS parachutes. This paper will examine the processes necessary to verify the flight performance requirements of the human rated spacecraft. The focus will be on the requirements verification and model validation planned on CPAS. Nomenclature CEV = Crew Exploration Vehicle CM = Crew Module CPAS = Crew Exploration Vehicle Parachute Assembly System DOE = Design of Experiments DOF = Degrees of Freedom DSS = Decelerator Systems Simulation ESCG = Engineering and Science Contract Group Gen = Generation GNC = Guidance, Navigation, and Control IPT = Integrated Product Team M&S = Modeling & Simulation NASA = National Aeronautics and Space Administration OFT-1 = Orion Flight Test 1 PTRS = Project Technical Requirements Specification RCS = Reaction Control System SE&I = Systems Engineering and Integration STD = Documentation Standard V&V = Verification and Validation I. Introduction T HE Crew Exploration Vehicle (CEV) Parachute Assembly System (CPAS) project flight performance requirements will be verified via analysis. Therefore, the simulations used to perform the analyses must accurately depict a physical environment similar to what the CEV will experience during flight. This is achieved by acquiring high fidelity data from multiple flight tests and refining model parameters based on the physics observed. After each flight, the simulation and modeling tools are incrementally refined to model phenomena observed in flight tests. However, not every aspect of the physical environment is being modeled because current models are 1 CPAS Analysis IPT Chair, Aerothermal and Flight Mechanics, 2222 Bay Area Blvd, Houston, Webster, TX, AIAA Senior Member. 2 CPAS Analysis Engineer, Aerothermal and Flight Mechanics, 2222 Bay Area Blvd, Houston, Webster, TX. 1 American Institute of Aeronautics and Astronautics based on simplified heritage models. In addition, current state of the art modeling capabilities are limited and cannot completely simulate the chaotic aerodynamic flow and parachute interaction events. Statistical techniques, including Design of Experiments (DOE)1,2 and Monte Carlo3 analyses will be implemented to understand the sensitivities of parachute performance parameters and to ensure the physical characteristics not modeled are taken into consideration via error distributions. The analytical approach for verifying performance parameters is unique when applied to a human rated parachute system. Many aspects of the system, such as the deployment sequence, cannot be modeled reliably and requires numerous test demonstrations to improve understanding and anchor current deployment models. It is impractical to conduct flight tests throughout the operational envelope; therefore, the flight performance requirements must be verified using the aforementioned simulations. The flight performance measurements, such as rate of descent and inflation loads imparted to the structure, can be modeled consistently using 6-Degree-of- Freedom (6-DOF) flight simulations such as DSSA and DSS. It is of utmost importance that the stakeholders responsible for verifying flight performance requirements have confidence in the simulation results to make an informed decision regarding the satisfactory verification of requirements. DOE studies and NASA standards for model development and use, such as NASA Technical Standard “Standard for Models and Simulations” (NASA- STD-7009)4, help to increase credibility of the simulations and therefore instill confidence in their results. II. Description of Requirements CPAS intends to analytically verify three categories of requirements. These requirement categories include terminal rate of descent, parachute loads, and torque required to reorient the vehicle prior to touchdown. These categories are represented by total of seven requirements. The requirements that must be verified are documented in the Project Technical Requirements Specification (PTRS) for CPAS (JSC-63497 Rev B)5 and are summarized in the subsequent sections. A. Terminal Rate of Descent Requirements The terminal Rate of Descent (ROD) is the velocity of Orion upon touchdown. Table 1 lists the two requirements that pertain to ROD. The first requires the touchdown velocity be less than 33.0 ft/s with a maximum Orion weight of 20,865 lbm. The second requires Orion to reach the 33.0 ft/s velocity at an altitude greater than 200 ft above mean sea level. As stated in the Rationale, the terminal ROD only applies in conditions with no wind. Also, these requirements must be applied to all potential failure scenarios. Table 1. Terminal rate of descent requirements from the CPAS PTRS Rev B. Requirement Requirement Text Rationale Title & Number CM Standard Day CPAS shall limit the terminal vertical This requirement applies in calm air (no wind). The Maximum descent rate of the CM to less than system-level landing analysis, performed by Lockheed Terminal Rate of 33.0 ft/s (10.07 m/s) at standard sea- Martin, will account for atmospheric dynamics including Descent level conditions (as defined in NASA- vertical and horizontal winds, elevation, temperature TM-X-74335, U.S. Standard effects, etc., and should be verified for all fault conditions Atmosphere, 1976) for a maximum defined in. The mass specified above is the total mass of CM mass of 20,865.0 lbm the CM, inclusive of the CPAS mass, at the time of drogue (9,464.2 kg). deployment. Minimum CPAS main parachutes shall reach the This requirement assumes calm air (no wind). While the Altitude for CM Standard Day Maximum Terminal CM nominally lands at zero feet MSL, this additional maximum Rate of Descent at no lower than 200 ft altitude is included to account for atmospheric variations Terminal Rate of (61.0 m) MSL when measured from (i.e. the “nonstandard day”). This requirement must be met Descent the CM center of gravity under in the presence of failures for nominal landings and aborts. standard atmospheric conditions (as defined in NASA-TM-X-74335, U.S. Standard Atmosphere, 1976). 2 American Institute of Aeronautics and Astronautics B. Parachute Load Requirements Quantifying parachute loads is critical to ensure human survival on Orion. Loads which act on the parachute canopy and risers must be bounded for the nominal flight conditions as well as all potential failure modes. An example loads trace of a nominal extraction (first peak), two Drogues, three Mains parachute system under nominal descent is shown in Figure 1. This figure shows that under a nominal descent, the highest loads occur at the inflation and disreefing stages. However, Figure 1. Example peak loads occurring at inflation and disreef stages the highest loads could occur during a nominal descent. at a different stage during Table 2. Parachute inflation and disreef load requirements from the CPAS PTRS Rev B. Requirement Requirement Text Rationale Title & Number Peak Drogue CPAS shall limit the total peak drogue parachute See Table 4. Loads cluster load to less than 67,300 lbf (299,400 N) under all failure conditions specified herein, including any one skipped drogue reefing stage, with the probabilities specified in Table 3.3-1, Simulated Parachute Load Distributions. Peak Main Loads CPAS shall limit the total peak main parachute cluster See Table 4. load to less than 89,700 lbf (399,000 N) under all failure conditions specified herein, including any one skipped main reefing stage, with the probabilities specified in Table 3.3-1, Simulated Parachute Load Distributions. The single riser load drives part of the design Peak Single CPAS shall limit the peak load on a single drogue of the CM attach points. CPAS must limit the Drogue Riser riser to less than 50,500 lbf (224,600 N) under all loads on these attach points in order to Loads failure conditions specified herein, including any one prevent catastrophic failure of the attach skipped reefing stage, with the probabilities specified points. The peak single drogue riser load is in Table 3.3-1, Simulated Parachute Load 75% of the peak drogue cluster load specified Distributions. in See Table 4. The single riser load drives part of the design Peak Single Main CPAS shall limit the peak load on a single main riser of the CM attach points. CPAS must limit the Riser Loads to less than 67,300 lbf (299,400 N) under all failure loads on these attach points in order to conditions specified herein, including any one skipped prevent catastrophic failure of the attach reefing stage, with the probabilities specified in Table points. The peak single main riser load is 75% 3.3-1, Simulated Parachute Load Distributions. of the peak main cluster load, as specified in Table 4. failure. 3 American Institute of Aeronautics and Astronautics The applicable requirements are broken into peak cluster and individual riser loads on the Drogue and Main parachutes. The load requirements for the Drogue phase, state that during each reefed stage, the cluster of two parachutes must not exceed a load higher than 67,300 lbf and a single riser must not exceed 50,500 lbf. The requirement for the Main phase restrict riser and cluster peak loads from exceeding 67,300 lbf and 89,700 lbf, respectively. The load requirements in Table 2 include all failure scenarios. (Each loads requirement calls out Table 3.3-1 in the CPAS PTRS; this table can be seen below as Table 4 in this document.) C. Torque Requirement Due to environmental conditions including wind gusts, the random aerodynamic flow, and parachute cluster interactions, Orion and CPAS may rotate and cause a torque which needs to be modeled time-accurately. Table 3 presents the formal Table 3. Torque requirement from the CPAS PTRS Rev B. wording and Rationale of the Requirement Requirement Text Rationale requirement as seen Title & Number in the CPAS PTRS Rotation Torque CPAS shall limit the torque The CM is designed to land in a feet-forward Rev. B. The torque Limit required to rotate the CM orientation. The CM Reaction Control System requirement flows about the gravity vector (RCS) must be able to overcome the parachute during the terminal descent harness/riser torque. The thrust and mass of down from parent phase to less than 450 ft∙lbf propellant to use for landing orientation is (610 N∙m) when the total limited. The Guidance, Navigation, and Control twist angle between the (GNC) system is responsible for not only vehicle and canopies is less nulling any rotation rate built up in the system, than 1,800 <TBR-497-007> but also “untwisting” the risers into the range degrees. of zero to five <TBR-497-007> total twists (0- 1,800 <TBR-497-007> degrees). requirements stating that the crew must land in a particular orientation. This requires integration between the vehicle subsystems including GNC and CPAS and currently the Figure 2. Ground program is test of twisted moving Table 4. Simulated parachute load distributions (PTRS Table 3.3- parachute riser towards an 1). lines. integrated performance by establishing the performance of CPAS. Ground tests such as the torque test depicted in Figure 2 help bound the amount of torque expected during flight. Risers in the configuration shown can create torque on the Orion vehicle or parachute system. If the torque seen during ground testing is within the GNC subsystem’s ability to control, no extra hardware is necessary on CPAS. D. Requirement Probability for Verification Since not every aspect of the physical environment can be modeled due to random aerodynamic flow characteristics and parachute interactions, the requirements must be verified through the use of Monte Carlo simulations and statistical interpretations of the resulting data. 4 American Institute of Aeronautics and Astronautics This necessitates that the requirements include a probabilistic measure for verification purposes. Table 4 presents the necessary probabilities to verify the load requirements with a 50% confidence. The ROD and Torque requirements must be met with a 99.87% probability and a 90% confidence, which will be included in the next revision of the Validation & Verification Document6. III. Description of Modeling Tools The Decelerator Systems Simulation (DSS)7,8 is the primary tool that will be used to verify requirements through analysis. It is a legacy 6 Degree-of-Freedom (DOF) parachute trajectory simulation based on the UD233A simulation used by the Space Shuttle Solid Rocket Booster parachute project. DSS is the highest fidelity NASA- maintained simulation used by the CPAS project. This simulation tool provides high fidelity results for predicting inflation loads, disreef loads, and terminal rate of descent; and has been proven by flight test reconstructions performed on X-38, Orion Pad Abort Demonstrator, and recent CPAS flight tests. In addition to flight data reconstructions performed using DSS, the CPAS project also uses off the shelf software to assess the statistical implications of measured rate of descent9 and torque models. At the time of this paper, a simpler, stand-alone inflation model is also being considered. A simple tool has the advantage of greatly reducing the amount of time necessary to implement test cases. A standard Monte Carlo involves hundreds or thousands of case runs with an integrated vehicle configuration. With the current tool, this consumes a large amount of time and computing power. Without decreasing the fidelity of results, a simple simulation tool would significantly reduce run time by avoiding the use of a fully integrated vehicle trajectory. IV. Process to Determine Sensitivity of Parameters The CPAS project is using DOE to determine which parachute model parameters are most sensitive to affecting loads and rate of descent outputs.10 Conducting a DOE study also identifies important parameter interactions. These techniques are applied to inflation and disreef loading models and for Main parachute full open rate of descent models. Once the most sensitive model parameters are determined, focused testing and measurements during flight tests will improve the determination of those sensitive factors and their dispersions. The details of how these factors are applied on CPAS are discussed in the CPAS performance modeling paper11. The process of this sensitivity study involves 1) developing benchmark cases that are representative of typical CPAS flight tests, 2) generating additional benchmark cases that are representative of typical Orion entry and abort trajectories, 3) choosing the “factors” or parachute parameters to vary, 4) creating and simulating a specific number of cases based on the factors and levels selected (a more comprehensive description of factors and levels may be found in an introductory DOE text1,2) and 5) interpreting the resulting statistical output. This process is illustrated in Figure 3. Once the parameter significance and interactions have been determined, the effects of the parameter dispersions will be investigated. Preliminary DOE studies indicate that several of the parachute parameters that are currently modeled separately are actually interrelated. This finding may lead to a reexamination of how these parameters are modeled, which will likely help improve and validate the simulations to make them more representative of real- world conditions. In turn, the improved simulations could be used to verify Figure 3. Process of Assessing Sensitivity of Parachute requirements by analysis with a higher Model Parameters degree of certainty. V. Model Credibility Score In order to determine whether or not a requirement is verified via analysis, it is imperative for decision makers to understand the credibility of the model used for analysis. NASA-STD-70094, which was developed following the Columbia Space Shuttle accident, aids in communicating to decision makers the fidelity and credibility of an analysis. It dictates standards to which simulation developers and users must adhere to in order to ensure the simulation tools are credible for making decisions related to human-rated spacecraft. These standards define a level of documentation during simulation development, training and education of the simulation users, and an understanding of the criticality of decisions that are to be made with the simulation. There are three categories: 5 American Institute of Aeronautics and Astronautics Modeling & Simulation (M&S) Development, M&S Operations, and Supporting Evidence, each of which has a few factors as shown in Figure 4. Each of these factors is assessed and a model and simulation credibility score may be assigned so that stakeholders understand the level of credibility of the model when making decisions. Using the guidelines outlined in Figure 4. Model Credibility Scoring Guidelines (from NASA Standard 7009)., the CPAS project performed an assessment of the credibility of DSS. Ultimately, DSS scored a 3.125/4.0 on the Model Credibility Assessment. This score is above average because though parachute dynamics are difficult to predict, the sensitivity of many parachute parameters are known, the DSS algorithm is able to process the same inputs in multiple ways which creates a nondeterministic analysis, and the DSS results as well as inputs agree with flight test data. All these factors and a handful of others provide a high score on NASA’s Model and Credibility Scoring Guidelines. Figure 5 summarize the scores for each factor. The rationale for each individual score is given Figure 4. Model Credibility Scoring Guidelines (from NASA Standard 7009). below. A. Verification – Received a score of 3.0/4.0. Due to flight test data, and as a result of the DOE study, formal numerical error estimation is possible. The results of the simulations will be assessed to provide 90% tolerance limits on the performance values, as required by the PTRS. B. Validation – Received a score of 3.0/4.0. Results agree with real world data for problems of interest. The project is focused on reconstructing observed flight test observations using the models. The use of the models to predict performance prior to the test, and to reconstruct the test results using observed environmental factors will provide validation of the models. C. Input Pedigree – Received a score of 3.0/4.0. Input data agrees with real world for problems of interest. Flight test data is being collected using calibrated, redundant data collection systems. This provides both information regarding level and variability that will be used in the model. For simulation purposes, the input data will be dispersed based on the flight test data collected. 6 American Institute of Aeronautics and Astronautics D. Results Uncertainty – Received a score of 3.0/4.0. Uncertainty results will be quantified with numerical analysis. Appropriate statistical analyses of the results will be provided to quantitatively define the results in a probabilistic manner. The requirements indicate that performance values must be expressed in terms of tolerance intervals (the probability CPAS Flight Simulation Model  that the population does not exceed a specified probability Credibility Score with a defined confidence). This 4 4 factor may be upgraded to 4.0/4.0 after the completion of this effort. 3.5 3 3 3 3 3 3 3 3 E. Rsceosruel tos f R3o.0b/u4s.t0n.e sTsh e– DROecEe ivsetudd ya e (0‐4) 2.5 to determine the parameter cor 2 S significance will identify the or  1.5 ct statistically significant parameters a F 1 and further evaluation of the 0.5 parameter dispersions will provide a quantitative 0 understanding of the sensitivity of the model to the parameters. The actual values of the parameters, and their dispersions, will be measured by ground and flight tests. This factor may be upgraded Model Assessment Factors to 4.0/4.0 after the completion of Figure 5. CPAS Model Credibility Assessment Score per this effort. Category (Current as of January 2011). F. Use History – Received a score of 3.0/4.0. The DSS tool has been used to make preflight predictions for parachute tests on a variety of programs including X-38, PAD, and CPAS. Results have compared favorably with mission data. G. M&S Management – Received a score of 3.0/4.0. Model improvements are tracked and under configuration control. This score may be upgraded to a 4.0/4.0 when more detailed documentation is created. H. People Qualifications – Received a score of 4.0/4.0. Each engineer associated with software updates and improvements has an advanced degree and multiple years of experience. The primary software author is the original architect of the system. Each senior analyst that reports progress meeting requirements has an advanced degree and multiple years of experience. Each set of pre-flight test predictions is generated by an experienced analyst, many of whom have advanced degrees. VI. Simulations Anchored to Flight Tests The credibility of CPAS models has a strong foundation in flight testing. Three generations of design and test flights (Gen I, Gen II, and Gen III) are being conducted and flight test results and reconstructions are used to increase confidence in the DSS and CPAS parachute models. Testing and model validation methods form the foundation for the work that must be done in the qualification phase of testing. “Run for the record” simulations will be performed at the end of qualification testing, and these simulations will ultimately be used to verify CPAS performance requirements. That is to say, a final assessment of the CPAS flight performance requirements will be performed at the end of all engineering development and qualification testing. Therefore, it is of critical importance to achieve valid and representative reconstructions of flight test data. The exactness with which real-world results are represented by the simulations is fundamental in demonstrating a reliable analysis to decision makers. Other work by the authors of this paper may be reviewed to gain a thorough understanding of parachute reconstruction methods used on the CPAS project.11 In order to develop and improve a model of CPAS parachutes, data from drop tests must be processed and the tests reconstructed in the simulations. On each test, a variety of instrumentation measures the necessary data: position and velocity, accelerations, riser and harness loads, atmospheric conditions, vehicle mass properties, and 7 American Institute of Aeronautics and Astronautics photogrammetry. To begin the reconstruction process, the flight test data are checked for validity and accuracy. Trajectory, winds, and atmospheric data are processed into “best estimate” files. These, along with the parachute loads and accelerometer data, provide the basis of the reconstruction. Vehicle mass properties and measurements are used to create an input file for DSS. Parameters from the latest modeling memo12 are used as a initial values for the inflation parameters of the test parachutes. Drag coefficients and reefing ratios are calculated from the flight data. All input parameters are checked by co-plotting the pre-flight simulation outputs with the flight data. Parameters are then changed iteratively until a best fit is found. A step-by- step example case follows. Note that the below figures are idealized representations and not actual flight data. It is important to note that the tests which have been performed have not used the actual Orion vehicle shape or architecture. Further information on this can be found in the AIAA paper titled “Challenges of CPAS Flight Testing”.13 Some of the elements specific to the testing techniques, such as the extraction of the test payload from an aircraft, are not modeled in DSS. To account for this, adjustments must be made during the reconstruction to ensure the correct conditions at the deployment of each parachute. This ensures the simulation will achieve the correct dynamic pressure and vehicle dynamics, factors that are vital in calculating the correct parachute loads The first step in reconstructing the performance of any parachute in the deployment sequence is to determine the steady-state drag coefficient. The output of the simulation using the calculated drag coefficient must fall in line with the flight drag area and dynamic pressure. If it does not, the drag coefficient must be adjusted. As shown in Figure 6, the drag coefficient is increased to raise the drag area and lower the dynamic pressure. Figure 6. Drag Coefficient Correction: Drag Area & Dynamic Pressure Effects Once steady state characteristics have been determined, the second step is correct the initial conditions. Figure 7 shows the initial dynamic pressure of the simulation being lowered to match the flight test data. The final step is to vary the inflation parameters: fill constant (n) over-inflation factor (C ), and opening profile exponent, expopen. k Each parameter is varied individually to match either the drag area or loads as shown in Figure 8. In many cases, all flight data traces cannot be matched using the same inflation parameters. Engineering judgment is used to determine a “best fit” set of parameters, based primarily on the accuracy of the data that is being matched and the Figure 7. Initial Condition Correction: Dynamic known deficiencies in the simulation being used. Pressure. 8 American Institute of Aeronautics and Astronautics This process is repeated for each stage of the parachute, and then for the rest of the test parachutes in the sequence. Figure 8. Best Fit Inflation Parameter: Drag Area & Parachute Loads. VII. Description of the Verification Cycle To verify that the CPAS system meets CEV flight performance requirements, the parachute parameters are compiled and iteratively compared against parachute performance requirements and integrated Orion requirements. Figure 9 graphical depicts these iteration cycles. First, an initial parachute system is conceptually designed (1), tested (2), and then data is gathered from the tests (3). Next, the CPAS Analysis Integrated Product Team (IPT) publishes a quarterly modeling memo which summarizes the parachute model parameters representing the observed test performance and the associated uncertainties of these parameters after each set of flight test reconstructions (4). The data from this memo is tested in benchmark simulation cases to determine if CPAS performance requirements are met and overall performance is tracked (5). If the CPAS requirements are not met, either the requirements documents or the CPAS design is refined (5a). Once the benchmark simulation study is complete, the memo is transmitted to groups responsible for Orion vehicle modeling and simulation such that integrated vehicle trajectories and performance may be evaluated and performance requirements may be assessed (6). Then, integrated flight tests are conducted to certify the integrated CPAS and Orion systems are acceptable for human flight (7). These integrated flights tests will include unmanned and manned tests such as Orion Flight Test 1 (OFT-1) and qualification tests. Lastly, a final set of simulations will be run (8) which will verify and formally document the vehicle for its intended operations. Figure 9. Parachute Performance Validation Data Flow 9 American Institute of Aeronautics and Astronautics VIII. Conclusion Flight testing of parachutes is an expensive and time consuming endeavor. It is impractical to test every flight condition and situation that a spacecraft recovery system is expected to encounter. Additionally, it is impossible to control all of the external environmental factors, or to provide exact replications of tests; therefore, it is necessary to verify some requirements through the use of flight simulation models. It is imperative that the decision makers involved are confident with the results of these simulation models. In the case of Orion parachutes, CPAS is using the DSS simulation to verify loads and rate of descent requirements, and an independent model to verify the torque requirement. DSS currently has a model credibility score of 3.125/4.0. This rating does not include the non-rated torque model. Efforts are underway to further quantify uncertainty and robustness of the results, which will likely lead to an increased score at the end of CPAS Gen III testing. In large part, the credibility of the DSS lies in the fact that it has been used to reconstruct numerous flight tests on several different NASA projects. Acknowledgments The authors wish to acknowledge Dave Wadsworth and Jim McMichael of the CPAS Systems Engineering and Integration (SE&I) IPT for their assistance and patience regarding DOE techniques and model credibility scoring. The authors also wish to acknowledge Koki Machin and Carol Evans of CPAS for discussions regarding requirements verification and human rating of spacecraft recovery systems. Peter Schulte of the University of Texas at Austin conducted a DOE study and provided valuable input in regard to verification, and Stephanie Shelton of the University of Alabama-Huntsville provided beneficial feedback and assistance formatting this paper. 10 American Institute of Aeronautics and Astronautics