06ICES-204 System Engineering and Integration of Controls for Advanced Life Support David Overland NASA Johnson Space Center Dr. Karlene Hoo Texas Tech University Marvin Ciskowski Hamilton Standard ABSTRACT of the investigation, the issues encountered, some tentative conclusions, and the direction expected for The Advanced Integration Matrix (AIM) project at the further research. Johnson Space Center (JSC) was chartered to study and solve systems-level integration issues for INTRODUCTION exploration missions. One of the first issues identified was an inability to conduct trade studies on control There are subsystem requirements above and beyond system architectures due to the absence of mature the functional requirements of a subsystem. These evaluation criteria. Such architectures are necessary to requirements derive from the need to integrate enable integration of regenerative life support systems. subsystems into an operational vehicle, and integrate A team was formed to address issues concerning that vehicle into an operational program. When there software and hardware architectures and system are dependencies between subsystems, functional controls.. requirements and interface descriptions are insufficient to constrain intersystem dynamics. This is the driving The team has investigated what is required to integrate rationale behind system engineering. controls for the types of non-linear dynamic systems encountered in advanced life support. To this end, a The more dynamic or autonomous subsystem activities water processing bioreactor testbed is being developed are, the greater the need for system engineering which will enable prototyping and testing of integration analysis and design methods. Even simple components strategies and technologies. Although systems such as can provide unexpected interactions when arranged the water bioreactors exhibit the complexities of improperly. Much of practical engineering experience is interactions between control schemes most vividly, it is geared to recognizing these hidden interactions and apparent that this behavior and its attendant risks will developing design approaches to eliminate or at least manifest itself among any set of interdependent mitigate the risks. autonomous control systems. A methodology for developing integration requirements for interdependent Control of non-linear, time-varying dynamic processes and autonomous systems is a goal of this team and this offer some of the most difficult system engineering testbed. challenges. Control of the Water Recovery Subsystem bioreactor preprocessors is an illustrative example of This paper is a high-level summary of the current status this. System engineering of the control system requires of the Air Revitalization Subsystem (ARS) from another understanding the system in context, to determine what project, and a scenario designed to elicit onboard and architecture and infrastructure is needed, and to ground-based task capture. Human factors will be understand the operational dependencies of the instrumental in defining the approach to task capture systems. Our investigation has developed some and analysis tools and methods. conclusions about the type of testing, modeling and analysis required to determine that context. Observing and documenting the controls design methodology is as much a part of the test as data Integrated Testing collected during the actual integrated test. Scenario development, task capture, operations concepts and The importance of integrated testing is historically well- data flow management prior to the test run itself, will known. The importance of designing for integration is also generate test products. not always as explicitly understood, especially for complex or interdependent systems. By observing the decision points associated with each task, the data required for decisions, the origin and data- The development methodology of stand-alone systems flow paths associated with that data, and its latency and is relatively well understood. But when you put systems reliability, the initial capabilities of the control together, interesting and unpredictable things can architecture can be bounded. happen. Simplistically put, the integrated system behaves differently than the stand-alone systems Using task analysis based on operational scenarios as behave separately. On previous integrated testing the framing presentation allows investigation into projects, integrating of Advanced Life Support (ALS) process and methodology, and into separation of subsystems has always driven out requirements and concerns between requirements and design. identified technology gaps in the ALS program [1]. The following components were initially proposed as test Based on that experience, both the Advanced articles: Integration Matrix program and its parent, the Bioastronautics program, have identified control system • Controls Investigation architecture and integration as a critical technology gap • Water Recovery Subsystem (WRS) for exploration missions [2]. As a result of an internal preprocessor systems with independent control working group analysis [3] and a NASA workshop [4], it systems for each reactor was proposed that an integrated test be developed to • Aerobic bioreactor explore the design constraints and integration • Anoxic bioreactor requirements of ALS control systems. • ARS simulation • Scenario development and Task analysis Test Objectives • Mapping command and data flows to capabilities AIM Test is intended to: • Narrative Integration • Stress interfaces Methodology • Identify information flows • Explore operations concepts and dependencies The investigation up to this point has uncovered several • Investigate architecture capabilities and methodologies that may provide assistance in managing requirements development of complex systems. Structured approaches to developing system requirements have In order to characterize the architecture requirements, it been used by the Department of Defense (DoD) to is important to determine the capabilities needed during identify and manage coupling between complex a mission, particularly to determine what types of data systems, of which our test is a microcosm. and autonomous capabilities will be required by crew, vehicle and ground control during complex mission The Joint Capabilities Integration and Development scenarios. System (JCIDS) [5] details a system engineering methodology that enables discovery and capture of root Test Components system requirements by first identifying capabilities required to support the program operations concept. Because of the scope of the investigation, development Those capabilities can include organizational and testing of the controls of ALS systems is only one infrastructure, manpower, logistics support, and component of the integrated test. In order to determine technology products. The JCIDS methodology allows top-level architecture requirements, the test was initially evaluation of an architecture’s ability to provide proposed as an exploratory test. Because of this, the identified capabilities, and allows technology gap test is in two parts. The first part of the test was analysis. scenario-driven, involving two bio-reactors, a simulation The JCIDS methodology influenced the decision to use Basic control engineering identifies three prerequisites capability definition as the starting point of this are required to develop a process control system. First, investigation. From capabilities identified as needed the process must be steady-state stable. Second, the during the scenario it is intended to capture what process must be controllable; i.e., there must be decisions must be made, where are they made, what controlled (dependent) parameters and manipulated information is needed to make those decisions, how (independent) parameters. Third, the process must be does the information get there, and the reliability of the observable; i.e., there must be observable parameters information. These need to be determined to identify that correspond to the controlled parameters. If these whether infrastructure and architecture can provide three conditions are met, then there are straightforward necessary mission capabilities. This is the beginning techniques available for controlling the process. If not, point for control architecture engineering comparison. other techniques must be brought to bear, including redesign of the system. Controls Investigation Two things need to be pointed out here. First, these Most of the efforts over the past year have focused on three prerequisites are design-dependent. If they are the controls investigation. This has primarily been due not met, additional control points, sensors or fluid to delays in the construction of the second bioreactor. pathways can be added to the system design to increase The importance of the work done so far cannot be the number of degrees of freedom to satisfy the underestimated. conditions. Even though the decision to investigate the WRS Second, control in the engineering sense refers to preprocessor bioreactors was imposed on the test plan, maintaining the output of the system within the range of the problem has illuminated a number of understandings desired output in the face of perturbations of the input that would not have been uncovered otherwise. Some values, of environmental (unmeasured) perturbations, are specific to the particular systems under and of variations in process dynamics. This is investigation; others are generically applicable to many accomplished by controlling various equipment other systems that will be needed for long-term (switches, valves, pumps), but component equipment exploration missions. control in and of itself does not equate to process control. The general question that sparked the initial investigation is: What requirements must be levied on Controls Models each subsystem to enable integration of the control systems? The investigation proposes that the two In order to determine whether these prerequisites are bioreactors can “stand in” for any two interdependent met, several types of modeling and analysis must be subsystems, e.g. ARS and WRS. performed. In this case, three types of models were required. The question is important because flight systems are developed independently by separate subcontractor Initially, in order to characterize the process at all, a organizations, often at different times in the program. stochiometric model had to be developed [8]. The basic Based on the traditional avionics control system chemistry of biological systems is inherently complex; development methodology, subsystems are designed however much of chemical engineering is initially based according to independently developed System on empirical equations which were available in the Requirements Specifications (SRS) with tightly literature. controlled Interface Requirements Definitions (IRD). It is assumed that subsystems can be controlled After the basic stochiometry was worked out, an independently of each other, and with few exceptions, it equilibrium model was developed which models the is assumed that software control requirements are dynamics of the system in equilibrium. This determines derived from hardware design specifications [6]. whether the first prerequisite is met: the process must be stable, requiring no controls to maintain equilibrium It became obvious early on in the controls investigation in the absence of system perturbations. The that the traditional approach to developing control mathematical techniques to determine stability are well- software for life support subsystems was inadequate to understood. The details of the model are described the task. Review of previous ALS control development elsewhere [9], and contains reaction information and efforts [7] showed that the process was not being substrate transport information. The model is a system controlled, only the test equipment. of partial differential equations with appropriate boundary conditions. The development approach to controlling the bioreactors that was adopted for this investigation is based on control engineering methods employed in the process Data generated by this model had to be compared control industry. against actual experimental data, in order to validate several assumptions in the theoretical model. This was Controls Development Process done using existing WRS testbeds at Texas Tech University. controllable and fully observable. Using a Proportional Integral Derivative (PID) controller on the Texas Tech Once stability has been theoretically established and University testbed confirmed this result. The team is in validated, a controls-relevant model has to be the process of redesigning the tubular reactor to add developed. Controls-relevant modeling is not as additional degrees of freedom. Instead of a PID straightforward as the previous two models. Relevance controller, a Model Predictive Controller is being in an engineering sense is dependent on the developed. optimization criteria chosen. Choices include performance efficiency, robustness to perturbations, rate Because the WRS laboratory has different test of processing, efficiency of processing, amount (or type) objectives, the redesigned system is being developed in of by-products, upstream or downstream constraints, the Bioastronautics Laboratory at JSC. maintenance frequency, cost of operation, etc. There are a number of mature, robust techniques to optimize profit, but it was understood these would be sub-optimal Lessons Learned from Integrated in this domain. Testing Because this investigation is based on existing WRS These lessons are a product of applying straightforward designs, the controls relevant model reflects the design methods to advanced systems. The lessons are optimization strategy of performance. The controls important because the engineering methods used are relevant model, a system-theoretical model, was used to fundamental to the disciipline of process control analyze system response to varying feed rates and feed engineering and are rarely used in aerospace composition. This model was also validated against engineering. These methods have not yet been applied experimental data in the Texas Tech University WRS to the types of complex systems that will be required for testbed. long-duration exploration missions. The control relevant model enables determination of the Lesson 1 - Systems must be designed for controllability second two prerequisites of controllability and observability. Eigenvalue analysis of the system As stated previously, control in this context means performance of the model in response to small bringing the process back into equilibrium in the desired perturbations in the biomass and substrate shows optimization range when the process is perturbed by whether the non-linear system is open-loop stable, open- input or environmental variations. loop state controllable, and open-loop feedback controllable. It also allows determination of Such controllability is design sensitive. As seen even in observability. our initial test development, small changes in design can markedly affect the stability, controllability, and Depending on the results of the analysis a control observability of the system. The lesson that may not be approach can be chosen. If the system is not obvious is that the control design precedes the hardware controllable, design changes can be made and different design of the system. System modeling, analysis, and approaches can be tested against the resulting model. experimental validation must precede both control and Of course, these control strategies must still be hardware design. compared against experimental data to validate the theoretical models and control algorithms. Because of this, controllability and observability dictate the sensor and effector selection and placement. This is Test Results likely to be a different set of selections than those chosen for performance testing of a process. The details of the models and test results are covered in other papers [10]. Several results need to be described When there are dependencies between systems, the here to set the stage for other conclusions. Two control design dictates the hardware requirements. As reactors are part of the controls investigation. The control algorithms are usually implemented in software, analysis, modeling and validation testing described another way of saying this is that software design must above are currently being performed for the packed bed precede hardware requirements for complex control anoxic reactor. The same process is expected to be problems. This stands in contrast to the decomposition completed this year for the aerobic tubular reactor. The of functional system requirements into hardware and initial set of models were developed and validated software requirements and interface definitions, followed against a similar set of reactors at Texas Tech by hardware and software design. University. Lesson 2 - Control (A) + Control (B) (cid:127) Control (A+B) The packed bed reactor was found to be open-loop stable. The tubular reactor was found to be open-loop Controllability is not additive for interdependent unstable. Because of this, the open loop coupled systems. This is a fundamental axiom from basic system is not fully state feedback controllable. The control theory. If there is no coupling, then subsystems open loop coupled system is fully output feedback can be specified, designed and operated independently of each other. But without appropriate analysis and deployment. Automated control may enable those modeling across systems, it is impossible to determine dependencies. Cross-system automation added after whether systems are independent or coupled, or what subsystem design will also generate dependencies interactions are possible. between subsystems that were designed independently. This conclusion was not obvious when we began our Importance test. The primary initial project goal of the project was to determine what requirements must be levied on The Constellation Program has automation individual control systems to assure integration after requirements different in kind from previous programs. design. It soon became clear that it is not possible to The design of automated systems requires risk levy integration requirements on each subsystem in the mitigation and engineering strategies different from absence of integrated design. previous programs. Subsystem requirements must be derived from integrated design, in contrast to deriving Hardware and software requirements follow system design from subsystem functional requirements and design for coupled systems. Such engineering analysis interface constraints. and integrated design must encompass the entire system. The requirements thus developed are References substantially more than the interface definition and performance and functional requirements levied during 1. “Lunar-Mars Life Support Test Project: traditional aerospace subsystem decomposition. Phase III Final Report,” CTSD-ADV-341, Engineering Directorate Crew and Thermal Systems This is important because once subsystem Division, 2000 decomposition occurs, subsequent analysis of system 2. “Bioastronautics Critical Path Roadmap (BCPR),” components provides no information about system JSC 62577, 2004 controllability, stability or observability. That analysis 3. “Investigation of Advanced Control Architecture must occur over the entire coupled set of subsystems, Issues And Technology Gaps,” CTSD-AIM-0027, not over each of the component subsystems. Engineering Directorate Crew and Thermal Systems Division, 2004 Lesson 3 - Interdependence Causes Complexity 4. “Final Report - Workshop on Advanced System Integration and Control for Life Support (ASICLS),” The complexity of integrating the bioreactor control JPL document D-30410, 2005 systems is not just an attribute of the biology, but also of 5. Chairman of the Joint Chiefs of Staff (CJCS) the interdependence of the processes. Instruction 3170.01C and CJCS Manual 3170.01, both dated 24 June 2003 Process in this context refers to a transformation of something to something else. Processes have rates, 6. “NASA Systems Engineering Handbook,” SP-6105, control variables, and dependent variables. 1995 Interdependence means that changes in the parameters 7. Bonasso, P., Kortenkamp, D., Thronesbery, C., of one system necessitate changes in the controls of “Intelligent control of a Water-Recovery System: another system, either automatically (as in the case of three years in the trenches” AI Magazine, Spring, the bioreactors) or by intent (a manual or autonomous 2003 command). 8. Hoo, K., Overland, D., “A Fundamental Mathematical Model of a Microbial Predenitrification System,” Internal JSC publication, 2004 Conclusion 9. Zhang, X., Hoo, K., Overland, D., “Modeling and Control Studies of an Integrated Biological Although the particular types of modeling and analysis Wastewater Treatment System,” 35th International performed are specific to process control problems it Conference on Environmental Systems and 8th can be generalized that the sequence of modeling, European Symposium on Space Environmental analysis and test must be performed whenever there Control Systems, paper ICES2005-01-2963, Rome, are dependencies between systems. Italy, 2005. 10. Ciskowski, M., Hoo, K., Overland, D., “Process Without these steps the risks associated with instabilities Control for Advanced Life Support, Biological Test associated with system coupling remain unknown until Bed,” 36th International Conference on after integration, sometimes until well after deployment. Environmental Systems, 06ICES-38, 2006 These lessons can be applied to all autonomous and Definitions, Acronyms, Abbreviations automated systems. The possibility of instability is one of the drivers which disallow automation of on-board systems except in exhaustively tested cases. Without AIM: Advanced Integration Matrix proper modeling, analysis and testing, system dependencies may not be discovered until after ALS: Advanced Life Support ARS: Air Revitalization Subsystem PID: Proportional Integral Derivative DOD: Department of Defense SRS: System Requirements Specification IRD: Interface Requirements Document WRS: Water Recovery System JCIDS: Joint Capabilities Integration and Development System JSC: Johnson Space Center