AIAA Space 2011 Conference NASA/MOD Operations Impacts from Shuttle Program Authors: Michael Fitzpatrick, Gregory Mattes, Michael Grabois, Holly Griffith Operations plays a pivotal role in the success of any human spaceflight program. This paper will highlight some of the core tenets of spaceflight operations from a systems perspective and use several examples from the Space Shuttle Program to highlight where the success and safety of a mission can hinge upon the preparedness and competency of the operations team. Further, awareness of the types of operations scenarios and impacts that can arise during human crewed space missions can help inform design and mission planning decisions long before a vehicle gets into orbit. A strong operations team is crucial to the development of future programs; capturing the lessons learned from the successes and failures of a past program will allow for safer, more efficient, and better designed programs in the future. No matter how well a vehicle is designed and constructed, there are always unexpected events or failures that occur during space flight missions. Preparation, training, real-time execution, and troubleshooting are skills and values of the Mission Operations Directorate (MOD) flight controller; these operational standards have proven invaluable to the Space Shuttle Program. Understanding and mastery of these same skills will be required of any operations team as technology advances and new vehicles are developed. This paper will focus on individual Space Shuttle mission case studies where specific operational skills, techniques, and preparedness allowed for mission safety and success. It will detail the events leading up to the scenario or failure, how the operations team identified and dealt with the failure and its downstream impacts. The various options for real-time troubleshooting will be discussed along with the operations team final recommendation, execution, and outcome. Finally, the lessons learned will be summarized along with an explanation of how these lessons were used to improve the operational preparedness of future flight control teams. NASA/MOD Operations Impacts from Shuttle Program Gregory W. Mattes1 NASA - Johnson Space Center, Houston, Texas, 77058 Michael J. FitzPatrick2 Holly A. Griffith3 Michael R. Grabois4 United Space Alliance - Johnson Space Center, Houston, Texas, 77058 This paper will focus on individual Space Shuttle mission case studies where specific operational skills, techniques, and preparedness allowed for mission safety and success. It will detail the events leading up to the scenario or failure, how the operations team identified and dealt with the failure and its downstream impacts. The various options for real-time troubleshooting will be discussed along with the operations team final recommendation, execution, and outcome. Finally, the lessons learned will be summarized along with an explanation of how these lessons were used to improve the operational preparedness of future flight control teams. I. Introduction S ince 1965, members of the National Aeronautics and Space Administration (NASA)’s Mission Operations Directorate (MOD) and its predecessor organizations have provided personnel to be flight controllers in Mission Control for the Gemini, Apollo, Skylab, Apollo-Soyuz, Space Shuttle, and International Space Station missions at the Johnson Space Center (JSC) in Houston, TX. These flight controllers train for hundreds of hours in countless simulation sessions to be the “steely-eyed missile men and women” responsible for mission safety and success. This training and preparedness has paid off numerous times during the actual missions. The philosophy behind training flight controllers is not to have them see every possible malfunction, or even every malfunction in which there is a procedure. Rather, they are trained so that they have a good understanding of the system and can diagnose an anomaly and identify the failed component(s) - sometimes from the perspective of “we have not seen this exact malfunction, but we have seen something similar, so let’s see if that helps us to solve this problem” - then determine the impact(s) of the failure, and how (or if) the problem can be solved, worked around, or mitigated. This is known on console as the “Failure - Impact - Workaround” (or FIW) philosophy. In addition to their technical knowledge of the hardware and system operation, flight controllers are trained to master the “soft” skills such as listening and speaking, from being able to monitor several voice communication loops to being able to succinctly explain the situation to the Flight Director on console. While flight controllers typically have degrees in engineering, it is not a typical engineering design job. There is a large focus on teamwork, communication, failure recognition, and failure response. After years of training, flight controllers become experts at being able to see when something does not look right in their system; the hard part is figuring out what to do when that happens. Flight controllers have to think of all the possible things you can do and weigh the pros and cons, which can be intimidating when there are many different options; it often becomes a risk vs. risk tradeoff decision that has to be made in a short amount of time. Although the spacecraft have changed over the years, the need for those with proper flight controller skills has not. This paper focuses on three case studies from the Space Shuttle Program in which the flight controllers were able to quickly diagnose and assess the problem, and to use their knowledge and training to present actions for the crew to take in order to save the crew and/or vehicle hardware. These events are representative of many other 1 MMACS Flight Controller, Space Shuttle Systems, 2101 NASA Parkway Houston, TX, 77058 2 EECOM Flight Controller, Space Shuttle Systems, 2101 NASA Parkway Houston, TX, 77058 3 EGIL Flight Controller, Space Shuttle Systems, 2101 NASA Parkway Houston, TX, 77058 4 Space Flight Training Instructor, Space Shuttle Systems, 2101 NASA Parkway Houston, TX, 77058 1 American Institute of Aeronautics and Astronautics scenarios seen over the 30 years of the Space Shuttle Program, and are used to illustrate the necessity of having well-prepared flight controllers during spaceflight operations. II. STS-27 Post-Landing Cooling Anomaly On December 6, 1988, the Space Shuttle Atlantis completed its third mission, STS-27 (the 27th Shuttle mission overall), with a landing at the Kennedy Space Center (KSC) in Florida. Atlantis carried a classified Department of Defense (DoD) payload, though the details of this post-landing scenario are not classified. While on orbit, the waste heat from the Shuttle’s crew compartment is transferred into water coolant loops, which in turn transfer the heat to Freon coolant loops via the Water/Freon Interchanger. The heat is dissipated overboard using a combination of radiators mounted inside the payload bay doors and the Flash Evaporator System (FES), which sprays water onto the hot Freon Loops. For entry, the radiators are cold-soaked and then bypassed to provide a cold sink to be used from approximately 100,000 ft altitude until rollout on the runway. Finally, Ammonia (NH3) Boilers are used during post-landing operations, in which liquid NH3 is sprayed onto the Freon Loops to provide cooling until the KSC ground team can connect a Ground Service Equipment (GSE) cooling unit. In order to have a safe transition to ground cooling, the NH3 must first be deactivated onboard and confirmed to not be leaking, after which the Flight Director coordinates with the ground team to activate ground cooling. Once ground cooling is stable, the Mission Control Center (MCC) team officially hands responsibility of the Orbiter over to KSC. Additionally, during this ground cooling timeframe, the astronaut crew is egressing the Orbiter and being replaced onboard by the Astronaut Support Personnel (ASP), a non-crew astronaut who finishes the post-landing procedures for the departing crew. Providing proper cooling to the Orbiter is crucial: If cooling is lost, a powerdown of the vehicle must be accomplished relatively quickly or the Fuel Cells could overheat and explode, while at the same time temperatures can’t be allowed to get too cold for danger of freezing the water at the Water/Freon Interchanger. As seen in Fig. 1, the Space Shuttle has two Freon Loops, both of which are typically running at all times. Post- landing, the ground Cooling Cart is connected to the Orbiter at the GSE Heat Exchanger, represented by the red arrow at top left. The Freon continues clockwise in this schematic, reaching the Ammonia Boiler and the FES before splitting into legs cooling the aft coldplates, the payloads, and the water coolant loops. A Flow Proportioning Valve can set the flows to be a ratio of either 90% at the Water/Freon Interchanger and 10% at the Payload Heat Exchanger, or 57% to 43%. From these legs, the Freon goes through a pump package, heat exchangers for the Fuel Cells and midbody coldplates, and finally the radiators before reaching the GSE Heat Exchanger again. The Water/Freon Interchanger at far right is where all of the Orbiter’s coolant loops intersect, and if the Freon temperature is cold enough it could freeze the water, rupture the interchanger, and cause the complete loss of cooling to the Orbiter. Figure 1. Space Shuttle Freon Coolant Loop schematic. 2 American Institute of Aeronautics and Astronautics A. Detailed Failure Description On console in Mission Control for the STS-27 post-landing was EECOM (then known as the Electrical, Environmental, and Consumables Management Officer) and in a nearby support room, his Thermal Officer. Much like a military hierarchy, MCC’s Flight Director has ultimate authority for the mission and is in charge of the Flight Control Room (FCR, or “Front Room”) controllers such as EECOM while the Front Room controllers are in charge of the Multi-Purpose Support Room (MPSR, or “Back Room”) controllers (such as THERMAL). Approximately one hour into the post-landing activities, MCC was in the process of transitioning the cooling from onboard resources to the ground Cooling Cart. The second of two NH3 tanks was close to depletion and the ground Cooling Cart was ready to be connected to the Orbiter. While EECOM monitored the vehicle cooling, many of the other flight controllers had finished monitoring their systems and had already begun their post-landing festivities and were preparing to hang the mission plaque in Mission Control to celebrate a successful flight. While this was happening around them, THERMAL noticed that the NH3 controller had switched over from the Primary controller to the Secondary controller. The NH3 controller had undertemp logic that would automatically switch from Primary to Secondary in an undertemp condition. However, the NH3 controller is upstream of any temperature sensors that are downlinked in the telemetry (the closest were the Evaporator Outlet Temperatures, or “Evap Out T”) so it could not immediately be determined as to why the controller switched, and whether it was an actual undertemp condition or a controller failure. At typical flow rates in the Freon Loops, it takes about 45 seconds for a temperature change at the NH3 controller to reach the Evap Out T sensors, and another 45 seconds for this cold mass of Freon to reach the Water/Freon Interchanger. After the 45 seconds had elapsed, the Evap Out T sensors plummeted to below 32° F, confirming a true undertemp condition and that the Interchanger was in danger of freezing. Both the KSC ground team and the EECOM team in the MCC noted this change in the data. B. Detailed Description of Real-time Troubleshooting and Resolution THERMAL’s initial call to EECOM was to deactivate the NH3 system, due to the possibility that the ground team had hooked up ground cooling without notifying the team in the MCC, resulting in two cooling systems operating at the same time. This configuration could result in an undertemp condition due to a design flaw in the NH3 system that allowed a minimum amount of NH3 flow whenever the NH3 controller was activated, even if this minimum flow would cause temperatures to be below the control band (35±3ºF). The MCC notified the crew onboard to deactivate the NH3 controller, and by doing so, EECOM and THERMAL believed they could regain temperature control and avoid deactivating the Freon Loops, a more drastic step to prevent the cold Freon from reaching the Freon/H2O Interchanger. However, before this call was received onboard the Shuttle, the temperatures had already gone colder than their Off-Scale Low (OSL) point of 24.9° F, and the MCC was forced to make the call to the crew to turn off both Freon Loops. The Capsule Communicator, or CAPCOM, relayed this call to the crew and there was no response, while the 45-second clock for the cold Freon to reach the Interchanger counted down. At this point, THERMAL noted the lack of action by the crew and wondered if there was some discussion between EECOM and the Flight Director (both in a different room from THERMAL) over the console and not on the communication loops about possible resistance to turning off both Freon Loops. Meanwhile, EECOM assumed that the crew would respond quickly to the Freon Loops deactivation call and was reviewing the recovery procedures, not realizing that the action to take both Freon Loops off had yet to be performed. EECOM had the CAPCOM repeat the call to deactivate the Freon Loops. The Astronaut Support Personnel (ASP, who takes over operations inside the crew compartment once the crew departs) responded and turned off the Freon Loops off only seconds before the cold slug of Freon reached the Interchanger. At this point, the EECOM team in the MCC had stopped the short-term problem of the cold slug reaching and rupturing the Water/Freon Interchanger but had introduced a longer term problem. Without Freon Loops running the vehicle had no cooling, and without either reestablishing cooling or powering down, the Fuel Cells would overheat and explode. Additionally, a long-term power disruption would cause the loss of science experiments and other payloads. THERMAL and EECOM elected to allow the Interchanger to continue to heat up, as the H2O loops were still running and collecting all the cabin equipment heat, and then reactivate a single Freon Loop at a time, knowing that they had approximately 10 minutes before the Fuel Cells would start to reach critical temperatures. This decreased the efficiency at the Interchanger since only one Freon Loop would initially be running, thereby slowing the transfer of heat (or cold in this case) from the Freon Loops to the H2O Loops. When the first Freon Loop was activated, the Interchanger was hot enough to absorb the cold slug without incident. Several minutes later, the MCC called the crew to reactivate the second loop, and the warm Interchanger was again able to absorb the cold slug without incident. 3 American Institute of Aeronautics and Astronautics In the post-flight debrief of this anomaly, several factors were noted that played into the actions and reactions of the flight controllers and onboard personnel: • Lack of situational awareness by the non-EECOM members of the Flight Control Team while post- landing operations were continuing prior to vehicle handover to KSC. Members of the team had already begun to engage in post-landing festivities and the Flight Control Room in the MCC was loud. • The mission crew had already exited the Shuttle and the vehicle was manned by the ASP who was not as familiar with all the switches in the vehicle as the mission crew. • Unbeknownst to the Flight Control Team in the MCC, when the crew exited the vehicle they took all of the Flight Data File (crew procedures) with them due to the classified nature of the flight. Thus when MCC referred the ASP to a procedure, there were none to be found. • At the time, the Flight Rules stated that the MCC in Houston was in charge of the mission from the point where the Shuttle cleared the tower just after liftoff until the crew exited the vehicle. At that time, the Ground Team cut off communication from MCC to the Shuttle for a short period and it was likely that the first call to deactivate the Freon Loops never made it to the crew or ASP. • The KSC Ground Team felt it was urgent to connect the GSE Cooling Cart to the Orbiter and begin ground cooling since the Shuttle was on its last cooling system. • The KSC Ground Team had also recently changed Cooling Cart procedures so that the ground cooling was recirculated inside the Cooling Cart, generating a very cold fluid. When this fluid was introduced into the Shuttle GSE heat exchanger it caused the Shuttle Freon Loop temperatures to drop quickly. C. How was the Flight Control Team Prepared for the failure? Up until this flight, post-landing procedures were not considered a high enough priority item to be trained in simulations. It was therefore incumbent upon the EECOM team to mentally walk through the procedures, determining the pitfalls and the remediation steps that would be required. This was possible because of the extensive systems and flight controller training that both EECOM and THERMAL had gone through up to that point. It took over a year to become an Orbit THERMAL and over three years to become an Orbit EECOM. Generically, training consists of first mastering the four main tools of flight control, 1) Operational drawings, 2) Systems briefs, 3) Flight Rules, and 4) Procedures, and then adding the flight controller skills of failure recognition, resolution, next worst failure mitigation, and communication. This training starts with workbooks and computer-based training followed by the integrated simulations in the MCC. Mastering these skills for a back-room position generally takes over a year. The newly-certified flight controller would then be an on-the-job-trainee for several flights to gain real experience before starting in either the Orbit EECOM or in the Ascent/Entry back-room flow. Each of these positions requires approximately a year of training, and then training begins in the Ascent/Entry EECOM flow. The whole process from new hire to Ascent/Entry EECOM historically averaged about 6 years. When this process is complete, the certified flight controller has a wealth of experience both in simulations and real-time flight operations to draw from to address the failure scenarios that inevitably arise. In the post-flight reviews several items were identified as opportunities for improvement: • Better coordination with the KSC ground team: The MCC and KSC established a dedicated communication loop and protocol to allow EECOM to talk directly with the cooling team. Previously, all communication had to be relayed from EECOM to the Flight Director who would talk to the Convoy Commander at KSC who would then relay it to the Cooling Cart team. • Better Flight Director awareness of the criticality of the handover of cooling: The Flight Directors were briefed on the criticality of the handover of cooling and what the potential dangers were if implemented incorrectly. • Better situational awareness inside the MCC: To avoid post-landing distraction, activities not directly related to landing and vehicle safing (e.g., plaque hanging) were moved off of landing day. • Updated Flight Rules: Flight Rule 1-1, which stated that MCC had operational control of the vehicle until crew egress, was changed to having control until crew egress or handover to ground cooling, whichever occurred last. • Additional training: Post-Landing operations and malfunction training were added to simulation sessions so that the procedures were well exercised and crew and Flight Control Team familiarized in that phase of the mission. • Improved procedures: The Evap Out T Low procedure that governed this scenario was reworked so that that the Orbiter and Payload hardware would be put in the safest configuration possible. 4 American Institute of Aeronautics and Astronautics D. Summary The Flight Control Team learns something from every mission. In this case, the team learned that even though it appears that the flight may be over and it is time to celebrate the team’s accomplishments, failures (man-made or otherwise) can still occur. In the midst of distraction the EECOM team identified the undertemp situation, diagnosed the cause of two cooling systems running simultaneously, and resolved the situation while preserving crew safety and all vehicle hardware. III. STS-76 Payload Bay Door Dual Microswitch Failure A. General Overview STS-76, another flight of the Space Shuttle Atlantis and the 76th Shuttle mission overall, launched in March 1996. This flight was the third to the Russian Space Station Mir, during which the United States astronauts carried a SpaceHab module in the payload bay. The crew also transferred numerous scientific instruments and payloads to Mir in addition to critical items such as water and food. During this mission the U.S. astronauts performed the first Extravehicular Activity (EVA) around two mated spacecraft. Atlantis undocked from Mir and the crew completed their orbital activities without incident. On the intended landing day, the crew began their preparations to reconfigure Atlantis into its entry configuration. Approximately 80 minutes into the four-hour-long procedure, the crew closed the Payload Bay Doors (PLBDs) per the timeline. However, the weather at the Kennedy Space Center (KSC) was not ideal for landing, so the Flight Control Team recommended waving off the landing. After two more orbits the team had decided that it was still unsafe to land on that day and that they would try again the next day. The crew was told to perform the Deorbit Prep Backout procedure, which would reconfigure Atlantis for orbit operations and open the PLBDs to allow for vehicle cooling. (Recall from the previous case that vehicle cooling on orbit is provided by the Freon circulating within the radiators on the inside of the doors, supplemented by the Flash Evaporator System (FES) which sprays water on the Freon Loops; during the deorbit prep time frame, the radiators are cold-soaked for use on entry once the vehicle reaches an altitude where the FES becomes ineffective.) With the PLBDs closed and no cooling from the radiators, the Shuttle would eventually run out of spray water for the FES and overheat, requiring an emergency deorbit. In the Mission Control Center (MCC), the Flight Director has ultimate responsibility for the mission and real- time actions of his or her team of flight controllers. The MMACS (Mechanical, Maintenance, Arm and Crew Systems) operator works for the Flight Director in the Flight Control Room, with a subordinate called Mechanical (or MECH) in a nearby support room. MMACS and MECH are responsible for the real-time operations of the Payload Bay Door System, which includes the port and starboard doors, the 16 latches that secure the doors to the forward and aft bulkheads, and the 16 centerline latches that secure the doors to each other (with hooks on the port door and corresponding rollers on the starboard door). In the normal door opening sequence, the centerline latches are opened first (the inner sets of latches 5-8 and 9-12 first, followed by the outer sets of latches 1-4 and 13-16), then the starboard forward and aft bulkhead latches, the starboard door, the port forward and aft bulkhead latches, then finally the port door. Each mechanism contains two independently powered alternating curent (ac) motors and four microswitches, two for each motor representing the range of motion for that mechanism (open and closed, latched and released, etc.). The microswitches are wired to the motors such that when the mechanism reaches the end of travel, the microswitch is activated (sending telemetry to MCC) and sends a signal to remove ac power from the motor. The entire opening and closing sequence is controlled via a series of software commands to the Shuttle’s computers on the crew’s display monitors and switches on nearby panels, and can be operated either automatically (the crew monitors the Shuttle’s computers operating each item in sequence) or manually by the crew (the crew selects an individual mechanism to move). In the Auto Mode, if the mechanism does not reach the intended end of travel in the time it would take if only one of the two motors was operating (called “single motor drive time”), the computer issues a sequence failure message, annunciates an alert tone to the crew, and stops the automatic sequence. In case of a mechanical jam of the mechanism, the motor is designed to slip once the torque reaches a certain value, to avoid damaging the motor and/or the mechanism. 5 American Institute of Aeronautics and Astronautics B. Detailed Failure Description During the PLBD opening procedure in the Deorbit Prep Backout on STS-76, both of the Centerline Latch 9-12 Release microswitches REL A and REL B failed to indicate Open (or Released) after the motors drove for single motor time (40 seconds). See Fig. 2 for the location of the latches. This condition resulted in an “S63 PBD SEQ FAIL” (Payload Bay Door Auto-Sequence Fail) alarm, which stopped the latch drive. Figure 2. Payload Bay Door Centerline Latch location and hook schematic. While the crew was attempting to open the doors on orbit, the MMACS and MECH operators were evaluating their telemetry on the ground, which included latch open and closed microswitches, real-time plots of ac currents, and the computer alarms. The MMACS team quickly concluded that the missing microswitch indications after single motor drive time triggered the “S63 PBD SEQ FAIL” alert message. However, the plots of ac current traces indicating the motor drive were difficult to interpret. Normally, Payload Bay Door operations drive Centerline Latch Gangs 5-8 and 9-12 simultaneously, and each Latch Gang has two motors, one driven by ac bus 1 and the other on ac bus 3. Thus, two motors drive on each ac bus. It was obvious from the ac plots that both Latch Gangs had initially started driving. The open position was indicated at the nominal time for Latch Gang 5-8 with an expected drop in current on both ac buses 1 and 3, as the microswitches removed power from the motor at the end of travel. See Fig. 3 for microswitch and ac motor traces. When the microswitches for latches 9-12 never triggered the open microswitches, ac buses 1 and 3 both continued to show a current level that could be expected with either nominal motor drive or the motors driving into the built-in slip clutches; the current from Latch Gang 5-8 drive had masked critical current draw information for the 9-12 gang. With any level of certainty, it was impossible to tell the position of Latch Gang 9-12 using the data in the MCC. 6 American Institute of Aeronautics and Astronautics Figure 3: Payload Bay Door Centerline Latch Gang 5-8 and 9-12 microswitch and ac motor data C. Detailed Description of Impact From the data the MMACS team had, the latches were in an indeterminate position. Without all of the PLBD centerline Latch Gangs open, it was impossible to open the doors. If it could be determined that the latch was only partially open and potentially jammed, it would be possible to slide the latch over the roller when the starboard door was opened; however, when the starboard door was re-closed the latch might interfere and prevent door closing and a safe vehicle re-entry. In the current condition with the doors both closed, there was no radiative cooling, with only the Flash Evaporator System (FES) spraying water onto the Freon Loops and a cold sink of Freon trapped in the bypassed radiators to provide vehicle cooling. Both, however, are limited consumables: when the cold soaked radiators warmed, the vehicle would then be dependent on the finite store of water in the supply water tanks for the FES. In this case, the only option would be to re-close all the latches and land somewhere, but the Flight Control Team had already passed up the last opportunity for landing at KSC, which was the only landing site called up for that day. There was a landing opportunity on the current revolution to Edwards Air Force Base in California, but this opportunity required a deorbit burn just thirty minutes after the PLBD latch failure. The next opportunity to a primary landing site after that was in another 7-8 orbits, and the water supply for the FES would not last that long. While the MMACS team was trying to determine the PLBD situation, the other operators in the room were preparing for a potential deorbit and also powering down to reduce the power and cooling load. It was clear that the best option was for the MMACS team to determine the position of the centerline Latch Gang 9-12 so they could determine if it was safe to press forward with PLBD opening and regain vehicle cooling. D. Detailed Description of Real-time Troubleshooting and Resolution Because the available microswitch and ac current trace data was inconclusive, the MMACS team relied on the preplanned procedure to have the crew visually verify the Latch Gang position, and compare it to a drawing in the procedure. This requires the crew to look out the aft flight deck windows at the centerline latches toward the aft of the vehicle. The MMACS team had the crew working on verifying the latch position while the ground team investigated further troubleshooting. The crew had binoculars at their disposal to investigate the position of the latch. See Fig. 4 for view out aft flight deck window. 7 American Institute of Aeronautics and Astronautics Figure 4: View from Port Aft Flight Deck Window with Payload Bay Doors closed. This mission carried a single module SpaceHab in the payload bay. The SpaceHab’s main purpose was to provide increased habitable volume for storage and science payloads, connected to the crew cabin via the external airlock. On the top of the SpaceHab was a viewport window that fortuitously was directly underneath the 9-12 Latch Gang. See Fig. 5 for a photo of the SpaceHab within the Shuttle’s payload bay; each gang of latches (1-4, 5-8, 9-12, and 13-16) occupies one of the four radiator panels that line the payload bay doors from forward to aft. From the aft windows of the flight deck, the crew reported that Latch Gang 9-12 appeared to be open, but due to the distance away from the window, they were not entirely certain, even using binoculars. The MMACS team asked the crew to ingress the SpaceHab module (which required opening the hatch from the middeck into the airlock) and look out the overhead viewport. The crew reported that the 9-12 Latch Gang was definitely open; they confirmed this by visually comparing to Latch Gang 5-8 which was already confirmed to be in the open position and to a diagram in an orbiter systems data book. The MMACS team then had the Figure 5: View of Shuttle showing location of SpaceHab viewport. crew drive the 9-12 centerline Latch Gang in the open direction to confirm they were no longer moving open and as a last attempt to trigger the 8 American Institute of Aeronautics and Astronautics open microswitches. As expected, the crew reported no visual movement of the latches, and the microswitches did not engage. Finally, the MMACS team was confident with the position of the Latch Gang and were comfortable pressing forward with the remainder of the door opening procedure. The crew then opened the remainder of the centerline latches and the starboard bulkhead latches to verify that the starboard door popped open slightly as it does per design. They then opened the doors themselves and re-initiated radiative cooling to space. Several hours later, the two failed centerline latch 9-12 release microswitch indications recovered and showed the nominal open position. This indicated the loss of these indications was likely due to thermal effects and/or misrigging, which had been seen on previous flights. E. How was the Flight Control Team Prepared for the failure? The MMACS team was highly prepared for such a failure. The MMACS flight controller has years of training and system support as both a MECH controller and a MMACS trainee before being certified to work in the primary Flight Control Room. They prepare through studying electrical and mechanical schematics, reading and updating systems briefs, helping to troubleshoot real-world ground hardware problems, spending countless hours in flight-like simulations, and creating operations products prior to every Shuttle flight. In this particular case, while the dual- microswitch failure had not been seen in flight or on the ground before (most likely due to the thermal effects of the unique radiator coldsoak), the MMACS team had documented single microswitch failures dozens of times over the history of the Shuttle program. Additionally, the team knew where the viewport was located on the SpaceHab, which allowed the crew to verify the position of the latches. The viewport location had been documented but had not been incorporated into any procedure for troubleshooting, and MMACS was able to use his systems knowledge outside of the procedures to come up with a time-critical plan. The preplanned procedures drove the crew to visually verify the position of the latches. The drawings on board the Shuttle and in the Flight Control Room allowed for the proper diagnosis of the latch position. This failure required the coordination and cooperation of multiple members of the Flight Control Team working seamlessly together to solve a complicated problem was a result of the hard work and dedication of the Shuttle flight controllers. F. Post Flight Work and Lessons Learned As a result of this failure, the Deorbit Prep Backout procedures were changed to separate the opening of centerline Latch Gangs 5-8 and 9-12 by driving them sequentially instead of simultaneously. Since both Latch Gangs drive one motor each on ac buses 1 and 3, by opening the Latch Gangs separately the currents are not overlaid and can be monitored individually to verify if the Latch Gang actually drove to the release position. Additionally, the PLBD Opening steps in the Backout procedures were moved earlier in the timeline to give more troubleshooting time should a failure occur. This would allow for more time to determine PLBD health and status as well as to target additional landing sites should one be needed. G. Summary The MMACS team used their operational expertise that day to safely open the Payload Bay Doors and avoid a potential emergency deorbit by the Space Shuttle, using all the resources on the ground as well as all the crew resources on-board. In the meantime, the remainder of the Flight Control Team was preparing for all potential outcomes by targeting new landing sites and powering down electrical equipment that required radiator cooling. It was a prime example of the entire team acting in unison to solve a complicated and time critical problem. III. STS-129 Cryogenic Oxygen Destratification Event A. General Overview The Space Shuttle Atlantis was in orbit, docked to the International Space Station (ISS) in November 2009 on mission STS-129, the 129th flight of the Space Shuttle program, also designated as ISS ULF3 (the third Utilization and Logistics Flight). Among those on console in Mission Control Center was the EGIL flight controller, with EPS in a support role in a nearby room. The EGIL (Electrical Generation and Illumination) and EPS (Electrical Power Systems) flight controllers manage and oversee the Space Shuttle’s entire Electrical Power System, from generation to distribution. The Shuttle uses three Fuel Cells to generate all of the needed electricity by conversion of cryogenic oxygen (O2) and hydrogen (H2) stored in tanks in the payload bay, with the resulting electricity distributed throughout the Orbiter via a series 9 American Institute of Aeronautics and Astronautics