AAS 00-076 t THE RECOVERY OF TOMS-EP Brent Robertson Phil Sabelhaus NASA Goddard Space Flight Center Todd Mendenhall TRW Lorraine Fesq Massachusetts Institute of Technology On December 13th1998, the Total Ozone Mapping Spectrometer - Earth Probe (TOMS-EP) spacecraft experienced a Single Event Upset which caused the system to reconfigure and enter a Safe Mode. This incident occurred two and a half years after the launch of the spacecraft which was designed for a two year life. A combination of factors, including changes in component behavior due to age and extended use, very unfortunate initial conditions and the safe mode processing logic prevented the spacecraft from entering its nominal long term storage mode. The spacecraft remained in a high fuel consumption mode designed for temporary use. By the time the onboard fuel was exhausted, the spacecraft was Sun pointing ina high rate flat spin. Although the uncontrolled spacecraft was initially in a power and thermal safe orientation, it would not stay in this state indefinitely due to a slow precession of its momentum vector. A recovery team was immediately assembled to determine if there was time to develop a method of de- spinning the vehicle and return it to normal science data collection. A three stage plan was developed that used the onboard magnetic torque rods as actuators. The first stage was designed to reduce the high spin rate to within the linear range of the gyros. The second stage transitioned the spacecraft from sun pointing to orbit reference pointing. The final stage returned the spacecraft to normal science operation. The entire recovery scenario was simulated with a wide range of initial conditions to establish the expected behavior. The recovery sequence was started on December 28 th 1998 and completed by December 31st. TOMS-EP was successfully returned to science operations by the beginning of 1999. This paper describes the TOMS-EP Safe Mode design and the factors which led to the spacecraft anomaly and loss of fuel. The recovery and simulation efforts are described. Flight data are presented which show the performance of the spacecraft during its return to science. Finally, lessons learned are presented. •Prepared for technical papers that may later be published inthe proceedings of the American Astronautical Society. INTRODUCTION The Total Ozone Mapping Spectrometer - Earth Probe (TOMS-EP) is a National Aeronautics and Space Administration (NASA) mission to continue the long-term daily mapping of the global distribution of Earth's atmospheric ozone layer. The satellite was built by TRW for NASA's Goddard Space Flight Center. TOMS-EP collects high resolution measurements of the total column of ozone. The NASA-developed instrument measures ozone directly by mapping ultraviolet light emitted by the Sun to that scattered from the Earth's atmosphere back to the satellite. The TOMS instrument has mapped in detail the global ozone distributions as well as the Antarctic "ozone hole," which forms September through November of each year. In addition, TOMS measures sulfur-dioxide released in volcanic eruptions which may be used to detect volcanic ash clouds that are hazardous to commercial aviation. TOMS-EP was inserted into orbit by the Pegasus XL booster on July 2, 1996. In the nine days following launch, the spacecraft executed a series of Delta V burns to reach a 500 km circular Sun-synchronous mission orbit with an ascending node mean local time crossing of 11:18 AM. Originally, the data obtained from TOMS-EP were intended to complement data obtained from ADEOS TOMS, which gave complete equatorial coverage due to its higher orbit. However, with the failure of ADEOS in June 1997, the orbit of TOMS-EP was boosted to 740 km and circularized to provide coverage that is almost daily. TOMS-EP is currently the only satellite providing scientific data with an operating TOMS instrument. A QuickTOMS mission is planned for launch in August, 2000 with another TOMS instrument. Figure 1 illustrates the TOMS-EP satellite. PITCH AXIS +y ESA 2 ROLL +Z AXIS YAW AXIS Figure 1 TOMS-EP Satellite 2 SYSTEM SAFE MODES To understand the anomaly, it is necessary to understand the system implementation of the active safe modes. The Safe Power Mode uses all standby redundant equipment. It has two submodes, Sun Point Recovery and Long Term Hold, whose functions are defined in Table 1. Both submodes point the +X (roll) spacecraft axis to the Sun. The coarse sun sensor assembly (CSSA) is used for pitch and yaw attitude error and a single two-axis gyro provides rate information about pitch and yaw. The spacecraft undergoes an open loop roll spin-up by two 1 pound hydrazine thrusters prior to entering Long Term Hold. Table 1 Safe Power Submodes Mode Submode Description Automatic Transitions Safe Power Sun Point Recovery Two axis inertial sun pointing Entry from any other mode mode. CSSA and gyro are used due to fault condition. Entry as sensors. Thrusters used as from ,Long Term Hold due to actuators excessive Sun pointing error. Long Term Hold Spin stabilized Sun pointing Entry from Sun Point precession control mode with Recovery only after successful two axis rate control. CSSA and Sun acquisition. Exit from gyro are used as sensors. mode if there is excessive Sun Thrusters used as actuators pointing error. ANOMALY OUTLINE The anomaly began when an event caused the spacecraft to transition from the prime processor to the redundant processor in response to a critical parameter that exceeded an established limit. The spacecraft successfully aligned the +X axis with the sun line using a two axis inertial controller based on processed coarse sun sensor measurements and a single two axis rate gyro. At this point, the flight software should automatically spin up the spacecraft about the roll axis and transition to a very low fuel consumption momentum based controller. At some point in the transition, the flight software failed to complete the transfer to the momentum based controller. Table 2 provides a concise timeline of events starting just before the processor reboot. Within approximately 6 hours from entering Safe Power Mode, TOMS-EP had used virtually all of the 25 lb of Hydrazine fuel that remained before the anomaly. The spacecraft was pointed at the Sun, but was uncontrolled and spinning at approximately 18 deg/sec about the +X (roll) axis. The large amount of thruster activity had a small effect on the TOMS-EP orbit. TOMS- EP is required to stay within an ascending node crossing time of between 11:03 and 11:30. Before the anomaly, ascending node crossing time evolution was not a science life-limiting factor. After the anomaly, the rate of change of the ascending node crossing time was increased by about 3.6 min per year. This rate of change still allows more than 4 years of operation before the ascending node crossing time begins to degrade science collection. Table 2 Anomaly Timeline Event Time Notes Corrupted Ephemeris Position (ECI) Data 347/15:11:26 Previous value of position was 2139.6, 4193.99, -5330.01. In Telemetry. Position readin_ at this time was 1042.67, 5484.46, 4412.70. Large Pitch Error. 347/15: I1:30 Error iscalculated by subtracting the onboard propagated position quatemion from the commanded quaternion. Since error did not appear in either roll or yaw, suspect variables for SEU are those related to time (onboard clock, software time or epoch time). First Thruster Firing To Counter Wheel 347/15:13:06 The first thruster activity occurs more than 1.5 minutes after the Spin Down. Redundant Processor boot isfinished. This isthe required time to configure the ADCS hardware and initialize Sun Point Recovery. Pitch thruster firings seem to be very clean. The system started virtually sun pointed. Correct thruster pair participate in the removal of wheel momentum as itbleeds into the spacecraft. End of minimum ten minute window 347/15:23:07 The sating logic waits aminimum of 10 minutes in Sun Point required in Sun Point Recover. Recovery to allow the wheels to run down. This should prevent momentum couplin£ while the spacecraft spins up. System begins to monitor the five 347/15:23:07 The five conditions required to start the transition are: required conditions necessary to begin the 1. No presence in Fine Sun Sensor #2 transition from two single axis inertial 2. Pitch rate within specified threshold controllers to aspin stabilized momentum 3. Yaw rate within specified threshold controller. 4. Pitch angle within specified threshold 5. Yaw angle within specified threshold At this time, the processed telemetry showed that all five of the conditions above were satisfied. The flight software changes the flag "runup" from 0 (as initialized) to 1to denote that the system is ready to be spun up. Start of Roll Spin-up 347/15:23:07 Immediately after the minimum time window, the roll thrusters begin to spin up the spacecraft. Telemetry from the thruster commands shows the total roll on time to be approximately 19.15 seconds. The expected roll rate with this duration pulse should be 3.9 to 4.5 deffsec. This matches with the algorithm in the flight software and the tank reading in telemetry of 36 counts (8 bit reading) which represents 85 psi. At the start of the roll spin up, the flight software sets the flag "runup" to 2to let the system know that the roll spin-up has started. Completion of roll spin up / transition to 347/15:24:04 The telemetry shows that the roll spin-up completed on time and spin stabilized controller. ]getthe system failed transition to the spin stabilized controller. Continuous Firing of Pitch Thrusters. 347/15:24:04 Once there was angular velocity in the roll axis, imperfections in the alignment of the inertial and control axes caused a constant pitch rate to appear on the pitch gyro. The inertial control law continuously fired the pitch thrusters to compensate for this rate. The thrusters were ineffective due to the spinning dynamics. A small torque coupling between pitch and roll resulted in a continuous increase in roll rate as the pitch thrusters were fired. 1stContact after anomaly. 347/16:01:00 Ground observes spacecraft in Sun Point Recovery. Ground acquires downlink with only 3 Tank pressure 84 psi. rain to Horizon LOS. First expiration of sun acquisition 347/17:08:11 The failure to reach the spin stabilized mode caused the timeout. Redundant Processor to reset after 7000 seconds and attempt to acquire the Sun again in Sun Point Recovery. This was the first of three or four resets due to this trigger. The subsequent attempts to acquire the Sun failed due to the s_,stem d_namics. 2m Conmct. 347/17:44:00 Ground observes Sun Point Recovery failure to acquire. Tank pressure 78 psi. 3raContact. 347/19:19 Ground evaluatin_ problem. 4tnContact. 347/20:57 Ground turns on GRA 1& 2. Spacecraft processor reset occurs during pass. Tank pressure 77 psi. 5thContact. 347/22:38 Tank pressure 9psi. Spacecraft spinnin_ at 18de_sec. 4 ANOMALY CONTRIBUTING FACTORS There were several factors that combined to produce the state of the spacecraft at the time all of the fuel was spent. This condition is referred to as the "end condition". These factors were distinguished as belonging to one of two classes: factors that were necessary for the end condition and factors that contributed to the end condition. Those that were necessary are: 1. Initial fail over, 2. Wheel bearing friction, 3. Safe mode transition logic, 4. Safe mode design philosophy, 5. Ground controller response. Those that were contributors are: 6. Location of the failure in the orbit, 7. Thruster force level. Each of these factors will be examined in the following section. Factor #1 Initial Fail Over The anomaly was started by what appears to be a Single Event Upset (SEU) in the on- board Primary Processor. The telemetry stream recorded a jump in the estimated position of the spacecraft at the UTC time 347/15:11:26. This position is calculated onboard to facilitate the nadir pointing function of the attitude control system. The change in position was calculated to be greater than 9888 km in 32.768 seconds. The nominal change in position should be around 245 km. After identifying and analyzing all reasonable candidates for this anomaly, it is believed the erroneous change in position was due to an SEU in the calculation of the spacecraft state (contained in the ephemeris routine). This conclusion is supported by the fact that: 1. The magnitude of the orbit position vector is consistent between the two vectors. This significantly narrows the possible locations in code for the SEU to occur; and 2. The angle between the position vectors was about 88 degrees. This error appeared in the pitch angle error telemetry as a value of 81.05 degrees (quaternion "small angle" approximation accounts for the difference). Virtually no error appeared in the roll or yaw angle telemetry. This suggests that the spurious position was in the correct orbit plane. Again, this points to a very limited number of points in the processing. Factor #2 Wheel Bearing Friction The initial behavior in Safe Power Mode was very nominal. This event represented the seventh entry into Safe Power Mode since the start of the mission and all other entries successfully safed the spacecraft. What made this occurrence different? The key can be found in the timing of the transition from the two axis controlled sun pointing inertial mode (Sun Point Recovery) to the spin stabilized sun pointing momentum based control (Long Term Hold). Initial examination of the playback data showed that there was an 5 anomalyin the dynamicsof the spacecraftduring the transition betweenSun Point RecoveryandLongTermHold.Although there is no direct evidence of the cause because both the attitude decoder electronics (ADE) and the motor driver electronics (MDE) are turned off during Safe Power Mode, the circumstantial evidence presented below points to residual momentum in the wheels. There is a minimum delay period of ten minutes that the system must spend in Sun Point Recovery before it is allowed to transition to Long Term Hold. This delay was designed to allow for wheel rundown. Thruster activity, gyro readings and CSSA data during this ten minute time period give us important clues about the dynamic condition of the spacecraft upon attempted entry into Long Term Hold. Figure 2 shows the thruster usage within the ten minute delay interval. Note that only thrusters number 2 and 3 are firing and that they are firing in perfect unison. Thrusters 2 and 3 provide positive pitch torque which would be expected as the negative pitch momentum bias is transferred from the wheels to the spacecraft body. Figure 3 shows the spacecraft body rates in the pitch and yaw axes (no roll information is available in the backup mode). The shape of the pitch rate curve shows classic saw-tooth behavior associated with a thruster based controller with a fixed minimum pulse width subject to a near constant disturbance torque (due to the wheel run-down). The total angular impulse provided to the system in this ten minutes adds up to between 2.0 and 2.25 N-m-sec. This is based on the expected force level of about 0.35 lbf per thruster and the telemetry data which showed 586 counts (2.93 sec) of pitch thruster firing. Since the wheels started with 3.0 N-m-sec of momentum at their nominal 2000 rpm, there was 0.75 to 1.0 N-m-sec of residual momentum in the system when the spacecraft attempted to spin up about the roll axis. This residual momentum would certainly cause the "wobble" observed as the spacecraft began to spin up in roll. This is an unusual case where lower than expected wheel bearing drag caused the problem. Figure 4 is generated from on orbit data and shows a plot of the average voltage needed to keep the TOMS-EP wheels at 2000 rpm over the life of the spacecraft. Based on a linear estimate of the voltage to torque ratio, the drag seems to have leveled off at around 2 mN- m. Figure 5 show the results of a type A scan wheel life test performed at Ithaco over the course of three years. This test was performed under flight like conditions (in vacuum). The data shows that the drag varied from 4.25 mN-m at near beginning of life to around 3.25 mN-m at the end of three years. The lower limit was actually established 16 months into the test. The shapes in Figures 4 and 5 are very similar. The data suggest that the wheels have reached a steady state and there is no reason for concern over the health of the wheels. The difference is the magnitude of the drop in drag torque. The test wheel showed less then a 25% drop in drag over a 3 year interval. The on orbit wheels show greater than a 60% drop in torque in less than 2 years. The analysis below will show how the unexpectedly low drag torque caused the system to fail. Thruster Activity During 10 Minute Wait Spacecraft Rates During 10 Minute Walt 7001 0.1 0 seeJ ,_ '-J-_-- -01 "3oo/ •,_'_'_ 0 -0.2 ¢ -0.3 -0.4 Two_condSamples 32SecondSamples I--P_ch t"*- Thruster1"='-Thruster2 Thruster3 " Thruster4J Rate --Yaw RateI Figure 2 Thruster Firing Figure 3 Spacecraft Body Rates Estimated Drag Torque 8 6 _2 -- RW1 --RW 2 o_ k- -2 -4 -6 Days Since Launch Figure 4 Lifetime Drag Torque (Estimated From Voltage) ORAG"tORQUEvs.RUNNING _IMZ SW lefttmpo.,_e_cdfo¢ s,ooe 4.1oo 4.O00 II.soo IL000 =,leo E E 2_ E 1,_ tb t._ I Q o.$1_ o.1_o i_ I I I I I I t _ I 4 II 12 1(I 20 24 28 32 311 Q Operating Time (Menthe) el, Figure 5 Ithaco SCANWHEEL Drag Torque Life Test Data Anomaly Simulations Simulations were run in an attempt to match the behavior of the anomaly. The attitude control and determination subsystem verification simulation (TOMSIM) was used to try and duplicate the behavior of the spacecraft at the time of the failure. Using initial conditions similar to the state of the spacecraft at the time of the failure, the transition from Normal Science Mode to Safe Power Mode was repeated for different levels of wheel bearing drag. The drag value was decreased until the system failed the transition from Sun Point Recovery to Long Term Hold. For reference, the top line in Figure 6 shows the drag torque requirement imposed on Ithaco during the procurement of the wheels. Drag Requirments For Ithaco Wheels 0.014 0.012 _Maximum Drag _, 0.01 z 0.008 --50% Drag 0.006 (Nominal) In r_ 0.004 25%Drag 0.002 0 i 0 2000 4000 6000 RPM Figure 6 Drag Data Used in Simulations is Derived From Max Drag Requirement Wheel Drag at 50% of the Maximum Allowed This simulation shows the expected end of life performance of the Normal Science Mode to Safe Power Mode transition. In this case, the wheel model used the 50% line from Figure 6. Figures 7-10 show the behavior of a system that has the same initial conditions as the anomaly. Figure 7 shows the wheel speeds. The wheel that starts near-2000 rpm is the +Y wheel and the wheel that starts near +2000 rpm is the -Y wheel. At 50% of the maximum specified friction, the wheels are run down before the 10 minute waiting period is finished. Figure 8 shows the spacecraft body rates. A saw-tooth pattern that is similar to the actual anomaly data can be seen. There is a small rate transient when the spacecraft is spun up in roll at around 775 seconds. Figure 9 shows the processed CSSA data which gives sun angles for pitch and yaw. At the time of spin-up, the pitch and yaw error do not exceed 5 degrees. Figure 10 shows the thruster command "on" flags. There is near continuous thruster activity during the spin-up but after the spin-up is completed, thruster usage drops to zero. 0I_ / 00_ 41 o- 05 ", 1 "',, -oo5 •,'f i -o I _Q_ _L _Z_ _2 _33 i i o2 o4 06 os 0_2 04 Q6 OS Tm_ [_1 Figure 7 Wheel Spin Down Figure 8 Spacecraft Body Rates _o 73 _o 15 to S" o" -s" .1o 0.2 04. O,6 _$ 0 02 o,= o6 oA Figure 9 Processed Sun Sensor Angle Figure 10 Thruster Commands 9 Wheel Drag at 20% of Maxinmm Allowed The second simulation case presented here shows what happens when there is too much momentum in the system at the time of roll spin-up. Figures 11-14 show the behavior of a system that has the same initial conditions as the anomaly but wheel drag is scaled to 20% of maximum. Figure 11 shows the wheel speeds. At 20% drag, there is still 800 rpm (1.2 N-m-sec) remaining in the wheels when the spacecraft begins to spin up about roll. Figure 12 shows the spacecraft body rates. Coning and nutation are now apparent in the motion of the spacecraft. Figure 13 shows the processed CSSA data which gives sun angles for pitch and yaw. The system is unable to complete the transition from Sun Point Recovery to Long Term Hold because the processed sun angle error is too large. Figure 14 shows the thruster command "on" flags. Since the spacecraft was unable to complete the transition to the momentum based controller, the system is now using a two axis inertial sun pointing control law (Sun Point Recovery) with a high roll rate. This controller is unsuited for systems with a large momentum bias and the pitch thrusters begin to fire continuously in a futile attempt to reduce the observed pitch rate (caused by misalignment of control and inertial axes and the presence of a significant roll rate). The combination of very small misalignments in the thrusters and CG migration over the life of the spacecraft caused a slight pitch/roll torque coupling. As the pitch thrusters continued to fire, the roll rate slowly increased to 18 deg/sec at which point the 25 lb of hydrazine was exhausted. _¢tion WIOI _ o[M_m Specl.r¢__t_Vhcdr_tmn R_u_ With 20_ ofM_m Spccif_ _1 Fr_d_ _J A q_ 'L O6' O4 _ ]o 'x o. -I -a4 i -15 _6 ....... -2 * I I .... 04 0S 12 1.6 04 O.S 16 Figure 11 Wheel Spin Down Figure 12 Spacecraft Body Rates 10