HRP Commanding Lessons Learned Commanding to HRF Racks Marc Perry / HRP ISS Medical Projects (ISSMP) Objective • For the benefit of the PD community, highlight why we recently reevaluated how we perform technical tasks on console • Communicate some of our Concept of Operations practices specific to: o Commanding o Building and analyzing telemetry displays using payload health and status o Redundancy and anomaly resolution responses • Applicable to: o PDs who perform commanding to hardware on ISS o PDs who make operational decisions based on telemetry displays o PDs who have multiple pieces of hardware or hardware with duplicate commands 2 Background • In Increment 43/44, ISSMP was commanding to the HRF Rack on ISS and unknowingly introduced a COL MDM error by sending a command to a non- powered Rack • The result was that commanding could not be performed until the commanding queue buffer was cleared • Our root cause analysis found: o The console team was unaware of risk increase associated with sending the wrong command to ISS (PEP software update) o Communication and anomaly resolution practices relevant to this anomaly were insufficient and resulted in a long recovery time, impacting the POIF Cadre and adding risk to other ISS users o ISSMP console practices did not mature with the risk/probability of the errors that we could introduce to the ISS • As a result, ISSMP revisited what were considered best practices and sought to inject, and in some cases, re-integrate robust mistake-proofing measures 3 Anomaly Reaction and Resolution Problem #1: Human Error This realization did not sit well with us and “It will be impossible to prevent an we know the ISS Program and our anomaly like this from occurring again.” research colleagues deserved better. Because of this, a focus was applied to the reduction of human-introduced error and the communication surrounding anomaly responses. Those two focus areas will reduce the impacts of human error and improve the community’s ability to recover. Problem #2: Improving Human Factors (Mistake Proofing) Problem #3: Anomaly Resolution Response and Communication 4 Problem #1 – Human Commanding Error • Accepted we would not have the resources or the interfaces to fully mitigate the lack of awareness a PD might have to a changing ISS environment • We chose to accept console errors and mitigate our highest risks via training and console mistake-proofing • Our evaluated error rate is 0.1 % (~3662 commands over 1 year) • For this specific error and impact, most PDs would not fall into this category • The execution environment could change on ISS and you won't know that you have the ability to impact a system larger than your hardware • This leads to problem 2. o If the best way to minimize risk is to mistake-proof our commanding, how can we do this? o And how close can we get to “target zero”? 5 Problem #2: Improving Human Factors 2.1 EPC commanding setup 2.2 Command and Data Handling Preparation 2.3 Reduce complacency and introduce some formality back into communicating commanding expectations 2.4 Other best practices to communicate 6 Problem #2.1: EPC Setup • If a specific command will be used frequently, create a dataset instead of manually editing the user form each time o This introduces reliability and mitigates typo errors • Filter commands in Command Operation that will only be used that day o Command Operation refreshes constantly to a default command instead of the queued command the PD is preparing to send o This helps the PD make sure the refresh doesn’t go unnoticed • Create displays that are payload/rack specific o Do not put commands to multiple racks in a single display o HRF had a duplicate payload in both racks so it built a display to handle commanding for both payloads o This introduced a chance for human error to inadvertently command to the wrong payload 7 Problem #2.1: EPC Setup • Create a display if a set of commands will be frequently sent in a specific order Example in o Create this display to flow in the order the commands will be sent backup charts o HRF has individual displays to activate the rack in different configurations, not just a generic rack activation display o Higher specialization is preferred as this removes command buttons from the display that could be selected by accident • If off-nominal (infrequent) commands will be used, create a command plan with predefined user form inputs o Review command plan with console team during session pre-coordination meeting 8 Problem #2.2: C&DH Preparation • Use naming conventions in PDL that are well defined and easily discernable o HRF’s Rack 1 and Rack 2 commands were only differentiated by 1 digit 1 2 • “H C42_TWO_STAGE_CMD” vs “H C42_TWO_STAGE_CMD” o Make command names that can easily be filtered • Also allows PRO to monitor commanding through Command Track with better transparency 9 Problem #2.3: Reduce Complacency • Accept that you need to put protections in place to protect you from yourself. Use your PRO/PSE experts to help with this! • The more common the commanding sessions become, the more your brain goes on autopilot • Mitigate the normalization of deviance o Gradual acceptance of sending the wrong command o Our operating environment allowed this as commands are no longer verbalized on the PRO loop as a mechanism for mistake proofing o Our internal HRF cadre reset the formality of the interface between our lead and our systems engineer, applied to the communication with the PRO as well 10