MODELING CONTROL ROOM CREWS FOR ACCIDENT SEQUENCE ANALYSIS by Y. Huang, N. Siu, D. Lanning, J. Carroll, and V. Dang Massachusetts Institute of Technology Nuclear Engineering Department December 1991 MODELING CONTROL ROOM CREWS FOR ACCIDENT SEQUENCE ANALYSIS by Y. Huang, N. Siu, D. Lanning, J. Carroll, and V. Dang Massachusetts Institute of Technology Nuclear Engineering Department December 1991 MITNE-296 Final Report Grant Number NRC-04-89-356 "A Systems Model for Dynamic Human Error During Accident Sequences" Project Officer: Joel Kramer Office of Nuclear Regulatory Research United States Nuclear Regulatory Commission Washington, D.C. 20555 ABSTRACT This report describes a systems-based operating crew model designed to simulate the behavior of an nuclear power plant control room crew during an accident scenario. This model can lead to an improved treatment of potential operator-induced multiple failures, since it deals directly with the causal factors underlying individual and group behavior. It is intended that the model, or more advanced developments of the model, will be used in the human reliability analysis portion of a probabilistic risk assessment study, where careful treatment of multiple, dependent failures is required. The model treats the members of the control room crew as separate, reasoning entities. These entities receive information from the plant and each other, process that information, perform actions that affect the plant, and provide information to the other crew members. The information retrieval, processing, and output activities are affected by the characteristics of the individual operator (e.g., his technical ability) and his relationship (measured in terms of "confidence level") with his fellow operators. Group behavior is modeled as the implicit result of individual operator behavior and the interactions bewteen operators. The model is applied towards the analysis of steam generator tube rupture (SGTR) accidents at a non-U.S. pressurized water reactor, using the SIMSCRIPT 11.5 programming language. Benchmark runs, comparing the model predictions with videotaped observations of the performances of three different crews during SGTR training exercises, are performed to tune a small number of model parameters. The tuned model is then applied in a blind test analysis of a fourth crew. In both the benchmarking and blind test runs, the model performs quite well in predicting the occurrence, ordering, and timing of key events. The model is also employed in a number of sensitivity analyses that demonstrate the robustness of the model (it generates plausible results even when the model parameters are assigned values not representative of observed crews) and the model's usefulness in investigating key issues (e.g., the effect of stress buildup on crew performance). i TABLE OF CONTENTS Page ABSTRACT TABLE OF CONTENTS ii LIST OF TABLES v LIST OF FIGURES vi ACKNOWLEDGMENTS vii 1. INTRODUCTION 1 1.1 Objectives 1 1.2 Motivation 1 1.3 Literature Survey 2 1.3.1 Cognitive Models for Individual Entities 3 1.3.2 Evaluating Group Performance 7 1.3.3 Models for Crew Behavior 9 1.4 Report Structure 11 2. CONTROL ROOM CREW MODEL 14 2.1 Conceptual Approach 14 2.2 Model Scope 15 2.3 Plant Model 16 2.4 Individual Modules: Overview and Qualitative Implementation 16 2.4.1 Task-Related Cognitive Activity 17 2.4.1.1 Monitoring Stage 18 2.4.1.2 Situation Assessment Stage 19 2.4.1.3 Planning Stage 22 2.4.1.4 Execution Stage 22 2.4.2 Memory 23 2.4.2.1 Knowledge Base (Long-Term Memory) 23 2.4.2.2 Short-Term Memory 24 2.4.3 Non-Task Related Activity: Stress 25 2.4.3.1 Sources of Stress 26 2.4.3.2 Effects of Stress on Individual Behavior 27 2.5 Interactions Between Operators 28 2.5.1 Task-Related Communication 29 2.5.2 Non-Task Related Communication 30 2.6 Summary 31 3. DATA COLLECTION AND ANALYSIS 39 3.1 Introduction 39 3.2 Background: Control Room Work Environment and Operations 39 3.2.1 Crew Composition 39 3.2.2 Control Room Layout and Areas of Responsibility 40 3.2.3 Crew Actions During Steam Generator Tube Rupture 41 3.2.3.1 SGTR Response Prior to Reactor Trip/ 41 Safety Injection 3.2.3.2 SGTR Response After Reactor Trip/ 43 Safety Injection 3.3 Field Study 44 3.3.1 Interviews with Control Room Operators 44 3.3.1 Interviews with Former Shift Engineers 45 3.3.3 Videotaped SGTR Exercises 45 ii TABLE OF CONTENTS (cont.) Page 3.4 Data Analysis 47 3.4.1 Data Consistency 47 3.4.2 Correlation Between Teamwork Quality and Time 52 3.5 Summary 53 4. APPLICATION OF CREW MODEL TO SGTR 71 4.1 Introduction 71 4.2 Simulation Language: SIMSCRIPT 11.5 71 4.3 Plant Model 72 4.4 Individual Operator Modules 74 4.4.1 Operator Module Characteristics 74 4.4.2 Functional Description of Individual Model Stages 75 4.4.2.1 Monitoring Stage 75 4.4.2.2 Situation Assessment Stage - Concern Generation 77 Substage 4.4.2.3 Situation Assessment Stage - Concern Merge 78 Substage 4.4.2.4 Situation Assessment Stage - Script Selection 78 Substage 4.4.2.5 Situation Assessment Stage - Control Activity 80 Substage 4.4.2.6 Execution Stage 82 4.4.3 Control Mechanisms 82 4.4.3.1 Action/Concern Manager 83 4.4.3.2 Concern Manager 83 4.4.3.3 Action Manager 84 4.4.4 Modeling Variability Among Individuals 85 4.4.4.1 Individual Responses to Cues 85 4.4.4.2 Time Spent in Task Execution 86 4.4.4.3 Stress Accumulation and Its Effects 87 4.4.4.4 Capacity and Decay of Short-Term Memory 90 4.5 Implementation of Crew Model 91 4.5.1 Interactions Between Operators and Plant 91 4.5.2 Interactions Between Individual Operators 91 4.5.2.1 Task-Related Communication 92 4.5.2.2 Non-Task Related Communication 93 4.6 Benchmarking and Testing Runs 94 4.6.1 Benchmarking Runs 94 4.6.2 Comparison of Simulated and Observed Crew Behavior 95 4.6.2.1 Performance of Crew #2 (Figure 4.10) 96 4.6.2.2 Performance of Crew #4 (Figure 4.11) 97 4.6.2.3 Performance of Crew #7 (Figure 4.12) 97 4.6.2.4 Performance of Crew #10 (Figure 4.13) 98 4.7 Summary 98 iii TABLE OF CONTENTS (cont.) Page 5. SENSITIVITY ANALYSES 131 5.1 Introduction 131 5.2 Individual Technical Ability 131 5.2.1 Extreme Crew Compositions 132 5.2.2 Intermediate Crew Compositions 132 5.3 Additional Hardware Failure 133 5.4 Capacity of Short-Term Memory 135 5.5 Stress 135 5.5.1 Sensitivity to Workload 136 5.5.2 Sensitivity to Frustration Stress 136 5.5.3 Sensitivity to Irritation Stress 137 5.5.4 Sensitivity to All Stress Sources 138 5.6 Self-Confidence 138 5.7 Relative Confidence Level 139 5.8 Summary 140 6. CONCLUDING REMARKS 148 6.1 Introduction 148 6.2 Crew Model Characteristics 148 6.3 SGTR Application Results 149 6.3.1 Benchmark and Blind Test Runs 149 6.3.2 Sensitivity Runs 150 6.4 Future Work 151 REFERENCES 153 APPENDIX A - Questions and Additional Data from Charles River A-1 Plant Field Trip APPENDIX B - Improved Questionnaire for Control Room Crew B-1 Interviews APPENDIX C - Overview of SIMSCRIPT 11.5 Programming Language C-1 APPENDIX D - Thermal-Hydraulic Model D-1 iv LIST OF TABLES No. Title Page 2.1 Summary Characteristics of Crew Model 33 3.1 Operator Nominal Responses During SGTR Scenario 55 3.2 Operator Assessments of Crew Technical Ability and Teamwork 56 Quality 3.3 Operator's Confidence Level in Fellow Crew Members (Raw Scores) 57 3.4 Example Rescaling of Expert-Rated Individual Technical Ability 58 and Crew Teamwork Quality (Expert #1) 3.5 Expert Rating of Individual Technical Ability and Crew Teamwork 59 Quality 3.6 Comparison of Crew Teamwork Quality Ratings Provided By Experts 60 and Current Shift Engineers 3.7 Normalized Times to Key Crew Actions in SGTR Simulator Exercises 61 3.8 Correlation of Crew/Individual Ratings with Crew Teamwork 62 Quality As Rated By Experts 3.9 Correlation Between Expert-Rated Individual Technical Ability 63 and Crew Member Relative Confidence Level 3.10 Scoring for Expert Comments on SRO 64 3.11 SRO Ratings Based on Expert Comments 65 3.12 Crew Performance Ratings Based On Team Performance Scale [25] 66 3.13 Correlation of Event Timings with Crew Teamwork Quality (C-TQexp) 67 4.1 Comparisons of Linear Regression Model and PRISM Predictions 101 4.2 Key Attributes of Operator Entity 102 4.3 Key Sets Owned by Operator Entity 106 4.4 Attributes of Key Model Entities 107 4.5 Summary of Concern Merge Functions 108 4.6 Observed Variations in Operator Behavior During SGTR Exercises 109 4.7 Criteria for Selecting Responses to Cues/Concerns 110 4.8 Characteristics of Operators in Simulated Crews 111 4.9 Nominal Work Times for Operator Tasks 112 4.10 Summary of Stress Level Parameters 113 4.11 Alarms and Their Setpoints 114 4.12 Boundary Conditions Affected By Operator 115 4.13 Comparison of Simulated and Observed Crew Behavior 119 5.1 Effect of Varying Individual Technical Abilities (Extreme 141 Crew Compositions) 5.2 Effect of Varying Individual Technical Abilities (Intermediate 141 Crew Compositions) 5.3 Effect of Additional Hardware Failure (Loss of BOP Radiation 142 Alarm) 5.4 Effect of Varying Burden Stress Parameters 143 5.5 Effect of Varying Frustration Stress Parameters 144 5.6 Effect of Varying Irritation Stress Parameters 145 5.7 Effect of Varying Parameters for All Stress Components 146 5.8 Effect of Varying SRO Self-Confidence 146 5.9 Effect of Varying Relative Confidence Levels 147 v LIST OF FIGURES No. Title Page 1.1 Simplified Flow Diagram for Cognitive Environment Simulation (CES) 12 1.2 Simplified Flow Diagram for Submarine Crew Model 13 2.1 Conceptual Model for Control Room Crew 36 2.2 Conceptual Model for Individual Operator 37 2.3 The Yerkes-Dodson Law Relating Performance and Arousal 38 3.1 Charles River Nuclear Power Plant Control Room Layout 68 3.2 Procedure Steps in EOP E-0 69 3.3 Correlation of Technical Ability and Teamwork Quality Ratings 70 3.4 Correlation of Teamwork Quality and Time-to-SGTR-Diagnosis 70 4.1 Schematic of SIMSCRIPT 11.5 Data Structure 120 4.2 Simplified Flow Chart for Individual Operator As Implemented 121 4.3 PHENOMENA.NOTICED Data Structure 122 4.4 CONCERN Data Structure 122 4.5 SCRIPT Data Structure 123 4.6 PROCEDURE.STEP Data Structure 123 4.7 "Production Rule" Trees Used in Fault Diagnosis 124 4.8 ACTION Process Routine Data Structure 125 4.9 Simple Model for "Frustration" Stress Buildup 126 4.10 Comparison of Simulated and Observed Crew Behaviors (Crew #2) 127 4.11 Comparison of Simulated and Observed Crew Behaviors (Crew #4) 128 4.12 Comparison of Simulated and Observed Crew Behaviors Crew #7) 129 4.13 Comparison of Simulated and Observed Crew Behaviors (Crew #10) 130 vi ACKNOWLEDGMENTS The authors would like to thank N. Rasmussen, T. Ryan, and J. Kramer for their useful comments and discussion. Special thanks are given to S. Kao and M. Boyle of New Hampshire Yankee for their generous technical support. This report, based upon the Ph.D. thesis of the first author, was prepared with the support of the U.S. Nuclear Regulatory Commission (NRC) under grant NRC-04-89-356. The opinions, findings, conclusions and recommendations expressed herein are those of the author and do not necessarily reflect the view of the NRC. vii 1. INTRODUCTION 1.1 Objectives The objectives of this study are to develop and demonstrate a framework for modeling dynamic crew behavior during accident scenarios in nuclear power plant operation. Interactions between individual operators as well as interactions between operators and the plant are treated. This framework is intended to provide a tool for better understanding and treatment of these dynamic interactions, which can be significant sources of common cause failures during severe accidents. 1.2 Motivation As demonstrated by past operational experience (e.g., the TMI and Chernobyl accidents), operator performance in accident sequences is a critical factor to nuclear power plant safety. Current probabilistic risk assessment (PRA) studies also predict that a number of operator errors are extremely important to risk, but do not explicitly treat a number of issues that may greatly affect predictions of the likelihood of multiple failures. To show this, consider five simple yet potentially important observations [1]: 0 Plant operators and plant components are interacting parts of an overall system that responds dynamically to upset conditions. 0 The actions of operators are governed by their beliefs as to the current state of the plant. 0 The operators have memory; their beliefs at any given point in time are influenced (to some degree) by the past sequence of events and by their earlier trains of thought. 0 A number of operators (more than one) are involved during the accident. 0 The event trees currently used in PRA to model accident scenarios are not literal simulations of the integrated plant/operating crew response, and, therefore, are not designed to formally treat the above concerns in detail. The lack of treatment of the dynamic interaction between the crew and the plant means that the context for any given operator action is not completely specified. For example, current PRA studies generally treat operating crew behavior only in terms of successful or unsuccessful performance of specified actions (or sets of related actions). This means that any variations (e.g., in terms of timing, event order) in operator performance of the subtasks underlying each action, the resulting variations in the plant response, and the subsequent operator responses to the plant behavior (keyed through training and procedures) are also not treated. As a second example, current PRA studies do not generally provide detailed information on dynamic process variable behavior, although this can be crucial in determining the likelihood of certain operator actions (as demonstrated by the crew's response to the rising pressurizer level in the TMI-2 accident). Since the proper context for operator actions is not established, causal, "limited rationality" models for human error [2] (i.e., models in which operators are assumed to make reasonable decisions given their state of knowledge, available resources, etc.) cannot be used, and the human reliability analysis must rely heavily on limited generic data and judgment. The second and third bullets refer to the cognitive behavior of a single operator. Neglect of this behavior greatly increases the difficulty of correctly analyzing accident scenarios in which multiple hardware failures are coupled by operator cognition. The 1