DMT-RNet: An Internet-based Infrastructure for Distributed Multidisciplinary Investigations of C2 Performance Eileen B. Entin Linda R. Elliott Samuel G. Schiflett Daniel Serfaty Veridian, Inc. Air Force Research Aptima, Inc. San Antonio, TX 78235 Laboratory 12 Gill Street, Suite 1400 [email protected] Brooks AFB, TX 78235 Woburn, MA 01801 [email protected] ebe, [email protected] Abstract1 We describe the first phase of a collaborative research program that leveraged emerging capabilities of the internet to connect a distributed PC-based synthetic task environment system for the purpose of research in distributed team training and decisionmaking. Using an internet-based version of the dynamic distributed decision making (DDD) paradigm, 16 participants from four multidisciplinary research groups (Brooks AFB, University of South Florida, University of Central Florida, and Aptima, Inc.) were connected and performed in a team-on-teams C2 scenario. Observers at the sites viewed scenario play in real time, and communicated with one another via voice and email. Immediately following the end of scenario play, team performance assessments were collected, integrated, and distributed back to all participants for use in an after-action review. This infrastructure will provide the means to investigate issues related to effectiveness in military distributed mission training (DMT) systems including: integration of training goals; use of constructed forces; training content, sequencing, and delivery; scenario fidelity; distributed online performance assessment; and interventions to improve learning and performance in mission planning, execution, and debriefing. The internet-enabled DDD provides a collaborative training space for investigation of these DMT issues while permitting experimental control, unrestricted data analysis, and cooperative research across distributed sites. Introduction There is no doubt that distributed mission training (DMT) will be critical to 21st century military training and performance (Kreisher, 2000). DMT links multiple high-fidelity simulators to allow operational personnel in distributed locations to train together in complex battlespace scenarios. However, high-fidelity simulators are limited in availability, and the scenarios are so realistic they run in classified mode, which impedes investigation and reporting from a researcher’s perspective. Indeed it impedes the participation of any researcher who does not have a security clearance. A need exists for an infrastructure that enables investigations of DMT that capture the requisite complexity while enabling systematic, controlled, and reportable studies. Such an infrastructure 1 Opinions, interpretations, conclusions, and recommendations are the authors and are not necessarily endorsed by the United States Air Force. This research was supported by work units 2313HA07 and 1123A002 through funding by AFOSR and AFRL. CCRT_2001 1 Distributed Teams Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 2001 2. REPORT TYPE 00-00-2001 to 00-00-2001 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER DMT-RNet: An Internet-based Infrastructure for Distributed 5b. GRANT NUMBER Multidisciplinary Investigations of C2 Performance 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Air Force Research Laboratory,Brooks AFB,TX,78235 REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES The original document contains color images. 14. ABSTRACT see report 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE 12 unclassified unclassified unclassified Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 would provide cost-effective and highly germane training experiences at moderate costs, with little or no danger to physical health or environmental consequence. In the first phase of a collaborative research program we successfully leveraged emerging capabilities of the internet (i.e., internet II) to connect distributed team players and distributed observers via a PC-based distributed synthetic task environment (STE) to investigate issues related to effectiveness in military distributed mission training (DMT) systems. The infrastructure we have developed provides a DMT research net (DMT- RNet) that will allow us to investigate issues related to distributed training performance assessment, and feedback, and to investigate the merits of alternative interventions to improve learning and performance in mission planning, execution, adaptive performance, and debriefing. In this paper we describe the distributed STE we have developed to conduct research on the “team of teams” scale. We then discuss issues related to scenario fidelity raised by use of a STE. Following that, we describe the scenario we have developed, and the procedures we developed and used for the collection and dissemination of performance data. We close with a discussion of theoretical and applied research issues enabled by this infrastructure. The Distributed Dynamic Decisionmaking Network (DDDNet) The Distributed Dynamic Decisionmaking (DDD) paradigm (Kleinman and Serfaty, 1989; Kleinman, Young, and Higgins, 1996) is a unique distributed multi-person simulation and software tool for understanding command and control (C2) issues in a dynamic team environment. It provides a distributed real-time simulation environment implementing a complex synthetic team task that includes many of the behaviors at the core of almost any team task: assessing the situation, planning response actions, gathering information, sharing information, allocating resources to accomplish tasks, coordinating actions, and sharing or transferring resources. Research applications using the DDD have demonstrated the paradigm’s flexibility in reflecting different domains and scenarios to study realistic and complex military team decision-making, and to explore issues related to team training. The DDD provides a substantial degree of control over scenario design, team organization, and data collection, while engaging the team players in a low-to- moderate degree of realism. The abstract, game-like nature of DDD scenarios makes them well suited for use by teams who have not yet acquired the level of knowledge and proficiency required to operate and benefit from a high fidelity simulator. In the first phase of a collaborative program of research on DMT, we developed an internet-based version of the DDD, DDD Network (DDDNet). The DDDNet is an integrated internet-enabled, collaborative gaming space that connects players to each other and to others, such as observers, confederates, trainers, or researchers. It allows players in distributed locations to connect and perform a distributed mission in real time. Observers and/or team trainers at any location in the network can observe the scenario play in real time. They can view the screen display and electronic communications of any player, and communicate to one another via email or voice. In addition, the DDDNet can connect players to one another for interactive mission planning, debriefings and after- action reviews (AARs). CCRT_2001 2 Distributed Teams The DDDNet provides the basis for the DMT Research Network (DMT-RNet). An integral part of the DMT-RNet is a suite of team performance and process measures that provide feedback on and evaluation metrics for team performance. These measures are available from three sources: the DDD simulation, distributed observers, and distributed players. They can be targeted at three levels: individual performance; team performance, and system-level, team-of-teams performance. The DDD-based performance outcome measures focus on effectiveness, timeliness, efficiency, speed, and latency. The DDD also provides real time individual and team performance measures that are calculated dynamically and visible throughout the performance of a scenario. The DDDNet provides an effective training environment because it collects all events and actions that occur over the course of a scenario run. Post-processing modules can parse the output files generated in a DDD run, building a database of the results for rapid manipulation, analysis, and feedback to players and/or use in performance evaluation. The output files provide the capability to play the game back after the team members have completed a scenario run, which can be particularly useful in an instructional environment. In successful tests of the DDDNet participants from four C2 research groups participating in the research program—Brooks Air Force Base, University of South Florida (USF), University of Central Florida (UCF), and Aptima, Inc. —were connected and performed in a team-on-teams C2 scenario. The DDDNet achieved and maintained a synchronized connection involving 16 participants. During the execution of the scenario, players were able to see dynamically changing measures of effectiveness. Observers were able to focus their observations at any level of detail they chose, ranging from the entire mission team to individual players on one team. At the end of play, both players and observers responded to an assessment questionnaire via the internet. As soon as that data was collected, it was tabulated and summarized, and made available to both players and trainers. At the same time, performance and process measures derived from the DDD simulation were calculated and distributed. A debriefing session was conducted in two phases. In the first phase observers provided performance feedback to the individual nodes and in the second phase observers and players at all the sites were interconnected for feedback and discussion. Scenario Development: Issues Regarding Fidelity A PC-based STE system such as the DDDNet will not be as realistic as a high-fidelity DMT system. These systems have a high degree of physical fidelity, in that the system replicates actual operational equipment, procedures, and task demands. DMT systems run in a highly secure classified mode within their own network and allow operators of different simulations systems to operate together in realistic battle scenarios while located at distributed sites. DMT systems allow highly realistic mission rehearsal practice in a multi-system multi-operator environment. CCRT_2001 3 Distributed Teams PC-based STE systems can also create complex environments for multi-operator training and performance research that network in unclassified mode on the internet. This allows the STEs to be deployed to almost any setting. It also allows university-based research to occur, based on systems that reflect essential components of operator expertise. STE systems should enable more controlled investigations while capturing performance phenomena in complex, multi-operator, expert-based operational performance (Driskell & Salas,1992). Effectiveness of an STE depends on appropriate scaling of the operational performance domain to assure that core functions and task characteristics are maintained at proper levels of complexity. STE platforms and scenarios are “synthetic” in that they are developed through functional and cognitive task analysis of an operational performance domain, and are thus analogues with regard to particular aspects of the performance domain. As operational analogues, they can be distinguished from other PC-based gaming environments. Physical Versus Psychological Fidelity The traditional approach to simulation systems is to maximize the realism, or physical fidelity, of equipment and procedures. This approach was assumed to maximize transfer of training to operational systems, based on the “identical elements theory” of transfer proposed at the turn of the 20th century by Thorndike and Woodworth (1909). Assuming that scenario content is also fully realistic, these systems will provide task- specific findings with the most accurate extrapolation for performance models. Fidelity is a difficult and complex multi-dimensional issue. Realistic equipment will certainly facilitate training and transfer of knowledge and procedural skill in equipment use. However, there are higher-level knowledge and skills that are independent from equipment procedures per se, such as knowledge of goals, roles, responsibilities, taskwork, tactics, strategies, teamwork, and/or dynamic problem solving. These aspects of expertise in a domain can be elicited, assessed, and/or trained without full equipment realism. Indeed, the fidelity and validity of a fully realistic simulation system is also limited by the degree of realism and operational relevance of its scenarios. Aspects of Psychological Fidelity Physical fidelity refers to the extent to which the system looks and performs like the “real thing.” Psychological fidelity is a broad term that refers to the degree to which the system captures functional and cognitive aspects of the performance domain. As such, it is related to issues of content validity and/or construct validity, depending on the purpose of the STE. Systems based on content/construct validity should also generalize to operational systems, despite limitations in physical fidelity (Berkowitz & Donnerstein, 1982; Bowers, et al., 1992; Cronbach, 1989; Mook, 1983). Table 1 lists several aspects of fidelity and their impact on three kinds of validity. Each aspect listed in Table 1 contributes to overall content validity, construct validity, and/or CCRT_2001 4 Distributed Teams external validity (e.g., generalizability, training transfer) of the STE system, and can also serve as a starting point for a more detailed analysis of the operational domain. For example, physical fidelity may affect content and external validity but is relatively independent of construct validity. Functional fidelity and cognitive fidelity are aspects contributing to content validity, and should therefore also relate to external validity. A platform may be high or low in each aspect of fidelity. For example, a platform may have high construct fidelity/validity but low functional fidelity, and vice versa. The research and/or training goals will help identify relevant task characteristics, levels of performance, and the level of expertise required for scenario performance. It should be noted that systems high in physical fidelity may still be low in content, construct, and external validity, depending on scenario content and training/research goals. Table 1. Dimensions of Fidelity: Impact on Validity DIMENSION DEFINITION IMPACT ON VALIDITY PHYSICAL Degree to which equipment and Content, Fidelity procedures are similar to operational External features. FUNCTIONAL Degree to which operational goals and Content, Fidelity functions are representative, (e.g., External mission / tactical goals, subgoals, operator roles, interdependencies, and specific decision / task events). COGNITIVE Degree of match in cognitive Content, Fidelity complexity of individual and team Construct, information processing demands. (e.g., External information monitoring, perception, interpretation, and decision making.) CONSTRUCT Degree to which performance constructs Construct, Fidelity of interest (e.g. planning, teamwork, External situation awareness, decisionmaking, problem solving, etc.) are inherent in both operational and STE settings. Scenario Characteristics Scenario Description In developing the scenario for the first phase of this collaborative research, a great deal of effort was focused on capturing important aspects of functional and cognitive fidelity. The scenario developed involves three nodes or teams, representing the functions of an Air Force AWACS team (AWACS), a Navy Carrier Battle Group (CVNBG) and an Air Force Combined Air Operations Center-Forward (CAOC-F) team. The Joint mission was designed to motivate intra- and inter-team collaboration and coordination of activities. CCRT_2001 5 Distributed Teams The primary constructs in this scenario are aspects of C2 teamwork. Teamwork in any setting, in its essence, is the effectiveness by which team members manage their interdependencies in order to achieve a shared goal. They may do this by specifying coordination mechanisms in advance (e.g. tactics, strategies, contingencies, etc.) or by responding effectively to unexpected events through general mechanisms of problem recognition and problem solving/decision making. Here, we wanted to ensure capture of both aspects of team performance as they affect (a) mission planning, (b) sequencing of events, according to plan, (c) sharing of information and/or assets, according to plan, and (d) dynamic problem solving. These aspects were identified through various cognitive task analysis methods that were focused on individual performance (Klinger, Andriole, Militello, Adelman, Klein, & Gomes, 1993) and team performance (MacMillan, Serfaty, Young, Klinger, Thordsen, Cohen, & Freeman, 1998; Elliott, Schiflett, Hollenbeck, & Dalrymple, in review; Coovert, Gordon, Foster, Riddle, Miles, Hoffman, & King, 1999) for the purpose of scaling STEs (Schiflett & Elliott, 2000). The overall context of the scenario is a coalition defense of a small friendly country against invasion from a large enemy country. The scenario unfolds over three phases: (a) seize air superiority; (b) defend against response; and (c) halt invasion. Each phase has tactical goals, such as “destroy the enemy AWACS asset.” Each tactical goal has subgoals, often accomplished by coordinated action. For example, the destruction of enemy AWACS is executed through a coordinated sequence of events by each team. The CAOC-F team draws out enemy fighter assets, while the AWACS team sends more powerful assets to destroy the enemy AWACS. Each tactical goal and subgoal is made explicit in the initial Air Tasking Order (ATO). In addition, further coordination and contingency planning is explicated during mission planning. These events and plans form a solid basis for the assessment of mission plan execution. The scenario enables event-based assessment of performance in mission planning, team coordination (execution of the plan), and adaptive problem solving, using an event-based measurement process similar to the approach described by Fowlkes, et al. (1994). Implementation on the DDDNet Figure 1 shows the structure and communication paths within and across teams that were used in the Phase I work. Team members at the individual nodes were co-located, while the team of teams was distributed across the continental U. S. At the start of planning, each team was given an ATO that described the current tactical situation and laid out the tactical goals and plans at an overall level. During mission planning the teams worked out details with regard to assets and targets. Team members worked out details with regard to which particular resources they would use to verify and prosecute targets. In planning target sequences, priority codes are considered, along with target window of opportunity (there may be a particular time when it should be attacked) and asset window of opportunity (location, fuel, etc). The CAOC-F node may have to assign resources to verify targets that are not listed as “certain”—the ATO may list locations where a target, such as a mobile missile launcher, is thought to be located. Then surveillance assets would have to go first to verify the target before sending assets to destroy it. CCRT_2001 6 Distributed Teams Participant Key OBSR = Observer (0,4,9,13) SD = Senior Director (1) OCA = Offensive Counter Air (2) OAG = Offensive Air to Ground (3) JFACC = Joint Forces Air Component Cmdr (5) ISR = Intel, Surveillance, Reconnaissance (6) DAS = Direct Air Suppor (7) AADC = Area Air Defense Cmdr (8) JFNCC = Joint Forces Naval Component Cmdr (10) AAW = Anti-Air Warfare Cmdr (11) ASuW= Anti-Surface Warfare Cdmr (12) Figure 1. Team Organization and Communication Network Explication of the mission plan, tactical goals, subgoals, and contingencies accomplished during mission planning served as a basis for investigation of mission performance. It is relatively straightforward to assess achievement of tactical goals and identify errors and/or deviations from the plan. At the same time, we wished to allow investigation of dynamic and adaptive performance as a response to unexpected events. In the scenario many of the planned targets either did not exist or were decoys. If the surveillance function identified these targets correctly, plans had to change in rapid response in order to avoid wasted effort and assets. Figure 2 shows a screen shot of the DDD-Net display captured during the final Phase I DMT-RNet demonstration. Training in use of the DDDNet was provided prior to the Phase I demonstration. Figure 2 highlights some of the features of the DDDNet simulation testbed and user interface. Scenario events are portrayed in the window on the left hand side of the display. Display controls are located in the middle of the right-hand side. Identification and prosecution of potential targets is accomplished with mouse clicks. Real-time measures of individual and team performance are displayed in the upper right-hand side. CCRT_2001 7 Distributed Teams Role Email comms Identified Hostile Unidentified Hostile ID Transfer Figure 2. DMT-RNet Screen from distributed scenario execution involving four nodes Collecting and Distributing Team Performance and Process Measures Measures of team performance and team processes were collected from three different sources: the DDD simulation, players, and observers. A goal of the research team was to be able to provide feedback to the participants in the scenario immediately following the end of the simulation, and this was successfully achieved in the Phase I demonstration. Simulator Data: DDD-based measures were derived from the performance and process data collected by the simulator during scenario play. At the end of the scenario, a post- processor at the Aptima DDD node aggregated the measures at three levels: individual, team, and team-of-teams. Table 2 shows some examples of measures that were obtained, the levels at which they were measured, and the type of feedback to players that they provided. These measures were sent via the internet to a node at the USF, where they were posted on an internet site accessible to all participants and researchers involved in the Phase I demonstration. CCRT_2001 8 Distributed Teams Table 2. DDD-Based Team Performance and Process Measures Level of Definition Type of Measurement Feedback Task, team-of- Gains/Losses (Displayed Dynamically) Role specific teams Offense: Value of enemy assets destroyed Individual performance (increase from zero at start) performance score; Defense: Diminished value of own assets Overall mission (decrement from total value at offense/defense outset) performance Task, team, (Note: friendly = role owned) Role specific team-of-teams # friendly assets not destroyed Individual functioning # friendly assets destroyed performance score; # friendly assets lost to hostile fire Offense/defense # friendly assets lost to fuel out actions – summed # hostile assets destroyed by friendly action across individuals in kill ratio team, Overall air refuelings completed mission performance Team Processes Number of transfers of resources Individual, node- # transfers in level, overall (team- # transfers out of-teams) teamwork Number of emails # emails sent # emails received Player Evaluation: At the end of the scenario run, players at each node responded to a questionnaire developed by researchers at the USF. This questionnaire tapped the players’ assessments of their and their teammates’ performance during the simulation, as well as their assessment of the effectiveness of team processes. The players’ responses were aggregated and posted at a USF internet site accessible to all participants. Observer Evaluation: At the end of the scenario run, the observers at each site also responded to a questionnaire that focused on the performance and processes of the team they observed. The observers’ responses were aggregated at USF and posted to the same internet site as the players’ responses. At the conclusion of the scenario execution, team performance was discussed and feedback provided at the individual team sites. Following that a distributed AAR involving players and researchers at all nodes was conducted. The ability to collect, aggregate, and make feedback available to all participants immediately following scenario play demonstrated a fundamental requirement for distributed online training and evaluation. As well as providing post-scenario feedback, the data and measures collected during and following the scenario run remain available to researchers for additional CCRT_2001 9 Distributed Teams