Calhoun: The NPS Institutional Archive DSpace Repository Theses and Dissertations 1. Thesis and Dissertation Collection, all items 2014-06 A comparison of tactical leader decision making between automated and live counterparts in a virtual environment Patton, Scott A. Monterey, California: Naval Postgraduate School http://hdl.handle.net/10945/42705 Downloaded from NPS Archive: Calhoun NAVAL POSTGRADUATE SCHOOL MONTEREY, CALIFORNIA THESIS A COMPARISON OF TACTICAL LEADER DECISION MAKING BETWEEN AUTOMATED AND LIVE COUNTERPARTS IN A VIRTUAL ENVIRONMENT by Scott A. Patton June 2014 Thesis Advisor: Quinn Kennedy Second Reader: Jonathan Alt This thesis was performed at the MOVES Institute Approved for public release;distribution is unlimited THIS PAGE INTENTIONALLY LEFT BLANK REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704–0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202–4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704–0188) Washington DC 20503. 1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED June 2014 Master’s Thesis 4. TITLE AND SUBTITLE 5. FUNDING NUMBERS A COMPARISON OF TACTICAL LEADER DECSISION MAKING BETWEEN AUTOMATED AND LIVE COUNTERPARTS IN A VIRTUAL ENVIRONMENT 6. AUTHOR(S) Scott A. Patton 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Naval Postgraduate School REPORT NUMBER Monterey, CA 93943–5000 9. SPONSORING /MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITORING N/A AGENCY REPORT NUMBER 11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government. IRB protocol number ____NPS.2014.009-IR ____. 12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE Approved for public release;distribution is unlimited A 13. ABSTRACT (maximum 200 words) The use of “responsible” autonomous systems may not be far away. Prior to developing or using responsible autonomous systems, it may be important to know if tactical leaders would make different types of decisions with automated systems than they would make with a human live crew. This work attempts to determine if decisions, time to make decisions, and confidence in decisions differ when tactical leaders rely on an autonomous wingman or a live wingman. Virtual Battlespace Simulation 2 was used to provide the virtual environment in which 30 military personnel completed a simulated mission that entailed five decision points. Participants were randomly assigned to have an autonomous or live wingman. Decision patterns were compared to a standard based on Army Doctrine for mechanized infantry Bradley sections and subject matter experts. Results indicated no significant group difference in decisions made, time to make decisions, and confidence in decisions. However, significant group differences emerged in the aspects of the wingman that participants trusted most and least. Although most participants indicated that they would not trust autonomous wingmen in real combat, results suggest that participants would revert to doctrinal decisions when faced with an unambiguous situation with an unmanned system with which they had some experience. 14. SUBJECT TERMS 15. NUMBER OF Decision Making, Unmanned Systems, Human Robotic Integration PAGES 119 16. PRICE CODE 17. SECURITY 18. SECURITY 19. SECURITY 20. LIMITATION OF CLASSIFICATION OF CLASSIFICATION OF THIS CLASSIFICATION OF ABSTRACT REPORT PAGE ABSTRACT Unclassified Unclassified Unclassified UU NSN 7540–01–280–5500 Standard Form 298 (Rev. 2–89) Prescribed by ANSI Std. 239–18 i THIS PAGE INTENTIONALLY LEFT BLANK i i Approved for public release;distribution is unlimited A COMPARISON OF TACTICAL LEADER DECSISION MAKING BETWEEN AUTOMATED AND LIVE COUNTERPARTS IN A VIRTUAL ENVIRONMENT Scott A. Patton Major, United States Army B.S., United States Military Academy, 2002 Submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE IN MODELING, VIRTUAL ENVIRONMENTS, AND SIMULATION from the NAVAL POSTGRADUATE SCHOOL June 2014 Author: Scott A. Patton Approved by: Quinn Kennedy, Ph.D. Thesis Advisor Jonathan Alt Second Reader Chris Darken, Ph.D. Chair, MOVES Academic Committee Peter Denning, Ph.D. Chair, Department of Computer Science ii i THIS PAGE INTENTIONALLY LEFT BLANK iv ABSTRACT The use of “responsible” autonomous systems may not be far away. Prior to developing or using responsible autonomous systems, it may be important to know if tactical leaders would make different types of decisions with automated systems than they would make with a human live crew. This work attempts to determine if decisions, time to make decisions, and confidence in decisions differ when tactical leaders rely on an autonomous wingman or a live wingman. Virtual Battlespace Simulation 2 was used to provide the virtual environment in which 30 military personnel completed a simulated mission that entailed five decision points. Participants were randomly assigned to have an autonomous or live wingman. Decision patterns were compared to a standard based on Army Doctrine for mechanized infantry Bradley sections and subject matter experts. Results indicated no significant group difference in decisions made, time to make decisions, and confidence in decisions. However, significant group differences emerged in the aspects of the wingman that participants trusted most and least. Although most participants indicated that they would not trust autonomous wingmen in real combat, results suggest that participants would revert to doctrinal decisions when faced with an unambiguous situation with an unmanned system with which they had some experience. v THIS PAGE INTENTIONALLY LEFT BLANK v i TABLE OF CONTENTS I. INTRODUCTION........................................................................................................1 A. PROBLEM STATEMENT .............................................................................1 B. RESEARCH QUESTIONS AND CORRESPONDING HYPOTHESES ................................................................................................1 C. EXPLORATORY QUESTION ......................................................................2 D. SCOPE OF THESIS ........................................................................................2 E. BENEFITS OF STUDY ...................................................................................3 F. THESIS ORGANIZATION ............................................................................3 II. BACKGROUND ..........................................................................................................5 A. CHAPTER OVERVIEW ................................................................................5 B. USE OF UNMANNED SYSTEMS IN THE MILITARY ............................5 1. Brief History of Unmanned Systems in the Military ........................5 2. Trust in Unmanned Systems ...............................................................6 C. STUDIES ON HUMAN-ROBOTIC INTEGRATION .................................6 D. TACTICAL DECISION MAKING .............................................................10 E. STUDIES USING VIRTUAL ENVIRONMENTS FOR DECISION MAKING EVALUATION ............................................................................13 1. Using VBS2 to Test Decision Making ...............................................15 F. LESSONS LEARNED FROM LITERARY REVIEW ..............................15 1. Understanding Human-Robotic Trust .............................................16 2. Decision Making .................................................................................16 3. Methods of Data Collection ...............................................................17 III. METHODS .................................................................................................................19 A. PARTICIPANTS............................................................................................19 B. DESIGN ..........................................................................................................21 1. Control ................................................................................................22 2. Randomization ...................................................................................23 3. Replication ..........................................................................................23 4. Blocking ..............................................................................................23 C. SCENARIOS ..................................................................................................23 1. User Interface .....................................................................................24 2. Practice Scenario ................................................................................26 3. UGV Scenario .....................................................................................27 4. Live Scenario ......................................................................................28 D. MEASURES ...................................................................................................28 1. Scenario Decision Measures ..............................................................28 a. Path Score ...............................................................................29 b. Movement Decision One .........................................................31 c. Tactical Decision One .............................................................31 d. Movement Decision Two .........................................................32 e. Tactical Decision Two .............................................................32 vi i