ARI Contractor Report 2004-03 Perspectives On Studying Collaboration In Distributed Networks. Karol G. Ross Klein Associates Inc ‘This report is published to meel legal and contractual requirements and may not meet ARI's scientiic or professional standards for publication, January 2004 United States Army Research Instituto for the Behavioral and Social Sclonces Appraved for pub lease; disthuton is united - 20040202 027 REPORT DOCUMENTATION PAGE 4 REPORT DATE Wd in| 7 RECHT TYEE ‘3. DATES COVERED tom ..t aang 2008 int ‘Apri 2002 — August 2003, Perspectives On Studying Collaboration n Distibuted BASWOI-02-P-0526 . Networks 1 PROGRAM E-CVENT HUME . 22065. 7, AUTHORS a PROT MOMBER ara0 _. Karol G, Rese ea WUMER 208 7 PE HACPSMING ORCAMIZATICN NANE(S) AND ADRESS) ‘3. PERFORMING ORGANIZAT ON RFPORT MUMDER. ein Associates In. 11750 Commerce Center Bld N arbor, OH 4524-6952 ©, SPONSORINGIAONITORING ASENCY NANE:S) AND ADDRESSES, | 1 MONTOR ACROWTYT U.S. Aimy Research Intute fre Betavral ‘and Social Suierces AR 5001 Eisemenior Avenue 1, HORTOR REPORT RUNGER ATTN: DAPE-ARIE ‘Avera, VA 22336-4841 Contacte Repost 7008-98 Approved for publ lass dstibuien is unin, 115. suf ERTART HOTER “Tis reports pubshed to meet legal and coriactl raqurnents and may no: meet ARs eciefe ardor oeasiont standards fe pubicson 1 ABSTRAGT farsa Pa i “This epor desortes cognive vanables and farnewerks thal ste useful in ha ineealgation of nctwork collaboration in reap ervnrmant, Netwark eullaboralin isc tent Under shady by the U.S. Army Research nse forthe ‘Behavioral and Social Sciences (Akl}ir he covert of performance in a sinulad network task using "he game SCUDHunt. SCUDHunt was develoned by ThoughiUnk, Ioarparate: ‘er to Dofonse Advanced Research Projects ‘Agency (DARPA), Thw use of SCUDHunt does net conetivie erdorsomntofthe product by ARI fhe U-S. Army or tha US Department ot Detenee. Kay verables tat fcifae funtion ht silted network enironment sre tiscussed. based on ie resus of inlervews wth profeiont SCUCHUnK players. Frameworks fr examin ng the dats frum te SCUDHntctsies are presertod, Finaly. bl reccrsmendtions for he dracions of furs research ints domain ae presente. Copritve Task Analysis Callateraton, Communication Glassiesticn, Distibsted Tearswork, Net-Centic, Network Contin, SCUDHunt, Stualinal awareness = cin engerTy Ses pasteer oe none one Urstimted Geriasesaes Unclassified 4a, TaSPAGE ‘Unciaesteg | 97, anstracr UUrdlassed Klein Associates te Commer DAS WOLD P0596 Purpose “The parposs of this report is to deserihe cognitive variables and frameworks that ate uceful inthe investigation of network co¥aboration in Army cavironments, Network collaboration is currently under stusly by the U.S. Army Research Insitute inthe content ‘of performance imu simulated network task using the SCUDHunt” gure, developed by DARPA, as purt of a study of distributed eltutional awareness (Perla, Markowitz, Nofi, ‘Weuve, Loughtan, ee Stahl, 2000), Key variables that Jaclitate metioning inthis avironment will be discusced, base on Ihe results oF iocerviews with proficicut SCUDInat players, Frameworks for examining the data from the SCUDHunt studios are also presented, Finally, bricf recommendations for the directions of fnture research in this domain avo prescntod “These perspectives on collaboration rescatch aro documented in response lo the cod to Heller understand the emerging notworked organivationa inthe Army’ (Aber, Garstka, & Stein, 1999), In ths batiloficlds where today's soldiers apply their skills, ‘commrunication is both more complex and more importanc than cver before, Military perations are widely distrbuted over space and time, and this necessitates x similar distribution for the individuals involved in those operations. There ean he no arguement the collaboration al shared understanding of tho batlespace a required for effective functioning between units fa this situation, bar ths individual sills (hat upport this ‘process ane aot well understood, What skills influence tho successor failure of ‘ommunication benvecn soldicrs? Whon these skill ane identiGed, wat steps must be tac in order to facilitate individuals ia become team mrembers who are fluent in the cffoetive methods of netoork collaboration? These skills are crucial for the U.S. Army’ fovees co maintain thelr curren level of information and technical superiority ‘Commnnication may tke place on the ballefield under a variety of circumstances, and with varying goals. ising uetwork resourocs for collaboration purposes has become a panicularly atvactive option, becanse of their versallity wd the possibility for eapid delivery of txt and graphic file, such as fragmentary orders and tactical maps, There are als polenta plfuls associated with a nerwork-based mncthod of ‘communication, hough. Many soldiers may be unfamiliar with the technology, or he technology may be non-intuitive and difficult to use: the necossary hardsware and software may nat bo cerizely voli cme pressure muy preclurle the use of certain _neteds of nctwork communication, beceuse of delays in tansfer. But, most important solders may just nat be skilled in how to collaborate inthis ype of envionment, This is ‘a bigh-stakes envizonment andthe skill to use the emerging technolagies must include ‘highly haned collaboration sills, Resoarch to understand the varies in network collaboration is erical to enpport the development uf training for collaboration skills, yh we of ACUI Tunl doe nt conse endreeet ofthe producthy ABI the CLS. Army oe UTS, Depoctneat of Defense. ‘lein Acacias oe. Contoet ASTWOI-D2-P.0526 “Understanding Collaboration Cognitive Task Analysis ‘One method for exploring the skills ‘hat will be nocded in notwork-collaborative cenvitomments is Cognitive Task Analysis (CTA), CTA is act of techniques (eg, Klein, Calderwood, & MucGregor, 1989) that were designed to ake advanage of the exporicuce ‘of a subjoct-malter expert (SME) in order to understand & complex provess, job, oF task, \sposially in euations where tino prossure, high stakes, and ambiguity of information are preseal, CTA allows the roscarchcr to probe the experiences of the expert for insights iano exaenly how exitical decisions were mutle To the context of network collaborations, experts wha have sed the relevant technology many times, under the environmental ‘conditions of interes, can provide valusble information sbout what makes this process <ifficull and where attention must bo fosusce in order to improve performance, all lout a ers point of view. Ibis difficult o fina such expense as these euvizouments are only naw emerging, in responso te changing onissions and technologics. To envision thin mew allied, the SCUDFunt gure is being used to simulate that environment. To understand the ‘eallahoration skills that may be needed in the mew hattleffeld environment. Klsin Astociatca condusted a CTA using researchers who were proficicnt at collaborating in the SCUDHunt context. The purpose of the CLA was to suggest the Qpes of collaboration skills tha ahonld be considered wher. the SCUDHuut data ae analyzed, Task Diagram ‘A Task Diagram (Milivllo, Hutlon, & Miler, 1996; Miltello, Hutton, Piske, ‘Knight, & Klcin, 1997) was used Mest fo lay out the major tasks inthe simulated ‘environment andl lo identify which tasks are most cognitively challenying, Three experienced SCUDE unt playese were intervicwod to determine ho primary tasks in playing the game, The interviewees were asked to come up with three to six mais tasks that caver the performance inthe gume and then 1 indicate which of these tasks were the msl cognitively challenging. Later, the results of the Task Diagram were compiled into ‘one graphic representation reflecting the input ftom all xc inlerviews, arul via Task ‘Diagram was reviewed with the intorsiewces before Finalization ‘The finol Task Diagram synthesizing the interviews with thros proficient ‘SCUDHnt players is shown in Figure 1. Twelve tasks were identified as comprising SCUDHunt performance: ‘© Know own and partner's assets ‘© Genecate own plan for asset placement ‘© Coordinate own plan with parmer's plan ‘tein Asoc f, Contact DASWOLIDP-0526 + Deploy own assets + Remember where assts ure deployed + Look a! share cesalis of ineligenoe gathoring +» Coordinate ta fiud out which assols were used by the partner to understand results Consider reliability of assets to assess resulls + Consider any provious intelligence and interpretations Generate own Strike Plan Trpot Surike Plan and get rors © Generate now asssl placement plan to refine conclusions. ‘mere yseL UNHCR 2g sssmof sees ee pena PORES OT Scsreti-romEvavennn03 5 SIEOREY OI ‘loin Asoc le, Contact DASWOI-O2 P0526 Of the 12 SCTDHunt tasks, 5 woro identified by the interviewees as cognitively challenging. These tasks include the following: + Generate own plan for asset placement ‘Coordinate own plan with parte’ plan 4 Consider relshility o asets to assess osu + Consider any previous intelligence and interpretations «Generate own Strike Plan Knowledge Audie A Knowledge Audit (Klein & Militell, in press) was used to further assess the ‘ive casks that were identifisd as cognitively challenging to understand the nature of the challenges and the ealluboralive aspects of the tasks. The Knowledge Andit consists of 8 set of probes designcd to extract expertiae along a aumber of dimexsions: widerslanding the big picture, noticing irmportant cues, strategies for working smart, noticing ‘opportunites for occas, und selC-monitoring. The interview is based on obtaining ‘ckamples of euch of these dimensions and probing to understand the cue and stotegies the proficient or expert performer user, as well as what they believe makes the task difficult for tho nove. ‘The results of the thee interviews were compiled inia ene Knowledge Audit ble and the strategies, cnos, and difficultice were reviewed to see What cognitive skills and ‘fmetions were indicated basell on the Trameorks reviewed balow. The following tasks ‘were combined io the final able of interviews esults, because the findings were 50 intertwined: consider reliabizity of assets co assoss Tosults, consider previous intelligence, and generate own Seike Plan. The findings are displayed in Appendix A. Coynitive Frameworks for Understanding Collaboration Rehaviors Tince sources were used as potentiat categorizations of cognitive functions. These Frameworks are showa in Appendix B. The first source is a report fiom x cognitive tesfing project that was conducted for the Navy (O'Donnell, 2001). A eognitive model and final list of cognitive functions for the project are shown. The second is the list of ‘taining cogmitive abjectiven that see generated for team decision skits training for Firefighters (Gurrs, Melek, Ross, & Thordscu, 2002), ‘Tho third is a matrix of counitive skills generated as part of a situation awarauers project (Klein, 2001) ‘Whion the results of the CTA ani the cognitive (ramewrorks wore combined, a ist ‘of taks, kills, apa functions was generated, These ate shown in ‘Ibe 1 below. ein Asa ne ‘Contet PASI 02-7-0526, Table 1 Cognitive Elements in SCUDFlunt Performance Cognitive ‘© Understand own capabilities (systems and personnel) © Envision goals and plans ‘© Define roles aad functions ‘© Unsestand enemy capabilities and behaviors Assess the situation ‘© Balance irarnediate concerns and longer term objertives = Corpensate to fil gaps in tea porformance Cognitive Skills ‘© Managing altention 1s Diseeiminsring eves and pattems | © Constructing mental models of cause wud e(Tect (inerpret infomation) ‘© Building 2 kloey frora incomplete infomation ‘© Decentering (understand hov own actions affoet partner) Monitoring one’s own perfornamee (e.g, avoiding fixation, avoiding over- enphasis on most recent information, realizing witen expectancies have heen violate) Cognitive Panetions ‘© Pereoplual processing + Short-term memory (o maintain overal picture of search arey ao td procedural and doolarative knowledas)* + Wosking memory * Tia this case, declacalive and procedural nowledge are not retrieved from Tong term memory for usc in working memory, but aro all eld in shoe term memory while other skiflsifmctions are excoulad, Memory aids (cheat sheets) ate used 10 suppatt short- term memcry. lin Arnsates as, (Contact DASWDL 02-P-0526, Communication ant Distributed Teamwork Kramoworks “for Understanding Collaboration Teams Communication Variables A communication framework was suggested as a manner of investigating, collaboration behaviors evidenced in SCUDHut. The framework war eryanived 10 facili « comtect analysis of te SCLDHmt data, Conlect anatysis "a research lechnigue for making fnfereuecs by systematically and objectively identifying specified characteristics within a text” (Stone, Dunphy, Smith, & Ogilvie, 1966). For those inuerested in exploring couunuication, con‘en| analysis allows for the examination and ‘deatifcation of messages found ina text, Content analysis was a very popular method of | analyzing public and mass medinled vornenunication in the 1970s and 1980s when hundreds of jocmal articles an television programming and political communication were published using thie meduedology (Comsicck, 1975; Jaskson-Feevk & Kras, 1980). ‘fhe main goal of comlent analysis is o describe the chatacterstes of messages camber in texts. Krippsndorf (1980) identified four advantages to content analysis. Fim, itis unobirusive tecanscitstudics texts that alcady oxist, Second. it wocepts unstructured material. which obscrvers eategorive, There is no need for simmetured interviews or quostionnaies, Third, content analysis allows a researcher to emdy dats ‘within a context. Last, iti able to handle massive amounts of data, Ono type of content analysis it convenation enalysis. Conversation analysis, cxarines mesnges exchanged daring dyadic or small group interactions in onder to scaver the “systematic and orderly propctics which are maningfol to conversents [al escatehcrs|" (leritage, 1989), Ths basie process af enversational anelysis is to obtain conversation da, transeribe From oral to writen form, develop coding schemes, calegrize messages inta schemes, analyze codings ta dscribe and draw inferences about the content, stmctur, or effects of conversation, and report th: findings. It should be notod tha! suzgestions and crilicians have heen made about the rolisbility and objectivity of conten! analysis. Spetiiealy. Kofbe and Burnet (1991) suggest that objectivity af eantent analysis studios can be impzoved by precise definitions and operational rules and by utilizing codars that are indapeudent of both the authors of the research and each other, They go on to suggest thal tho metboxt for determining inkereode reliability should be documented in the reseacch A framework for performing a content anelysis of the SCUDHun! data is shown in Appendix C. Itinoludes variables aseocialed with “Task Roles,” “Tear Maintenance Roles,” and “Self-Oriented Behaviots.” In addition ta Appendix C providi framework, it also includes part of the SCLDHunt data coded into the appropriate categories ae a sample of tow the analysis could be earried out ‘There are generally rvo types of roles individuals nood to teke-on fo make s team function in collaborative mennst—Ihove concerned with completing Ue fears tach sein hsociates Te. Contact DASWDI-UZ-P-SI6 nal those coneemed withthe social and cmotionst needs of the group (Ketrow. 1991). ‘Bath types af roles are cescntal ta an effective team. Task roles are concerned with getting te jab done. Those who function in task roles help move the grou fowant ita ‘goal and help lhe group be aluptable to needs and changes along the way. Micnfenance roles are convernel wilt he team getting along, and facus on developing, ways la express and del with eraolianal dimensions of working ina group, In addition, there are self ssentered roles that are identified as ween individuals focus on their oven n2eds, which Alsirats from collaharation. Mast effective team members will he able to adapt to tace on ‘a viriely of these roles as needed (and avoid the selE-centered hehaviors} to keep the tear functioning effectively, Ihe categories forthe coding scheme are derived from Mare (1994) and Keto (1999), ‘Ta catty ont the content analysis, 4 minimum of fur cadens should be used, ‘There should be an equal division of men #nd women t control for gender differences in perception of bohaviors, Each should be trained in the coding procedures, They should be ‘ven a coding shect along with an exploration of the procedures. They should be ‘qucstioned to determine their underatanding of these procedures, ‘ogether, cars shoul ‘gonduct u mack analysis af oue interaction, Then coders should cade a second interaction separately, After complcting this interaction, coders shold meet ta discuss their degroc of agrooment, Afler Ihe initial waining, coders should be assigned to code ivaraetions nepurately. OT uvese intezaetions, two should overlap to sorve asa measure of iniercoder reiabity Disivibuied Tearovorke From our work with distributed teams (Wiggins & Klinger, 2001) we compiled a umber of challenges or points at which distributed tearmvorc cun break dawn, These are cxeliages of information, yal eonllicis, synchnnizaiion averhead, diagnostic overhead. builling wal rnaintsining situation awareness, dfstributingsitustioual awaronoss, and sigidity. We used these clomenss to construct anether matrix that could be used fo analyze the collaborative exchanges during SCUIDHluet. This matris is shown in Appendix D. ‘The matry also provides an exaniple of sonia of the SCUDHunt dita coded into there categories. Future Research -Funue research rmust examine the bchaviors of trained persennel eollaborating in sich contexts, instances of the distributed teamwork breslalown discussed ahave must he caplured and analyze to understand the ups in performance. This very likely that for lower-level antdirs ta be good collaborators, thoy will require training that quickly sttengthus their tactical thinking skills even at lower ranks and eshctons so they eon recognize what is important to report. We must also increuse their awareness of hey areas in which distributed (sarmork ean break down, and tein behaviors to monitor and repair such breakdowns. This may include taining in task roles, team maintenance roles, mid self-monitoring rotes indicated inthe research frameworks discussed here,