HUMAN-COMPUTER INTERACTION, 2000, Volume 15, pp.69–74 Copyright © 2000, Lawrence Erlbaum Associates, Inc. Introduction to This Special Issue on New Agendas for Human–Computer Interaction Wendy A. Kellogg, Clayton Lewis, and Peter Polson IBM T.J. Watson Research Center University of Colorado ThefivearticlescontainedinthisspecialissueofHuman–ComputerInterac- tion(HCI)wereoriginallypresentedataworkshopatSnowMountainRanch, Colorado,inFebruary1999.Thatworkshopmarkedboththe10thanniver- saryoftheHuman–ComputerInteractionConsortium(HCIC)andtheendof themillennium.Accordingly,theworkshoptheme,“Human–ComputerIn- teractioninthe21stCentury:ProspectsandVisions,”calledonparticipantsto takestockandlookforwardtonewdirectionsforHCI.Theresults,wethink, are thought provoking. Togetaperspectiveontheevolutionofthefield,itisusefultorevisitthean- alog of this special issue—an earlier HCI special issue, Foundations of Hu- man–ComputerInteraction(Card&Polson,1990)—whichoriginatedfromthe firstmeetingoftheHCICin1989.Indoingsowearestruckbytwoobserva- tions.First,someoftheworkintheearlierissuehasprovedtobefoundational; indeedsomeofthearticlesarecitedinthisissue.Second,thecomparisonre- vealshowthefoundationsofthefieldhaveshifted,inthatissuesofmajorfocus in this issue were barely hinted at in 1990. In particular, the two collections clearlyshowthegrowingawarenessthatdesignfortheisolatedindividualuser isinadequatetoproducesystemsofpracticalvalueinsupportingtherealityof people working together. All of the articles in this special issue reflect the field’s evolution toward morecomplexandmorecontextualizedviewsofinteractivesystemsandtheir 70 KELLOGG, LEWIS, POLSON use.Thisconsiderationofmorerealistictasks,models,andsituationsofuse canbeseeninthevariousdomainsofinquiryrepresentedhere,fromtask-ac- tionmodelsintheGOMSfamilytosupportforcollaborativework.Itisalsoan impetus for more sweeping proposals to rethink how HCI research is orga- nizedandhowitrelatestootherareasofscience.Thearticleshereincollec- tivelyofferasetofproposalsforthefuture:newagendasthatcantakeresearch in HCI and Computer-Supported Cooperative Work (CSCW) to the next level. Thesearticlesareatthesametimeconservativeandradical.Theyarecon- servativeinthattheyareskepticalofthepopularhypeprevailingatthecen- tury’s end, the grandiose visions of an all-powerful technological transformationofexistence.Theyareradicalinthatallsharplyquestionpre- vailing methods and approaches in HCI and CSCW. TheconservativethemecanbeheardclearlyinAckerman(thisissue).Con- trarytomillennialoptimism,hearguesthatweremainunabletodevelopsuc- cessfultechnologicalsupportforsocialprocesses.Similarly,OlsonandOlson (this issue) argue that the media-touted view of communication technology erasing distinctions of time and space is seriously in error. At another level, BhavnaniandJohn(thisissue)warnthatharnessingnewtechnologyrequires farmorethancreatingausableorlearnableinterfaceandputtingitinfrontof people who will then absorb and master it with little effort. The radical theme is also discernable across these articles. Whittaker, Terveen, and Nardi (this issue) are strongly critical of prevailing research practice in HCI, building on the earlier critique of Newman (1994) that called attention to the disproportionate emphasis on radical innovation ratherthanevolutionaryimprovementinthefield.BhavnaniandJohn’s(this issue)attackontheincompletenessoflearnabilityandusabilityasgoalsfor system design strikes not only at attitudes toward technology, but also at common thinking within the HCI research community. Both Olson and Olson(thisissue)andAckerman(thisissue)callforsignificantreorientations ofresearch;techno-hypemustbesupplantedbyhardworkonadeepintel- lectual agenda. Finally, Furnas (this issue) boldly outlines a theoretical frameworkthatchallengesHCIresearcherstothinkseriouslyandsystemati- callyaboutthemultifariouscontextualinteractionsthatwilldefinethesuc- cessorfailureofsystems. Althoughthesethemestiethearticlestogether,eacharticlealsopresentsa specificcritiqueofthefieldandanagendaformovingtheworkforwardinnew ways. Here is a preview of the arguments. Whittakeretal.(thisissue)urgethattheHCIresearchcommunityborrow thereferencetaskapproachthathasbeeneffectiveinorganizingresearchin speechrecognition.TheyarguethatHCIresearchsofarhassufferedfromdis- persion,witheachworkerorresearchgroupsettinganagendaonlyweaklyre- INTRODUCTION 71 lated to what others are doing. Under the Whittaker et al. proposal, HCI workerswouldproposeandagreeonarelativelysmallsetofcommonprob- lems,embodiedinsampleusertasks,andwouldfocuseffortonprovidingim- provedsupportforthese.Overtime,thestandardofaccomplishmentinthe fieldwouldriseasgroupssucceedinbetteringthemeasuredqualityofearlier solutions. Thereisnoquestionthat,ifthisproposalisadoptedbyaquorumofwork- ers,theHCIlandscapeofthefuturewillbeverydifferentfromtoday’s.How- ever, questions abound. Do we understand any important interactive tasks well enough to propose performance measures adequate to guide progress overanextendedperiod?Willtasksremaincomparableovertime,giventhe rapid evolution of technologies? BhavnaniandJohn(thisissue)showthatusersofcomplexapplicationsdo notspontaneouslyusestrategiesthatmakeuseofoperationsuniquetocom- puters(e.g.,definingaggregationsofobjectsandthenusingasinglecommand tooperateonallobjectsintheaggregate).TheyuseGOMSmodelstoshow that these strategies define methods that have very different structures from typical GOMS methods (e.g., a method to delete a word). Theseresultshavefundamentalimplications.Landauer(1995)arguedthat computer applications have not improved the productivity of knowledge work because flawed product development processes produce applications with poor user interfaces. Although his conclusion is certainly justified, BhavnaniandJohn(thisissue)showthateveniftheinterfacetoacomplexap- plicationisrelativelygood,usersdonotspontaneouslylearnandusestrategies that would make them far more productive. In addition, strategies useful in complexapplicationsarenotdescribedintrainingandreferencedocumenta- tion.Thedominantparadigmforusertrainingistoprovidealimitedintroduc- tion to the user interface and basic functionality of a complex application. Usersareexpectedtoacquireadvancedskillsastheygainmoreexperience withanapplication.Minimalistdocumentation(Carroll,1990)anduserinter- facesthatsupportlearningbyexplorationarewell-known,successfulmethod- ologies designed to support this training paradigm. Bhavnani and John’s resultsshowthatthisparadigmhasfundamentallimitations;itwillnotsupport thedevelopmentofthekindsofpowerfulstrategiestheydescribe.However, userscanbetrainedtousethesestrategies,andtheypresentpreliminaryevi- dencethatthesestrategiesareretainedandgeneralized.Thus,theyarguethat theagendaforHCIneedstoshiftawayfromsupportforlearningbyexplora- tion to support for training users on productive strategies. Widespread deployment of organizational intranets and access to the Internet has led to expectations that individuals and teams will be able to seamlesslycollaborateremotely.In“DistanceMatters,”OlsonandOlson(this issue)arguethatthissimplisticviewiswrong.Theyreviewover10yearsof 72 KELLOGG, LEWIS, POLSON fieldandlaboratoryinvestigationsofcollocatedandnoncollocatedsynchro- nousgroupcollaborations,providingasignificantnewanalysisoffactorscon- tributing to a group’s ability to collaborate over distances and an outline of whatmaybepossibleinthefutureastechnologyevolvesandusers’andorga- nizations’sophisticationintheuseoftechnologyandcollaborationincreases. Inparticular,theyidentifyfourkeyconcepts:commonground,thecoupling of work, collaboration readiness, and collaboration technology readiness. Thisanalysisframesthediscussionofdistantcollaborationintermsthatrange fromtheartifactsandgroupwarethatsupportcollaborationtocomplexand subtlefactorsincludingcontextandtrust,timezones,andcultures.Although theOlsonsremainskepticalaboutthepotentialforeffortlessremotecollabo- ration—nomatterhowgoodthetechnologybecomes—theirfocusedreviewof alargebodyofempiricalevidenceandanalysisofthecriticalfactorsinfluenc- ing distance collaboration sets the stage for future work in this area. Ackerman(thisissue)takesskepticismtothenextlevelbypointingoutthat althoughtechnicalsystemsarestilltoobrittleandrigidtoaccommodatethe subtletyofsocialprocesses,thisisnolongerduetoignoranceonthepartof technologistsabouttherelevantsocialprocessestosupport.Hecallsintoques- tionwhetheranyamountofunderstandingofsocialrequirementscanleadto thecreationofsufficientlyresponsivesystems.Dubbingthisthe“social–tech- nicalgap,”Ackermanarguesnotonlythatitmaybeimpossibletoabolishbut also that embracing and addressing it is the heart and soul of CSCW’s mis- sion—its primary intellectual challenge. After reviewing seminal findings in CSCW,heconsideredthegapinmoredetailthroughanexamplethatfocuses onprivacyininformationsystems.Thisleadstoadiscussionofpotentialreso- lutions for the social–technical gap and a proposal to adapt Simon’s (1996) ideaofascienceoftheartificialasaframeworkforfutureworkinCSCW.The proposalisprovocative,suggestingthatthewayforwardistorecognizetheso- cial–technicalgapasfundamentalandtheprospectforbuildingasciencethat bridges the gap as the central intellectual mission of the field. Goingevenfurther,Furnas(thisissue)outlinesanintellectualframework thattiesHCIresearchintobroadthemesthatrunthroughmuchorallofsci- ence.HedevelopsananalysisofinteractingsystemsthatpermitsissuesinHCI tobeidentifiedwithissuesgeneraltoadaptivesystemsofallkindsinbiology, economics,ormanagement.ThekeyabstractionistheMoRAS,amosaicof responsiveadaptivesystems.Inthemosaic,thesuccessorfailureofonesub- systemistiedtothepatternofinteractionswithothersubsystems.Information technologyhastobeunderstoodnotonlyassupportingsomekindsofsubsys- temsbutalsoasalteringthewaysubsystemsarecoupled.Understandingthe coupling is crucial to successful design. SituatingHCIintheMoRASmeansnotonlythat,forexample,economic perspectivesarerelevanttoHCIbutalsothatsomeproblemsstudiedinHCI INTRODUCTION 73 canbeseenasfundamentallythesameassomeproblemsstudiedineconom- ics.Thisisaboldprogram,onethatprovidesafruitfulbasisforawholerange of new collaborations for HCI. Takentogether,thesearticlesdoaremarkablejoboftakingstockofwhere weareandlookingforwardtowhatcomesnext.Weapplaudtheauthorsfor theirwillingnesstodothehardworkoflookingbackandforwardinthesame momentandfortheircourageinrecommendingthesenewdirectionstoallof usforthebenefitofthefield.Wehopeyouwillfindtheirthoughtsasstimulat- ingaswehaveandfindyourselfreflectingonhowtheirproposalscouldaffect your own area of work. NOTES Acknowledgments. We thank the reviewers who participated in the process of bringingthisspecialissuetofruition,aswellasTomMoran,EditorofHuman–Com- puter Interaction. This creation of this special issue was closely coordinated with a similar“millennium”specialissueoftheACMTransactionsonComputer–Human Interaction,editedbyJohnM.Carroll. REFERENCES Card,S.K.,&Polson,P.(Eds.).(1990).Foundationsofhuman–computerinteraction [Specialissue].Human–ComputerInteraction,5(2–3). Carroll,J.M.(1990).TheNurnbergfunnel:Designingminimalistinstructionforpractical computerskill.Cambridge,MA:MITPress. Landauer,T.(1995).Thetroublewithcomputers:Usefulness,usability,andproductivity. Cambridge,MA:MITPress. Newman,W.(1994).ApreliminaryanalysisoftheproductsofHCIresearch,using proformaabstracts.ProceedingsoftheCHI’94ConferenceonHumanFactorsinCom- putingSystems,278–284.NewYork:ACM. Simon,H.A.(1996).Sciencesoftheartificial(3rded.).Cambridge,MA:MITPress. ARTICLES IN THIS SPECIAL ISSUE Ackerman,M.S.(2000).TheintellectualchallengeofCSCW:Thegapbetweenso- cial requirements and technical feasibility. Human–Computer Interaction, 15, 179–203. Bhavnani,S.K.,&John,B.E.(2000).Thestrategicuseofcomplexcomputersys- tems.Human–ComputerInteraction,15,107–137. Furnas,G.W.(2000).FuturedesignmindfuloftheMoRAS.Human–ComputerInter- action,15,205–261. 74 KELLOGG, LEWIS, POLSON Olson,G.M.,&Olson,J.S.(2000).Distancematters.Human–ComputerInteraction, 15,139–178. Whittaker,S.,Terveen,L.,&Nardi,B.A.(2000).Let’sstoppushingtheenvelope andstartaddressingit:AreferencetaskagendaforHCI.Human–ComputerInterac- tion,15,75–106. HUMAN-COMPUTER INTERACTION, 2000, Volume 15, pp.75–106 Copyright © 2000, Lawrence Erlbaum Associates, Inc. Let’s Stop Pushing the Envelope and Start Addressing It: A Reference Task Agenda for HCI Steve Whittaker, Loren Terveen, and Bonnie A. Nardi ATT Labs–Research ABSTRACT Weidentifyaproblemwiththeprocessofresearchinthehuman–computerinterac- tion (HCI) community—an overemphasis on “radical invention” at the price of achievingacommonresearchfocus.Withoutsuchafocus,itisdifficulttobuildon previouswork,tocomparedifferentinteractiontechniquesobjectively,andtomake progressindevelopingtheory.Theseproblemsattheresearchlevelhaveimplica- tionsforpractice,too;asresearchersweoftenareunabletogiveprincipleddesign advice to builders of new systems. We propose that the HCI community try to achieveacommonfocusaroundthenotionofreferencetasks.Weofferargumentsfor theadvantagesofthisapproachaswellasconsiderpotentialdifficulties.Weexplain SteveWhittakerisacognitivepsychologistwithinterestsinthetheory,design, and evaluation of collaborative systems and speech access; he is a senior re- search scientist in the Human Computer Interaction Department of AT&T Labs–Research,FlorhamPark,NJ,USA.LorenTerveenisacomputerscien- tistwithinterestsinrecommendersystemsandonlinecommunities;heisare- search scientist in the Human Computer Interaction Department of AT&T Labs–Research, Florham Park, NJ, USA. Bonnie Nardi is an anthropologist withaninterestinsocialnetworksandactivitytheory;sheisaresearcherinthe HumanComputerInteractionDepartmentofAT&TLabs–ResearchinMenlo Park,CA,USA. 76 WHITTAKER, TERVEEN, NARDI CONTENTS 1. THE PROBLEMS WITH HUMAN–COMPUTER INTERACTION AS RADICAL INVENTION 1.1. RadicalInventionIsNotAlwaysEffective 1.2. WhatWeDon’tKnow:Requirements,Metrics,andUsesofEveryday Technologies 1.3. HowWeDon’tKnowIt:TheDisseminationProblem 2. THE REFERENCE TASK SOLUTION 2.1. ReferenceTasksinOtherDisciplines SpeechRecognition(TheDARPAWorkshops) InformationRetrieval(TheTRECConferences) DigitalLibraryandMachineLearning 2.2. LessonsFromDARPAandTREC CriteriaforSelectingReferenceTasks PotentialObjectionstoOurProposal 3. HOW TO DEFINE A REFERENCE TASK 4. AN EXAMPLE REFERENCE TASK: BROWSING AND RETRIEVAL IN SPEECH ARCHIVES 4.1. SelectingandSpecifyingReferenceTasksintheDomainofSpeech Archives 4.2. DefiningMetrics 4.3. Task-OrientedEvaluationofaSpeechBrowsingSystem 4.4. GeneralIssuesArisingFromReferenceTask-BasedEvaluation 5. CONCLUSIONS howreferencetaskshavebeenhighlyeffectiveinfocusingresearchintoinformation retrievalandspeechrecognition.Wediscusswhatfactorshavetobeconsideredinse- lectingHCIreferencetasksandpresentanexamplereferencetask(forsearching speecharchives).Thisexampleillustratesthenatureofreferencetasksandpointsto theissuesandproblemsinvolvedinconstructingandusingthem.Weconcludewith recommendationsaboutwhatstepsneedtobetakentoexecutethereferencetaskre- searchagenda.Thisinvolvesrecommendationsaboutboththetechnicalresearch thatneedstobedoneandchangesinthewaythattheHCIresearchcommunityop- erates.Thetechnicalresearchinvolvesidentificationofimportantusertasksbysys- tematicrequirementsgathering,definitionandoperationalizationofreferencetasks andevaluationmetrics,andexecutionoftask-basedevaluation,alongwithjudicious useoffieldtrials.Perhapsmoreimportant,wehavealsosuggestedchangesincom- munitypracticethatHCImustadopttomakethereferencetasksideawork.Wemust createforumsfordiscussionofcommontasksandmethodsbywhichpeoplecan comparesystemsandtechniques.Onlybydoingthiscanthenotionofreference tasksbeintegratedintotheprocessofresearchanddevelopment,enablingthefield toachievethefocusitdesperatelyneeds. REFERENCE TASK AGENDA 77 1. THE PROBLEMS WITH HUMAN–COMPUTER INTERACTION AS RADICAL INVENTION Researchinhuman–computerinteraction(HCI),particularlyasembodied intheCHIconference,focuseslargelyonnovelproblemsandsolutionsthat pushthetechnologyenvelope.Mostpublicationsdescribenoveltechniques ornovelapplicationsofexistingtechniques.AstudybyNewman(1994)pro- videdquantitativeevidenceforthis.HecomparedCHIwithfiveotherengi- neeringresearchfields,suchasthermodynamicsandaerodynamics.Heused contentanalysistoclassifyabstractsofpublishedarticlesintermsofthetypeof contributiontheymadetothefield.Hefoundthatinotherengineeringdisci- plines,over90%ofpublishedresearchbuiltonpriorwork.Therewerethree majorwaysthatresearcheffortscouldextendpublishedwork:(a)bettermod- elingtechniques(usedformakingpredictionsaboutdesigns),(b)bettersolu- tions (to address previously insoluble problems), and (c) better tools and methods(toapplymodelsorbuildprototypes).Thepicturewascompletely different for HCI. Newman (1994) conducted a similar analysis of CHI ab- stractsforthe5yearsfrom1989to1993,attemptingtoclassifyabstractsasde- scribing one of the three types of enhancements previously identified. However, only about 30% of articles fit into these categories of developing prior work. The majority of CHI articles either reported “radical” solutions (new paradigms, techniques, or applications) or described experience and heuristics relating to radical solutions. 1.1. Radical Invention Is Not Always Effective ThisanalysisstronglysuggeststhatCHIisdifferentfromotherengineering researchdisciplines.But,isthisgoodorbad?Isitaproblemthatourfieldis dominatedbyattemptsatradicalinvention,apparentlycrowdingouttheprac- ticeof“normalscience”(Kuhn,1996)?Orisitavirtue?Weofferarguments thatthecurrentstateofaffairsisproblematicbasedontwodifferentcriteriafor success in our field. One criterion for success that is consistent with the radical invention ap- proachistechnologytransfer.Astrongmotivationforconstantinnovationisthe exampleofwholenewindustriesbeingcreatedbyuserinterfaces(UIs).Peo- ple are aware that applications such as Visicalc and Lotus® 1-2-3 drove the early PC market, and Mosaic/Netscape® led to the Web explosion. In this view, HCI research is an engine room from which novel interaction tech- niquesaresnatchedbywaitingtechnologycompanies;orbetteryet,research- ers start their own companies. There are undoubtedly successes originating fromwithintheHCIcommunity,includingUItoolkitsandgeneralprogram- mingtechniques(Rudisill,Lewis,Polson,&McKay,1996),aswellastheideas