ManMachine Dialogue ManMachine Dialogue Design and Challenges Frédéric Landragin Series Editor Patrick Paroubek Firstpublished2013inGreatBritainandtheUnitedStatesbyISTELtdandJohnWiley&Sons,Inc. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced, storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentionedaddress: ISTELtd JohnWiley&Sons,Inc. 27-37StGeorge’sRoad 111RiverStreet LondonSW194EU Hoboken,NJ07030 UK USA www.iste.co.uk www.wiley.com ©ISTELtd2013 TherightsofFrédéricLandragintobeidentifiedastheauthorofthisworkhavebeenassertedbyhimin accordancewiththeCopyright,DesignsandPatentsAct1988. LibraryofCongressControlNumber: 2013939407 BritishLibraryCataloguing-in-PublicationData ACIPrecordforthisbookisavailablefromtheBritishLibrary ISBN:978-1-84821-457-6 PrintedandboundinGreatBritainbyCPIGroup(UK)Ltd.,Croydon,SurreyCR04YY Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv PART 1. HISTORICAL AND METHODOLOGICAL LANDMARKS . . 1 Chapter1.AnAssessmentoftheEvolution ofResearchandSystems . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.Afewessentialhistoricallandmarks . . . . . . . . . . . . . . . . 5 1.1.1.Firstmotivations,firstwrittensystems . . . . . . . . . . . . 6 1.1.2.Firstoralandmultimodalsystems . . . . . . . . . . . . . . . 11 1.1.3.Currentsystems: multiplicityoffieldsandtechniques . . . 14 1.2.Alistofpossibleabilitiesforacurrentsystem . . . . . . . . . . . 16 1.2.1.Recordingdevicesandtheiruse . . . . . . . . . . . . . . . . 17 1.2.2.Analysisandreasoningabilities . . . . . . . . . . . . . . . . 20 1.2.3.Systemreactiontypesandtheirmanifestation . . . . . . . . 22 1.3.Thecurrentchallenges . . . . . . . . . . . . . . . . . . . . . . . . 23 1.3.1.Adaptingandintegratingexistingtheories . . . . . . . . . . 23 1.3.2.Diversifyingsystems’abilities . . . . . . . . . . . . . . . . . 25 1.3.3.Rationalizingthedesign . . . . . . . . . . . . . . . . . . . . 26 1.3.4.Facilitatingtheimplementation . . . . . . . . . . . . . . . . 27 1.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Chapter2.Man–MachineDialogueFields . . . . . . . . . . . . . . . 29 2.1.Cognitiveaspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.1.1.Perception,attentionandmemory . . . . . . . . . . . . . . . 32 vi Man–MachineDialogue 2.1.2.Representationandreasoning . . . . . . . . . . . . . . . . . 35 2.1.3.Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.2.Linguisticaspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.2.1.Levelsoflanguageanalysis . . . . . . . . . . . . . . . . . . 41 2.2.2.Automaticprocessing . . . . . . . . . . . . . . . . . . . . . . 44 2.3.Computeraspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.3.1.Datastructuresanddigitalresources . . . . . . . . . . . . . 45 2.3.2.Man–machineinterfaces,plasticinterfaces andergonomics . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Chapter3.TheDevelopmentStagesofaDialogueSystem . . . . . . 47 3.1.Comparingafewdevelopmentprogresses . . . . . . . . . . . . . 48 3.1.1.Ascenariomatchingthe1980s . . . . . . . . . . . . . . . . 48 3.1.2.Ascenariomatchingthe2000s . . . . . . . . . . . . . . . . 49 3.1.3.Ascenariotoday . . . . . . . . . . . . . . . . . . . . . . . . 51 3.2.Descriptionofthemainstagesofdevelopment . . . . . . . . . . 52 3.2.1.Specifyingthesystem’staskandroles . . . . . . . . . . . . 52 3.2.2.Specifyingcoveredphenomena . . . . . . . . . . . . . . . . 54 3.2.3.Carryingoutexperimentsandcorpusstudies. . . . . . . . . 55 3.2.4.Specifyingtheprocessingprocesses . . . . . . . . . . . . . 58 3.2.5.Resourcewritinganddevelopment . . . . . . . . . . . . . . 59 3.2.6.Assessmentandscalability . . . . . . . . . . . . . . . . . . . 61 3.3.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Chapter4.ReusableSystemArchitectures . . . . . . . . . . . . . . . 63 4.1.Run-timearchitectures . . . . . . . . . . . . . . . . . . . . . . . . 64 4.1.1.Alistofmodulesandresources . . . . . . . . . . . . . . . . 64 4.1.2.Theprocessflow . . . . . . . . . . . . . . . . . . . . . . . . 66 4.1.3.Moduleinteractionlanguage . . . . . . . . . . . . . . . . . . 68 4.2.Design-timearchitectures . . . . . . . . . . . . . . . . . . . . . . 69 4.2.1.Toolkits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2.2.Middlewareforman–machineinteraction . . . . . . . . . . 71 4.2.3.Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 TableofContents vii PART 2. INPUTS PROCESSING . . . . . . . . . . . . . . . . . . . . . . 75 Chapter5.SemanticAnalysesandRepresentations . . . . . . . . . . 77 5.1.Languageindialogueandinman–machinedialogue . . . . . . . 78 5.1.1.Themaincharacteristicsofnaturallanguage . . . . . . . . . 78 5.1.2.Oralandwrittenlanguages . . . . . . . . . . . . . . . . . . . 82 5.1.3.Languageandspontaneousdialogue . . . . . . . . . . . . . 83 5.1.4.Languageandconversationalgestures . . . . . . . . . . . . 84 5.2.Computationalprocesses: fromthesignaltothemeaning . . . . 85 5.2.1.Syntacticanalyses . . . . . . . . . . . . . . . . . . . . . . . . 86 5.2.2.Semanticandconceptualresources . . . . . . . . . . . . . . 87 5.2.3.Semanticanalyses . . . . . . . . . . . . . . . . . . . . . . . . 89 5.3.Enrichingmeaningrepresentation . . . . . . . . . . . . . . . . . . 91 5.3.1.Attheleveloflinguisticutterance . . . . . . . . . . . . . . . 91 5.3.2.Atthelevelofmultimodalutterance . . . . . . . . . . . . . 93 5.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Chapter6.ReferenceResolution . . . . . . . . . . . . . . . . . . . . . 95 6.1.Objectreferenceresolution . . . . . . . . . . . . . . . . . . . . . 96 6.1.1.Multimodalreferencedomains . . . . . . . . . . . . . . . . 98 6.1.2.Visualsceneanalysis . . . . . . . . . . . . . . . . . . . . . . 100 6.1.3.Pointinggestureanalysis . . . . . . . . . . . . . . . . . . . . 101 6.1.4.Referenceresolutiondependingondetermination . . . . . . 102 6.2.Actionreferenceresolution . . . . . . . . . . . . . . . . . . . . . 105 6.2.1.Actionreferenceandverbalsemantics . . . . . . . . . . . . 105 6.2.2.Analyzingtheutterance“putthatthere” . . . . . . . . . . . 108 6.3.Anaphoraandcoreferenceprocessing . . . . . . . . . . . . . . . 109 6.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 Chapter7.DialogueActsRecognition . . . . . . . . . . . . . . . . . . 113 7.1.Natureofdialogueacts . . . . . . . . . . . . . . . . . . . . . . . . 114 7.1.1.Definitionsandphenomena . . . . . . . . . . . . . . . . . . 114 7.1.2.Theissuewithindirectacts . . . . . . . . . . . . . . . . . . 116 7.1.3.Theissuewithcompositeacts . . . . . . . . . . . . . . . . . 118 7.2.Identificationandprocessingofdialogueacts . . . . . . . . . . . 119 7.2.1.Actidentificationandclassification . . . . . . . . . . . . . . 119 7.2.2.Indirectandcompositeacts . . . . . . . . . . . . . . . . . . 121 7.3.Multimodaldialogueactprocessing . . . . . . . . . . . . . . . . 122 7.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 viii Man–MachineDialogue PART 3. SYSTEM BEHAVIOR AND EVALUATION . . . . . . . . . . . 125 Chapter8.AFewDialogueStrategies . . . . . . . . . . . . . . . . . . 127 8.1.Naturalandcooperativeaspectsofdialoguemanagement . . . . 128 8.1.1.Commongoalandcooperation . . . . . . . . . . . . . . . . 128 8.1.2.Speakingturnsandinteractiveaspects . . . . . . . . . . . . 130 8.1.3.Interpretationandinferences . . . . . . . . . . . . . . . . . . 132 8.1.4.Dialogue,argumentationandcoherence . . . . . . . . . . . 133 8.1.5.Choosingananswer. . . . . . . . . . . . . . . . . . . . . . . 135 8.2.Technicalaspectsofdialoguemanagement . . . . . . . . . . . . 136 8.2.1.Dialoguemanagementandcontrol . . . . . . . . . . . . . . 136 8.2.2.Dialoguehistorymodeling . . . . . . . . . . . . . . . . . . . 138 8.2.3.Dialoguemanagementandmultimodalitymanagement. . . 143 8.2.4.Canadialoguesystemlie? . . . . . . . . . . . . . . . . . . . 145 8.3.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Chapter9.MultimodalOutputManagement . . . . . . . . . . . . . . 149 9.1.Outputmanagementmethodology . . . . . . . . . . . . . . . . . 151 9.1.1.Generalprinciplesofoutputmultimodality . . . . . . . . . 151 9.1.2.Humanfactorsformultimediapresentation . . . . . . . . . 153 9.2.Multimediapresentationpragmatics . . . . . . . . . . . . . . . . 156 9.2.1.Illocutionaryforcesandvalues . . . . . . . . . . . . . . . . 156 9.2.2.Perlocutionaryforcesandvalues . . . . . . . . . . . . . . . 157 9.3.Processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 9.3.1.Allocationoftheinformationover communicationchannels . . . . . . . . . . . . . . . . . . . . 159 9.3.2.Redundancymanagementandmultimodalfission . . . . . . 161 9.3.3.Generationofreferringexpressions . . . . . . . . . . . . . . 162 9.3.4.Valorizingpartoftheinformationandtext tospeechsynthesis. . . . . . . . . . . . . . . . . . . . . . . . 163 9.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Chapter10.MultimodalDialogueSystemAssessment . . . . . . . . 167 10.1.Dialoguesystemassessmentfeasibility . . . . . . . . . . . . . . 168 10.1.1.Afewassessmentexperiments . . . . . . . . . . . . . . . . 170 10.1.2.Man–machineinterfacemethodologies . . . . . . . . . . . 172 10.1.3.Oraldialoguemethodologies . . . . . . . . . . . . . . . . . 174 TableofContents ix 10.1.4.Multimodaldialoguemethodologies. . . . . . . . . . . . . 175 10.2.Multimodalsystemassessmentchallenges . . . . . . . . . . . . 176 10.2.1.Globalassessmentorsegmentedassessment? . . . . . . . 176 10.2.2.Shouldamultimodalcorpusbemanaged? . . . . . . . . . 178 10.2.3.Canwecompareseveralmultimodalsystems? . . . . . . . 179 10.3.Methodologicalelements . . . . . . . . . . . . . . . . . . . . . . 180 10.3.1.Userexpertiseandsystemcomplexity . . . . . . . . . . . . 181 10.3.2.Questionnairesforusers . . . . . . . . . . . . . . . . . . . 183 10.3.3.ExtendingDQRandDCRtomultimodaldialogue . . . . . 185 10.3.4.Towardsotherassessmentmethods . . . . . . . . . . . . . 189 10.4.Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Preface The preparation of this book was carried out while preparing an accreditation to supervise research. This is a synthesis covering the past 10 years of research, since my doctorate [LAN 04], in the field of man–machine dialogue. The goal here is to outline the theories, methods, techniques and challenges involved in the design of computer programs that are able to understand and produce speech. This synthesis covers the presentation of important works in the field as well as a more personal approach, visible through the choice of the themes explored, for example. How can a machine talk, understand what is said and carry out a conversation close to natural conversation between two human beings? What are the design stages of a man–machine dialogue system? What are the understanding, thinking and interaction abilities expected from such systems? How should they be implemented? How can we get closer to the realistic and fluid aspect of humandialogue?Canadialoguesystemlie? These questions are at the origin of my path, which oscillated between linguistics and computer science, between pure research and development, between public and private research laboratories: INRIA, then Thales and currently the CNRS. These are also questions that second-year Masters students asked me during the man–machine dialogue class that I held at University Paris Diderot for a few years. Thus, this book draws inspiration in part from the class preparation and aims to be accessible to a public with linguistic and automatic language processing notions, and not necessarily withknowledgeoftheman–machinedialoguedomain. xii Man–MachineDialogue The goal here is to explain the main issues created by each stage of the design of a man–machine dialogue system, and to showa fewtheoretical and technicalpathsusedtodealwiththeseissues.Thepresentationwillnotcover all the wealth of existing works, but beyond that, it will aim to provide the readersaglimpseofthefield,whichmightmakethemwanttoknowmore. The goal here is to show that today there still is a French school of man–machine dialogue, which has been especially active in the past few years, even if it was at times a bit slower and at times it appeared that the man–machine dialogue was an aporia. The French school is characterized by its multidisciplinary approach, its involvement in different fields, such as system development (university prototypes, general public systems, as well as – and we tend to forget them since they are confidential – military systems), implementation of assessment methods and campaigns, and software architecture design. There is a French school for multimodal dialogue, for ergonomics, for embodied conversational agents, and even for theapplicationofmachinelearningtechniquestotheman–machinedialogue. Not all the links between these specialties are completely finalized, but the generaldynamicsareundeniableandencouraging. Asusualinresearchwork,whatispresentedinthisbookisindebtedtothe encouragement, advice and more generally speaking the sharing of an efficient and enjoyable work environment. For their institutional as well as scientific and human encouragement, I would like to thank Francis Corblin, Catherine Fuchs, Valérie Issarny, Jean-Marie Pierrel, Laurent Romary, Jean-PaulSansonnet,CatherineSchnedecker,JacquesSiroux,MariëtTheune, Bernard Victorri and Anne Vilnat. For the incredibly enriching Ozone experiment during my postdoctorate fellowship at INRIA, I would particularly like to thank Christophe Cérisara, Yves Laprie and especially Alexandre Denis on whom I was able to rely to implement a memorable demonstrator. For the equally memorable experiment of Thales R&T, I would like to thank, more specifically, Claire Fraboulet-Laudy, Bénédicte Goujon, Olivier Grisvard, Jérôme Lard and Célestin Sedogbo. For the wonderful workplace that is the Lattice laboratory, a Joint Research Unit of the CNRS, I would like to thank, without repeating those whom I have already mentioned, Michel Charolles for our very enriching exchanges on reference, Shirley Carter-Thomas and Sophie Prévost for the information structure, Thierry Poibeau and Isabelle Tellier for natural language processing, my successive colleagues Sylvain, Laure, Frédérique, as well as