Intelligent Systems Reference Library 164 Laszlo Hunyadi István Szekrényes Editors The Temporal Structure of Multimodal Communication Theory, Methods and Applications Intelligent Systems Reference Library Volume 164 Series Editors Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland Lakhmi C. Jain, Faculty of Engineering and Information Technology, Centre for Artificial Intelligence, University of Technology, Sydney, NSW, Australia; Faculty of Science, Technology and Mathematics, University of Canberra, Canberra, ACT, Australia; KES International, Shoreham-by-Sea, UK; Liverpool Hope University, Liverpool, UK The aim of this series is to publish a Reference Library, including novel advances and developments in all aspects of Intelligent Systems in an easily accessible and well structured form. The series includes reference works, handbooks, compendia, textbooks,well-structuredmonographs,dictionaries,andencyclopedias.Itcontains well integrated knowledge and current information in the field of Intelligent Systems. The series covers the theory, applications, and design methods of IntelligentSystems.Virtuallyalldisciplinessuchasengineering,computerscience, avionics, business, e-commerce, environment, healthcare, physics and life science are included. The list of topics spans all the areas of modern intelligent systems such as: Ambient intelligence, Computational intelligence, Social intelligence, Computational neuroscience, Artificial life, Virtual society, Cognitive systems, DNA and immunity-based systems, e-Learning and teaching, Human-centred computing and Machine ethics, Intelligent control, Intelligent data analysis, Knowledge-based paradigms, Knowledge management, Intelligent agents, Intelligent decision making,Intelligent network security, Interactiveentertainment, Learningparadigms,Recommendersystems,RoboticsandMechatronicsincluding human-machine teaming, Self-organizing and adaptive systems, Soft computing including Neural systems, Fuzzy systems, Evolutionary computing and the Fusion of these paradigms, Perception and Vision, Web intelligence and Multimedia. ** Indexing: The books of this series are submitted to ISI Web of Science, SCOPUS, DBLP and Springerlink. More information about this series at http://www.springer.com/series/8578 á é Laszlo Hunyadi Istv n Szekr nyes (cid:129) Editors The Temporal Structure of Multimodal Communication Theory, Methods and Applications 123 Editors LaszloHunyadi István Szekrényes Department ofGeneral andApplied Institute of Philosophy Linguistics University of Debrecen University of Debrecen Debrecen, Hungary Debrecen, Hungary ISSN 1868-4394 ISSN 1868-4408 (electronic) Intelligent Systems Reference Library ISBN978-3-030-22894-1 ISBN978-3-030-22895-8 (eBook) https://doi.org/10.1007/978-3-030-22895-8 ©SpringerNatureSwitzerlandAG2020 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodologynowknownorhereafterdeveloped. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregard tojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland This book is dedicated to researchers and developers wishing to deepen their understanding of the complexities of human-human communication as found in multimodal interactions and to seek inspiration to apply specific patterns of behavior in building future smart systems at the crossroads of social sciences and engineering. Preface Themostwidelyacceptedclaimincontemporarymultimodalityresearchisthatthe different channels of communication complement each other. Sharing a similar view and further extending it, this book sets as its main goal to show that the perceivedasymmetrybetweentheverbalandnonverbalmarkersofcommunication can be reconciled by demonstrating that all these markers are part of a multimodal temporal structure in which their local alignment is just one constituent in a tem- porally distributed and hierarchically organized structure. By applying a variety of methodologies, including but not restricted to T-pattern analysis, the latter specif- icallydesignedforthestudyoftemporalpatternsinhumaninteractions,thepresent volumeintendstooffernewinsightsinthetheoryofmultimodalcommunicationas well as pragmatics in general and thus contribute to dialogue modeling both in human and robotic applications. The most researched challenge in applied multimodal communication is the development of multimodal user interfaces which are intended to make human-computerinteractionmorehuman-likeanduser-friendly.Forachievingthis goal,multimodaluserinterfaces(robotsorvirtualagents)havetobecapableofthe appropriate interpretation and generation of both verbal and nonverbal behavior (e.g., prosody and gestures) controlled by their communication engine. Research is highly interdisciplinary involving a number of fields, such as computerandcognitivescience,psychology,linguistics,biology,androbotics.The main research areas focus on the following questions: Motion Intelligence: How can we integrate perception with action to allow robots to move smartly and assist humans in unexpected, everyday environments? Attentive Systems: What mechanisms enable artificial systems to understand and actively focus on what is important, to ignore irrelevant detail and to share attention with humans? Situated Communication: How can we coherently coordinate language, per- ception,andactionsothatcooperationbetweenhumansandartificialsystemstakes place in natural efficiency? MemoryandLearning:Whatmemoryarchitecturesenableasystemtoacquire, store, and retrieve knowledge, and to improve its capabilities by learning? vii viii Preface There is a wide range of applications from industry to home, from health to security,eachhavingthemaingoalofuncovering,embedding,andimplementinga rich spectrum of knowledge about human behavior and cognition for both under- standing ourselves as humans in general, and the building of more effective human-machine interfaces. This volume offers a broad view on the theory, methodology,andpracticeofthestudyofmultimodalbehaviorthatcanbeapplied to a number of future applications. The authors are representatives of fields of research that can significantly contribute to the achievement of the above goals, coming from specific areas of psychology, computer science, speech science, pragmaticsandcomputationallinguistics.Thetopicspresentedinthebookinclude an overview of the theory of pragmatics related to multimodal communication, a methodology for the detection of structures in indirect observation, an experiment on how the involvement of a direct observer in a dialogue can be measured, a description of a methodology of discovering and applying paraverbal, kinetic behavioral patterns in an educational environment, the description of multimodal patterns of uncertainty in a dialogue, and how machine learning techniques can be applied tothe automatic recognition of topic change ina conversation. Time plays an important role throughout the book, demonstrating that behavior needs to be captured as a function of time across all modalities. Special features of the book: 1. It offers a comprehensive analysis of multimodal communication based on a variety of corpora, including one of the most extensively annotated corpora of dialogues (HuComTech). 2. It captures the temporal patterns of a wide range of behaviors discovered in human-human communication. 3. It presents work from several areas of communication research with all the benefits of interdisciplinarity. Debrecen, Hungary Laszlo Hunyadi January 2019 István Szekrényes Contents Part I Theoretical Overview of Multimodal Communication 1 Linguistic and Contextual Clues of Intentions and Perspectives in Human Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Enikő Németh T. 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Models of Human Verbal Communication . . . . . . . . . . . . . . . . . . 5 1.2.1 The Basic Features of Human Verbal Communication . . . . 5 1.2.2 Code Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.3 Inferential Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.4 Ostensive-Inferential Model . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.5 The Generative Theoretical-Technological Model . . . . . . . 12 1.3 Perspectives, Intentions and Their Clues in Verbal Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Part II Methods of Observation 2 The Teacher’s Body Communicates. Detection of Paraverbal Behaviour Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Marta Castañer and Oleguer Camerino 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.1.1 Introjection, Extension and Projection of the Teaching Discourse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.1.2 The Non-linearity of Human Movement . . . . . . . . . . . . . . 28 2.1.3 Paraverbal Communication and Body Language . . . . . . . . 28 2.1.4 From Kine to Gesture . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 ix x Contents 2.2 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.2.1 Pattern Analysis and the Systemic Approach. . . . . . . . . . . 32 2.2.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.2.3 Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2.4 Materials and Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3 Is It Possible to Perform “Liquefying” Actions in Conversational Analysis? The Detection of Structures in Indirect Observations. . . . 45 M. Teresa Anguera 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.2 Towards an Optimal Methodology for Conversational Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2.1 Systematic Observation . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.2.2 Basic Decisions in the Analysis of Turn-Taking . . . . . . . . 49 3.3 Methodological Process: Transcription, Use of Observation Instrument, Coding and Data-Quality Control. . . . . . . . . . . . . . . . 53 3.3.1 Transcription. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.3.2 Use of the Observation Instrument . . . . . . . . . . . . . . . . . . 54 3.3.3 Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.3.4 Data Quality Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.4 T-Pattern Detection in Conversational Analysis . . . . . . . . . . . . . . 58 3.5 What Do “LIQUEFYING” Actions in Conversational Analysis Involve?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4 Research Methods for Studying Daily Life: Experience Sampling and a Multilevel Approach to Study Time and Mood at Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Mariona Portell, Robin M. Hogarth and Anna Cuxart 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.2 The Challenge to Study Time and Mood at Work . . . . . . . . . . . . 71 4.3 The Role of Temporal Factors. . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.1 Diurnal Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.2 Day-of-the-Week Effects . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.3.3 Weather Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.3.4 Other Factors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.4 The Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.4.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.4.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.4.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77