Embodied Communication in Humans and Machines This page intentionally left blank Embodied Communication in Humans and Machines Edited by Ipke Wachsmuth, Manuela Lenzen, Günther Knoblich 1 1 Great Clarendon Street,Oxford OX2 6DP Oxford University Press is a department ofthe University ofOxford. It furthers the University’s objective ofexcellence in research,scholarship, and education by publishing worldwide in Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark ofOxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc.,New York © Oxford University Press,2008 The moral rights ofthe authors have been asserted Database right Oxford University Press (maker) First published 2008 All rights reserved.No part ofthis publication may be reproduced, stored in a retrieval system,or transmitted,in any form or by any means, without the prior permission in writing ofOxford University Press, or as expressly permitted by law,or under terms agreed with the appropriate reprographics rights organization.Enquiries concerning reproduction outside the scope ofthe above should be sent to the Rights Department, Oxford University Press,at the address above You must not circulate this book in any other binding or cover and you must impose this same condition on any acquirer A catalog record for this title is available from the British Library Data available Library ofCongress Cataloging in Publication Data Data available Typeset by Cepha Imaging Private Ltd.,Bangalore,India Printed in Great Britain on acid-free paper by Biddles Ltd.King’s Lynn,Norfolk ISBN 978–0–19–923175–1 10 9 8 7 6 5 4 3 2 1 While every effort has been made to ensure that the contents ofthis book are as complete,accurate and-up-to-date as possible at the date ofwriting,Oxford University Press is not able to give any guarantee or assurance that such is the case. Readers are urged to take appropriately qualified medical advice in all cases.The information in this book is intended to be useful to the general reader,but should not be used as a means ofself-diagnosis or for the prescription ofmedication. Contents Abbreviations vii Contributors ix 1 Introduction to embodied communication:why communication needs the body 1 Ipke Wachsmuth,Manuela Lenzen,and Günther Knoblich 2 Some boundary conditions on embodied agents sharing a common world 29 John Barresi 3 Toward a theory ofembodied communication:self-sustaining wild systems as embodied meaning 53 J.Scott Jordan 4 Synchrony and swing in conversation:coordination,temporal dynamics, and communication 75 Daniel C.Richardson,Rick Dale,and Kevin Shockley 5 The visual perception ofdynamic body language 95 Maggie Shiffrar 6 Mirrors for embodied communication 111 Wolfgang Prinz 7 The role ofthe mirror system in embodied communication 129 Natalie Sebanz and Günther Knoblich 8 Everything is movement:on the nature ofembodied communication 151 Elisabeth Oberzaucher and Karl Grammer 9 Communication and cooperation in living beings and artificial agents 179 Achim Stephan,Manuela Lenzen,Josep Call,and Matthias Uhl 10 Laborious intersubjectivity:attentional struggle and embodied communication in an auto-shop 201 Jürgen Streeck 11 The emergence ofembodied communication in artificial agents and humans 229 Bruno Galantucci and Luc Steels 12 Dimensions ofembodied communication—towards a typology ofembodied communication 257 Jens Allwood 13 Neurological disorders ofembodied communication 285 Elisabeth Ahlsén vi CONTENTS 14 Gestural imagery and cohesion in normal and impaired discourse 305 Susan Duncan 15 Conversational metacognition 329 Joëlle Proust 16 Imitation in embodied communication—from monkey mirror neurons to artificial humans 357 Stefan Kopp,Ipke Wachsmuth,James Bonaiuto,and Michael Arbib 17 Persuasion and the expressivity ofgestures in humans and machines 391 Isabella Poggi and Catherine Pelachaud 18 Implementing a non-modular theory oflanguage production in an embodied conversational agent 425 Timo Sowa,Stefan Kopp,Susan Duncan,David McNeill,and Ipke Wachsmuth 19 Towards a neurocognitive model ofturn taking in multimodal dialog 451 James Bonaiuto and Kristinn R.Thórisson Index 485 Abbreviations %REC percent recurrence ICM interactive communication ACA activity based communication management analysis IDFs image description features ACE articulated communicator engine IDT imagistic description tree ACQ augmented competitive queuing IFG inferior frontal gyrus aIPS anterior intraparietal sulcus IPT information processing theory APML affective presentation markup LIS Italian sign language language LMP local motor program BEAT behavior expression animation MCP motor control program toolkit MEI main evocative function BDAE Boston diagnostic aphasia MM main message examination MURML multimodal utterance representation CAS complex adaptive system markup language CC centripetal-centrifugal OCM own communication management CM communication management OVPT observer viewpoint CPU continue,perceive,understand PD Parkinson’s disease CQ competitive queuing PET positron emission tomography CRP cross-recurrence plot QoM quantity ofmovement CVPT character viewpoint REA real estate agent DST dynamical systems theory RP recurrence plot ECA embodied conversational agent RQA recurrence quantification analysis EEA environment ofevolutionary RT reaction-time adaptedness STS superior temporal sulcus EF embodied feedback STSp superior temporal sulcus,posterior FACS facial action coding system region fMRI functional magnetic resonance TD temporal difference imaging TMS transcranial magnetic stimulation GEMEP Geneva multimodal emotion portrayals TOM theory ofmind GP growth point TTS text-to-speech system hACQ hierarchical augmented competitive WAB western aphasia battery queuing WST wild systems theory This page intentionally left blank Contributors Elisabeth Ahlsén Susan Duncan Department ofLinguistics Psychology Department Göteborg University University ofChicago Göteborg Sweden Chicago,IL USA Jens Allwood Department ofLinguistics Bruno Galantucci Göteborg University Psychology Department Göteborg Yeshiva University Sweden New York,NY USA Michael Arbib Department ofComputer Science Karl Grammer Neuroscience Graduate Program Ludwig-Boltzmann-Institute for University ofSouthern California Urban Ethology Los Angeles,CA Vienna USA Austria John Barresi J.Scott Jordan Department ofPsychology Department ofPsychology Dalhousie University Illinois State University Halifax,Nova Scotia Normal,IL Canada USA James Bonaiuto Günther Knoblich Neuroscience Graduate Program School ofPsychology University ofSouthern California University ofBirmingham Los Angeles,CA Birmingham UK USA Stefan Kopp Josep Call Technische Fakultät Max Planck Institute for Evolutionary Universität Bielefeld Anthropology Bielefeld Department ofDevelopmental and Germany Comparative Psychology Manuela Lenzen Leipzig Germany Zentrum für interdisziplinäre Rick Dale Forschung (ZiF) Department ofPsychology Universität Bielefeld University ofMemphis Bielefeld Memphis,TN Germany USA