The Pragmatic Turn Toward Action-Oriented Views in Cognitive Science Strüngmann Forum Reports Julia Lupp, series editor The Ernst Strüngmann Forum is made possible through the generous support of the Ernst Strüngmann Foundation, inaugurated by Dr. Andreas and Dr. Thomas Strüngmann. This Forum was supported by the Deutsche Forschungsgemeinschaft (German Science Foundation) The Pragmatic Turn Toward Action-Oriented Views in Cognitive Science Edited by Andreas K. Engel, Karl J. Friston, and Danica Kragic Program Advisory Committee: Andreas K. Engel, Karl J. Friston, Bernhard Hommel, Danica Kragic, Julia Lupp, Paul F. M. J. Verschure, and Gottfried Vosgerau The MIT Press Cambridge, Massachusetts London, England © 2015 Massachusetts Institute of Technology and the Frankfurt Institute for Advanced Studies This volume is the result of the 17th Ernst Strüngmann Forum, held March 16–21, 2014, in Frankfurt am Main, Germany. Series Editor: J. Lupp Assistant Editor: M. Turner Photographs: B. Fuge Lektorat: BerlinScienceWorks All rights reserved. No part of this book may be reproduced in any form by electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. The book was set in TimesNewRoman and Arial. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Names: Engel, Andreas K., editor. | Friston, K. J. (Karl J.), editor. | Kragic, Danica, editor. Title: The pragmatic turn : toward action-oriented views in cognitive science / Andreas K. Engel, Karl J. Friston, and Danica Kragic, eds. Description: Cambridge, MA : MIT Press, [2016] | Series: Strüngmann forum reports | Includes bibliographical references and index. Identifi ers: LCCN 2015048280 | ISBN 9780262034326 (hardcover : alk. paper) Subjects: LCSH: Cognition. | Sensorimotor integration. | Action theory. | Cognitive science. Classifi cation: LCC BF311 .P724 2016 | DDC 153--dc23 LC record available at http://lccn.loc.gov/2015048280 10 9 8 7 6 5 4 3 2 1 Contents The Ernst Strüngmann Forum vii List of Contributors ix 1 Introduction: Where’s the Action? 1 Andreas K. Engel, Karl J. Friston, and Danica Kragic Development, Acquisition, and Adaptation of Action-Oriented Processing 2 The Contribution of Pragmatic Skills to Cognition and Its 19 Development: Common Perspectives and Disagreements Giovanni Pezzulo 3 The Development of Action Cognition 35 Antonia F. de C. Hamilton, Victoria Southgate, and Elisabeth Hill 4 Acting Up: An Approach to the Study 49 of Cognitive Development Giovanni Pezzulo, Gottfried Vosgerau, Uta Frith, Antonia F. de C. Hamilton, Cecilia Heyes, Atsushi Iriki, Henrik Jörntell, Peter König, Saskia K. Nagel, Pierre-Yves Oudeyer, Robert D. Rupert, and Antonella Tramacere Action-Oriented Models of Cognitive Processing 5 Can Cognition Be Reduced to Action? Processes That 81 Mediate Stimuli and Responses Make Human Action Possible Lawrence W. Barsalou 6 The Mindful Filter: Free Energy and Action 97 Karl J. Friston 7 Prediction, Agency, and Body Ownership 109 Jakob Hohwy 8 Sensorimotor Contingencies and the Dynamical Creation 121 of Structural Relations Underlying Percepts Jürgen Jost 9 Language, Action, Interaction: Neuropragmatic Perspectives 139 on Symbols, Meaning, and Context-Dependent Function Friedemann Pulvermüller 10 Action-Oriented Models of Cognitive Processing: A 159 Little Less Cogitation, A Little More Action Please James Kilner, Bernhard Hommel, Moshe Bar, Lawrence W. Barsalou, Karl J. Friston, Jürgen Jost, Alexander Maye, Thomas Metzinger, Friedemann Pulvermüller, Marti Sánchez-Fibla, John K. Tsotsos, and Gabriella Vigliocco vi Contents Action-Oriented Understanding of Consciousness and the Structure of Experience 11 Extending Sensorimotor Contingencies to Cognition 175 Alexander Maye and Andreas K. Engel 12 What’s the Use of Consciousness? How the Stab 193 of Conscience Made Us Really Conscious Chris D. Frith and Thomas Metzinger 13 Pragmatism and the Pragmatic Turn in Cognitive Science 215 Richard Menary 14 Consciousness in Action: The Unconscious Parallel Present 235 Optimized by the Conscious Sequential Projected Future Paul F. M. J. Verschure 15 Action-Oriented Understanding of Consciousness 261 and the Structure of Experience Anil K. Seth, Paul F. M. J. Verschure, Olaf Blanke, Martin V. Butz, Judith M. Ford, Chris D. Frith, Pierre Jacob, Miriam Kyselo, Marek McGann, Richard Menary, Ezequiel Morsella, and J. Kevin O’Regan Implications of Action-Oriented Paradigm Shifts in Cognitive Science 16 Do We (or Our Brains) Actively Represent or 285 Enactively Engage with the World? Shaun Gallagher 17 Ways of Action Science 297 Wolfgang Prinz 18 Learning Action-Perception Cycles in Robotics: A 309 Question of Representations and Embodiment Jeannette Bohg and Danica Kragic 19 Action-Oriented Cognition and Its Implications: 321 Contextualizing the New Science of Mind Tony J. Prescott and Paul F. M. J. Verschure 20 Implications of Action-Oriented Paradigm 333 Shifts in Cognitive Science Peter F. Dominey, Tony J. Prescott, Jeannette Bohg, Andreas K. Engel, Shaun Gallagher, Tobias Heed, Matej Hoffmann, Günther Knoblich, Wolfgang Prinz, and Andrew Schwartz Bibliography 357 Subject Index 411 The Ernst Strüngmann Forum Science is a highly specialized enterprise—one that enables areas of enquiry to be minutely pursued, establishes working paradigms and normative standards, and supports rigor in experimental research. Some issues, however, do not fall neatly into the purview of a single disciplinary fi eld. Here, specialization can hinder conceptualization and limit the generation of potential problem-solving approaches. The Ernst Strüngmann Forum was created to address such topics. Founded on the tenets of scientifi c independence and the inquisitive nature of the human mind, the Ernst Strüngmann Forum is dedicated to the continual expansion of knowledge. Its activities promote interdisciplinary communica- tion on high-priority issues encountered in basic science. Through its innova- tive communication process, the Ernst Strüngmann Forum provides a creative environment within which experts scrutinize high-priority issues from mul- tiple vantage points. This process begins with the identifi cation of themes. By nature, a theme constitutes a problem area that transcends classic disciplinary boundaries—a topic of high-priority interest that requires concentrated, multidisciplinary pe- rusal. Proposals are received from leading scientists who are active in their fi eld and reviewed by an independent Scientifi c Advisory Board. Once approved, a steering committee is convened to refi ne the scientifi c parameters of the pro- posal and select participants. Approximately one year later, a central gathering, or Forum, is held to which circa forty experts are invited. Expansive discourse is employed to approach the problem. Often, this necessitates reexamining long-established ideas and relinquishing conventional perspectives, yet when accomplished, new insights begin to emerge. The resultant ideas and newly gained perspectives from the entire process are communicated to the scientifi c community for further consideration and implementation. Preliminary discussion for this theme began in 2012, based on an ongoing “pragmatic turn” in cognitive science: one moving away from the traditional representation-centered framework toward understanding cognition as “enac- tive.” Although action-oriented views have been present in the individual fi elds of cognitive science, neuroscience, psychology, robotics, and philosophy of mind, strong links across domains were lacking. Thus, need was expressed for a collective discussion to confront different approaches and types of data, to elaborate key concepts, and to facilitate interdisciplinary interactions. From October 25–27, 2013, the Program Advisory Committee (Andreas Engel, Karl Friston, Bernhard Hommel, Danica Kragic, Julia Lupp, Paul Verschure, and Gottfried Vosgerau) met to defi ne the scientifi c framework for this Forum, which was held in Frankfurt am Main from October 26–31, 2014. This volume communicates the synergy that emerged from a very diverse group of experts and is comprised of two types of contributions. Background viii The Ernst Strüngmann Forum information is provided on key aspects of the overall theme. These chapters, drafted before the Forum, were reviewed and subsequently revised. In addi- tion, Chapters 4, 10, 15, and 20 report on the extensive discussions of the working groups. These chapters are not consensus documents; they transfer the essence of the multifaceted discourse, expose areas where opinions diverge, and highlight topics in need of future enquiry. An endeavor of this kind creates its own unique group dynamics and puts demands on everyone who participates. Each invitee played an active role and embraced the process with a willingness to probe beyond that which is evi- dent. For their efforts and commitment, I am grateful to all. A special word of thanks goes to the Program Advisory Committee, to the authors and reviewers of the background papers, as well as to the moderators of the individual work- ing groups (Gottfried Vosgerau, Bernhard Hommel, Paul Verschure, and Tony Prescott). The rapporteurs of the working groups (Giovanni Pezzulo, James Kilner, Anil Seth, and Peter Dominey) deserve special recognition: to draft a report during the Forum and fi nalize it in the months thereafter is no simple matter. Most importantly, I extend my sincere appreciation to Andreas Engel, Karl Friston, and Danica Kragic: as chairpersons of this 18th Ernst Strüngmann Forum, their commitment ensured a most vibrant intellectual gathering. A communication process of this nature relies on institutional stability and an environment that encourages free thought. The generous support of the Ernst Strüngmann Foundation, established by Dr. Andreas and Dr. Thomas Strüngmann in honor of their father, enables the Ernst Strüngmann Forum to pursue its work in the service of science. In addition, the following valu- able partnerships are gratefully acknowledged: the Scientifi c Advisory Board, which ensures the scientifi c independence of the Forum; the German Science Foundation, for its supplemental fi nancial support; and the Frankfurt Institute for Advanced Studies, which shares its intellectual setting with the Forum. Long-held views are never easy to put aside. Yet, when this is achieved, when the edges of the unknown begin to appear and the resulting gaps in knowledge are able to be identifi ed, the act of formulating strategies to fi ll such gaps becomes a most invigorating activity. On behalf of everyone involved, I hope that this volume will convey a sense of this lively exercise and further the emergence of a novel understanding of cognition. Julia Lupp, Program Director Ernst Strüngmann Forum Frankfurt Institute for Advanced Studies (FIAS) Ruth-Moufang-Str. 1, 60438 Frankfurt am Main, Germany https://esforum.de/ List of Contributors Bar, Moshe The Gonda Brain Research Center, Bar-Ilan University, Israel Barsalou, Lawrence W. Institute of Neuroscience and Psychology, University of Glasgow, Glasgow G12 8QB, U.K. Blanke, Olaf Laboratory of Cognitive Neuroscience, Center for Neuroprosthetics, Swiss Federal Institute of Technology (EPFL), 1202 Geneva, Switzerland Bohg, Jeannette Autonomous Motion Department, Max Planck Institute for Intelligent Systems, 72076 Tübingen, Germany Butz, Martin V. Cognitive Modeling, Department of Computer Science and Department of Psychology, Faculty of Science, University of Tübingen, 72076 Tübingen, Germany Dominey, Peter F. Inserm U846, Stem Cell and Brain Research Institute, Human and Robot Cognitive Systems, 69675 Bron Cedex, France Engel, Andreas K. Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, 20246 Hamburg, Germany Ford, Judith M. San Francisco Veterans Administration Medical Center and University of California, San Francisco, U.S.A. Friston, Karl J. Wellcome Trust Centre for Neuroimaging, Institute of Neurology, UCL, London WC1N 3BG, U.K. Frith, Chris D. Wellcome Trust Centre for Neuroimaging at University College London, London WC1N 3BG, U.K., and Interacting Minds Centre, Aarhus University, 8000 Aarhus C, Denmark Frith, Uta Institute of Cognitive Neuroscience, University College London, London WC1N 3AR, U.K. Gallagher, Shaun Department of Philosophy, University of Memphis, Memphis, TN 38152, U.S.A. and Faculty of Law, Humanities and the Arts, University of Wollongong, Australia Hamilton, Antonia F. de C. Institute of Cognitive Neuroscience, University College London, London, WC1N 3AR, U.K. Heed, Tobias Faculty of Psychology and Human Movement Science, University of Hamburg, 20146 Hamburg, Germany Heyes, Cecilia All Souls College, University of Oxford, Oxford OX1 4AL, U.K. Hill, Elisabeth Department of Psychology, Goldsmiths College, University of London, London SE14 6NW, U.K. Hoffmann, Matej iCub Facility, Istituto Italiano di Tecnologia, 16163 Genoa, Italy Hohwy, Jakob Cognition and Philosophy Lab, Monash University, Clayton, VIC 3800, Australia