The Oxford Handbook of Perceptual Organization The Oxford Handbook of Perceptual Organization Edited by Johan Wagemans 1 1 Great Clarendon Street, Oxford, OX2 6DP, United Kingdom Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide. Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries © Oxford University Press 2015 The moral rights of the author have been asserted First Edition published in 2015 Impression: 1 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, by licence or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above You must not circulate this work in any other form and you must impose this same condition on any acquirer Published in the United States of America by Oxford University Press 198 Madison Avenue, New York, NY 10016, United States of America British Library Cataloguing in Publication Data Data available Library of Congress Control Number: 2014955474 ISBN 978–0–19–968685–8 Printed and bound by CPI Group (UK) Ltd, Croydon, CR0 4YY Oxford University Press makes no representation, express or implied, that the drug dosages in this book are correct. Readers must therefore always check the product information and clinical procedures with the most up-to-date published product information and data sheets provided by the manufacturers and the most recent codes of conduct and safety regulations. The authors and the publishers do not accept responsibility or legal liability for any errors in the text or for the misuse or misapplication of material in this work. Except where otherwise stated, drug dosages and recommendations are for the non-pregnant adult who is not breast-feeding Links to third party websites are provided by Oxford in good faith and for information only. Oxford disclaims any responsibility for the materials contained in any third party website referenced in this work. Foreword Stephen E. Palmer The topic of perceptual organization typically refers to the problems of how the visual informa- tion is structured into qualitatively distinct elements over time and space during the process of perceiving and how that structuring influences the visual properties observers experience. Corresponding work on analogous topics in other sensory modalities is also an active area of research (see Section 7), but the vast majority of the literature concerns perceptual organization in vision (as reflected in the rest of the volume). If one grants that the smallest, lowest-level visual elements are likely to be the outputs of retinal receptors and that the largest, highest level ele- ments are the consciously experienced, meaningful environmental scenes and events that human observers use to plan and execute behaviors in their physical and social environments, then the fundamental question of perceptual organization is nothing less than this: how does the visual system manage to get from locally meaningless receptor outputs to globally meaningful scenes and events in the observer’s perceived environment? When stated in this way, the field of percep- tual organization encompasses most of human perception, including the perception of groups, patterns, and textures (Section 2), contours and shapes (Section 3), figures, grounds, and depth (Section 4), surfaces and colors (Section 5), motion and events (Section 6), as well as analogous issues in other sensory modalities (Section 7). (The present volume also includes two further sections on topics that have evolved from the material covered in Sections 2-7, one on special- ized topics (Section 8) and another on practical applications (Section 9).) Indeed, nearly the only aspects of perception typically excluded from discussions of perceptual organization are very low- level sensory processing (such as detecting lines and edges) and very high-level pattern recogni- tion (such as recognizing objects and scenes). This division has led to a somewhat unfortunate and uninformative classification of vision into low-level, mid-level, and high-level processing, with perceptual organization being identified with mid-level processing: essentially, whatever is left over between basic sensory processing and pattern recognition of known objects and scenes. Even so, some topics are more closely associated with the field of perceptual organization than others, and the ones represented in this volume constitute an excellent sample of those topics. Perceptual organization not only spans a wide array of empirical phenomena in human vision, but the approaches to understanding it encompass four distinct, but tightly interrelated domains: phenomenology, physiology, ecology, and computation. Phenomenology concerns the conscious appearance of the visible world, seeking to answer questions about the structural units of visual experience (e.g., regions, surfaces, and volumetric objects) and the properties people experience as defining them (e.g., their colors, shapes, sizes and positions). Physiology (i.e., neuroscience) concerns how neural events in the brain produce these experiences of perceived elements and properties, addressing the problem of how the brain achieves that organization of visual experi- ences. Ecology concerns the relation between observers and their environments (including physi- cal, social, and cultural aspects), attempting to determine why the world is experienced in terms of these units rather than others and why the brain processes the corresponding sensory informa- tion in the way it does. Computation concerns formal theories of how perceptual organization vi ForEWord might be achieved by the processing of information at a more abstract level than that of physi- ological mechanisms in the brain. Computation thus provides a theoretical interlingua in which the other three domains can potentially be related to each other. All four domains are crucial in understanding perceptual organization and are mentioned throughout this volume. They are also addressed quite explicitly in the final, theoretical section (Section 10). The topic of perceptual organization in vision has a fascinating, roller-coaster history that is relevant to understanding the field. Until the late 19th and early 20th centuries, organizational issues in vision, at least as they are currently considered, were virtually nonexistent. The reason is that the dominant theoretical paradigm in18th century philosophy came from British empiri- cists, such as Locke, Berkeley, and Hume, who proposed that high-level perceptions arose from a mechanistic, associative process in which low-level sensory atoms — i.e., primitive, indivisible, basic elements (akin to the outputs of retinal receptors) — evoked other sensory atoms that were linked together in memory due to repeated prior joint occurrences. The result of these activated associations, they believed, was the perception of meaningful objects and scenes. This atomistic, associative view, which became known as “Structuralism” in the hands of 19th century psycholo- gists, such as Wundt and Titchener, includes no interesting role for structure between low-level sensory atoms and high-level perceptions, as if the latter arose from unstructured concatenations (or “summative bundles”) of the appropriate sensory atoms. The theoretical landscape became more interesting in the late 19th century with the develop- ment of philosophical phenomenology (see Chapter 2), in which the structure of internal experi- ences was ascribed a much more important role. Phenomenologists, such as Brentano, Husserl, and Merleau-Ponty, analyzed the subjective organization and content of internal experiences (i.e., the appearance of perceptual objects) into a sophisticated taxonomy of parts and wholes. The development of such ideas in the hands of philosophers and early psychologists eventually led to the seminal singularity in the history of perceptual organization: the advent of the Gestalt revolution in the early 20th century. “Gestalt” is a German word that can roughly be translated as “whole-form” or “configuration,” but its meaning as the name for this school of psychology goes considerably beyond such superficial renderings because of its deep theoretical implications. Gestalt psychology was nothing less than a revolutionary movement that advocated the over- throw of Structuralism’s theoretical framework, undermining the assumptions of both atomism and associationism. Following important earlier work by von Ehrenfels on the emergent quali- ties of melodies, Gestalt psychologists, most notably including Wertheimer, Köhler and Koffka, argued forcefully against the Structuralist views of Wundt and his followers, replacing their claims about atomism and associationism with the opposing view that high-level percepts have intrinsic emergent structure in which wholes are primary and parts secondary, the latter being determined by their relations to and within the whole. This viewpoint is often expressed through the well- known Gestalt rallying cry that “the whole is different from the sum of its parts.” Indeed, it was only when the Gestaltists focused attention on the nature and importance of part-whole organiza- tion that it was recognized as a significant problem for the scientific understanding of vision. It is now a central – though not yet well understood – topic, acknowledged by virtually all perceptual scientists. The historical evolution of the Gestalt approach to perceptual organization is described in scholarly detail in Chapter 1. Gestalt psychologists succeeded in demolishing the atomistic, associative edifice of Structuralism through a series of profound and elegant demonstrations of the importance of organization in visual perception. Indeed, these demonstrations, which Koenderink (Chapter 3) calls “compelling visual proofs,” were so clear and definitive that they required only a solid consensus about the subjective experiences of perceivers when viewing the examples, usually ForEWord vii without reporting quantitative measurements. Their success is evident in the fact that many of these initial demonstrations of organizational phenomena have spawned entire fields of subsequent research in which more sophisticated, objective, and quantitative research meth- ods have been developed and employed (see Chapter 3). Indeed, the primary topic of this handbook is the distillation of current, cutting-edge knowledge about the phenomenologi- cal, physiological, ecological, and computational aspects of perceptual organization that have been achieved using these modern methods. Research on the initial organizational phenomena discovered by Gestalt psychologists, such as grouping (Chapter 4), apparent motion (Chapter 23), and other forms of organization in motion and depth (Chapter 25), got off to a quick start, impelled largely by their crucial role in undermin- ing the Structuralist dogma that held sway during the early 20th century, especially in Europe. (The Gestalt approach was not as successful in the US, largely because American psychology was mired in theoretical and methodological Behaviorism.) Indeed, Gestalt theorists advanced some claims about alternatives to Structuralism that were quite radical. Among them were Köhler’s claims that the brain is a “physical Gestalt” and that it achieves perception through electrical brain fields that interact dynamically to minimize physical energy. Gestalt theorizing encountered resistance partly because it went against the accepted consensus that science makes progress by analyzing complex entities into more elementary constituents and the interactions among them, a claim explicitly rejected by Gestalt theorists. More importantly, however, acceptance of Gestalt theory plummeted when Köhler’s electrical field hypothesis was tested physiologically and found to be inconsistent with the results (see Chapter 1 for details). The wholesale rejection of Gestalt ideas that followed was an unfortunate example of throwing the baby out with the bathwater. The poorly understood problem is that Gestalt theory was (and is) much more general and abstract than Köhler’s electrical field theory or indeed any other par- ticular implementation of it (see Palmer, 2009, for further explanation). For example, one of the most central tenets of Gestalt theory is the principle of Prägnanz (or simplicity), which claims that the organization of the percept that is achieved will be the simplest one possible given the available stimulation. That is, the visual system attempts both to maximize the “goodness-of-fit” between the sensory data and the perceptual interpretation and to minimize the perceptual interpretation’s complexity (see Chapters 50 and 51). Köhler identified complexity with the energy of the electri- cal brain field, which tends naturally toward a minimum in dynamic interaction within a physical Gestalt system, which he claimed the brain to be. It is tempting to suppose that if electrical field theory is incorrect, as implied by the results of experiments, then Gestalt theory in general must be incorrect. However, subsequent analyses have shown, for example, that certain classes of neural networks with feedback loops exhibit behavior that is functionally isomorphic to that of energy minimization in electrical fields. If perception is achieved by activity in such recurrent networks of neurons, then Gestalt theory would be vindicated, even though Köhler’s electrical field conjec- ture was incorrect. An equally important factor in the stagnation of research on perceptual organization was the advent of World War II, which turned attention and resources away from scientific enterprises unrelated to the war effort and sent many prominent German Gestaltists into exile in the US. The Gestalt movement retained a significant prominence in Italy, however, where psychologists such as Musatti, Metelli, and Kanizsa kept the tradition alive and made significant discoveries concern- ing the perception of transparency (Chapters 20 and 22) and contours (Chapters 10–12). Other important findings about perceptual organization were made by Michotte (in Leuven, Belgium), whose analysis of the perception of causality challenged the long-held philosophical belief that causality was cognitively inferred rather than directly perceived. These and other contributions to viii ForEWord the phenomena of perceptual organization kept the field alive, but the period from the 1940s to the 1960s was a nadir for research in this field. A variety of forces converged since the 1960s to revitalize interest in perceptual organization and bring it into the mainstream of the emerging field of vision science. One was the use of mod- ern, quantitative methods to understand and extend classic Gestalt phenomena. These include both direct psychophysical measures of organization (e.g., verbal reports of grouping) and visual features (e.g., surface lightness) and indirect measures of performance in objective tasks (e.g., reaction time measures of interference effects). Among the many important examples of such research are Wallach’s and Gilchrist’s contributions to understanding lightness constancy, Rock’s work on reference frames in shape perception, Palmer’s studies of new grouping principles and measures, Kubovy’s quantitative laws for integrating multiple grouping principles, Peterson’s exploration of the role of past experience in figure-ground organization, Navon’s work on global precedence, and Pomerantz’s research into configural superiority effects. Such empirical findings intrigued a new generation of vision scientists, who failed to find low-level sensory explanations of them – hence the invention of the term “mid-level vision.” A second force was the healthy desire to shore up the foundations of Gestalt theory by formalizing and quantifying the Gestalt principle of Prägnanz. This enterprise was advanced considerably by seminal contributions from Attneave, Hochberg, Garner, Leeuwenberg, van der Helm, and others who applied concepts from informa- tion theory and complexity theory to phenomena of perceptual organization. A third force that eventually began to have an effect was the study of the neural mechanisms of organization. Hubel and Wiesel revolutionized sensory physiology by discovering that the receptive fields of neurons in visual cortex corresponded to oriented line- and edge-based structures. Their results and the explosion of physiological research that followed is not generally discussed as being part of the field of perceptual organization – rather, it is considered “low-level vision” – but it surely can be viewed that way, as it specifies an early level of structure between retinal receptor outputs and high-level perceptual interpretations. Subsequent neuroscientific research and theory by pioneers such as von der Heydt, Lamme, von der Marlsburg, and van Leeuwen addressed higher-level structure involved in figure-ground organization, subjective (or illusory) contours, and grouping. A fourth converging force was the idea that perception – indeed, all psychological processes – could be modeled within an abstract computational framework. This hypothesis can ultimately be traced back to Turing, but its application to issues of visual organization is perhaps most clearly represented by Marr’s influential contributions, which attempted to bridge subjective phenom- ena with ecological constraints and neural mechanisms through computational models. More recently, Bayesian approaches to the problem of perceptual organization are having an increas- ing impact on the field due in part to their generality and compatibility with hypotheses such as Helmholtz’s likelihood principle and certain formulations of a simplicity principle. Many of the theoretical discussion in this volume are couched in computational terms, and it seems almost certain that computational theory will continue to loom large in future efforts to understand per- ceptual organization. The present volume brings together all of these diverse threads of empirical and theoretical research on perceptual organization. It will rightly be considered a modern landmark in the com- plex and rapidly evolving history of the field of perceptual organization. It follows and builds upon two extensive scholarly review papers that were published exactly 100 years after Wertheimer’s landmark 1912 article on the phi phenomenon that launched the Gestalt movement (see Wagemans Elder, Kubovy, Palmer, Peterson, Singh, & von der Heydt, 2012; Wagemans, Feldman, Gepshtein, Kimchi, Pomerantz, van der Helm, & van Leeuwen, 2012). The 51scholarly chapters it contains are authored by world-renown researchers and present comprehensive, state-of-the-art ForEWord ix reviews about how perceivers arrive at knowledge about meaningful external objects, scenes, and events from the meaningless, ambiguous, piecemeal evidence registered by sensory receptors. This perceptual feat is nothing short of a miracle, and although we do not yet understand how it is accomplished, we know a great deal more than was known a century ago when the enterprise began in earnest. This handbook is thus equally suitable for students who are just beginning to explore the literature on perceptual organization and for experts who want definitive, up-to-date treatments of topics with which they are already familiar. And it is, above all, a fitting tribute to the founding of an important field of scientific knowledge that was born a century ago and the quite remarkable progress scientists have made in understanding it during that time. Stephen E. Palmer Professor of the Graduate School Psychology & Cognitive Science University of California, Berkeley, CA U.S.A. References Palmer, S. E. (2009). Gestalt theory. In Bayne, T., Cleeremans, A., & Wilken, P. (Eds.). (2009). The Oxford Companion to Consciousness (pp. 327–330). Oxford, U.K.: Oxford University Press. Wagemans, J., Elder, J. H., Kubovy, M., Palmer, S. E., Peterson, M. A., Singh, M., & von der Heydt, R. (2012). A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure–ground organization. Psychological Bulletin, 138(6), 1172–1217. Wagemans, J., Feldman, J., Gepshtein, S., Kimchi, R., Pomerantz, J. R., van der Helm, P. A., & van Leeuwen, C. (2012). A century of Gestalt psychology in visual perception: II. Conceptual and theoretical foundations. Psychological Bulletin, 138(6), 1218–1252.