The Effects of Eye Gaze and Emotional Facial Expression on the Allocation of Visual Attention Robbie Mathew Cooper Department of Psychology University of Stirling Submitted for the degree of Doctor of Philosophy July 2006 Abstract This thesis examines the way in which meaningful facial signals (i.e., eye gaze and emotional facial expressions) influence the allocation of visual attention. These signals convey information about the likely imminent behaviour of the sender and are, in turn, potentially relevant to the behaviour of the viewer. It is already well established that different signals influence the allocation of attention in different ways that are consistent with their meaning. For example, direct gaze (i.e., gaze directed at the viewer) is considered both to draw attention to its location and hold attention when it arrives, whereas observing averted gaze is known to create corresponding shifts in the observer’s attention. However, the circumstances under which these effects occur are not yet understood fully. The first two sets of experiments in this thesis tested directly whether direct gaze is particularly difficult to ignore when the task is to ignore it, and whether averted gaze will shift attention when it is not relevant to the task. Results suggest that direct gaze is no more difficult to ignore than closed eyes, and the shifts in attention associated with viewing averted gaze are not evident when the gaze cues are task-irrelevant. This challenges the existing understanding of these effects. The remaining set of experiments investigated the role of gaze direction in the allocation of attention to emotional facial expressions. Without exception, previous work looking at this issue has measured the allocation of attention to such expressions when gaze is directed at the viewer. Results suggest that while the type of emotional expression (i.e., angry or happy) does influence the allocation of attention, the associated gaze direction does not, even when the participants are divided in terms of anxiety level (a variable known to influence the allocation of attention to emotional expressions). These findings are discussed in terms of how the social meaning of the stimulus 2 can influence preattentive processing. This work also serves to highlight the need for general theories of visual attention to incorporate such data. Not to do so fundamentally risks misrepresenting the nature of attention as it operates out-with the laboratory setting. 3 Table of Contents Abstract................................................................................................................2 Acknowledgements..............................................................................................6 Publications..........................................................................................................7 Chapter 1 - General Introduction.........................................................................8 Control of Visual Attention..............................................................................11 Stimulus-Driven Control of Attention..............................................................13 Evidence for Faces Being ‘Special’..................................................................17 Psychological Evidence..............................................................................17 Neuropsychological Evidence....................................................................19 Neural Evidence..........................................................................................21 The Processing of Specific Facial Signals........................................................22 Eye Gaze.....................................................................................................23 Emotional Facial Expressions.....................................................................25 Interaction between Gaze and Emotion Processing..........................................27 Face Processing and Attention..........................................................................31 Eye Gaze and Attention....................................................................................36 Emotional Facial Expressions and Attention....................................................39 Gaze Direction, Emotional Facial Expressions, and Attention........................43 Thesis Overview...............................................................................................45 Chapter 2 - Effects of Distractor Congruency on Gaze Processing...............48 Experiment 1.....................................................................................................50 Experiment 2.....................................................................................................53 Experiment 3.....................................................................................................55 Experiment 4.....................................................................................................63 General Discussion...........................................................................................65 Chapter 3 - Effects of Task-Irrelevant Averted Gaze on Attention..................70 Experiment 5.....................................................................................................74 Experiment 6.....................................................................................................79 Experiment 7.....................................................................................................83 Experiment 8.....................................................................................................86 Experiment 9.....................................................................................................90 General Discussion...........................................................................................94 4 Chapter 4 - Effects of Eye Gaze and Emotional Expression on the Allocation of Attention to Faces........................................................................................ 101 Experiment 10.................................................................................................109 Experiment 11.................................................................................................122 Experiment 12.................................................................................................131 General Discussion.........................................................................................139 Chapter 5 - Effects of Facial Signals on the Allocation of Attention to Faces in Anxiety.............................................................................................................. 145 Experiment 13.................................................................................................152 Chapter 6 - General Discussion and Future Directions................................. 165 References........................................................................................................ 178 5 Acknowledgements Greatest thanks go to my supervisor Steve Langton for his support and advice throughout this project. As my second supervisor, I am also grateful to Vicki Bruce for sage council and a critical pair of eyes. This research was supported by an ESRC Postgraduate Training Award, PTA-030-2002-00724. 6 Publications Experiment 10 (Chapter 4) has been accepted for publication: Cooper, R. M. & Langton, S.R.H. (2006). Attentional bias to angry faces using the dot-probe task? It depends when you look for it. Behaviour Research and Therapy, 44, 1321 - 1329. 7 Chapter 1 General Introduction Our brains are constantly bombarded with information from our senses. For example, as you read the words on this page your brain is receiving not only the appropriate visual information on which you are trying to concentrate, but also other visual, auditory, olfactory, and tactile information from a huge variety of sources. The mechanism which serves to filter this information is called attention. Only a fraction of the incoming information is selected by attention for further processing (e.g., Broadbent, 1952; Sperling, 1960). The extent to which information from these potential sources is selected depends on at least two factors: 1) the behavioural relevance of the information to ongoing task goals and 2) the ability of that information to capture attention regardless of those goals. Consequently, information relevant to the current task is processed more deeply than task- irrelevant, unattended information (e.g., Treisman, 1960). However, the processing of task-irrelevant information increases markedly when that information is sufficiently salient (e.g., Moray, 1953; although see Harris, Pashler, & Coburn, 2004). This ability to process certain information regardless of task demands has been characterised as a “circuit breaker” of voluntary attention (Corbetta and Shulman, 2002) which ensures the processing of stimuli with potential behavioural significance. From an evolutionary perspective, an individual with a system that allocates attention to the appearance of sudden and unexpected items (e.g., an approaching predator) would be at an advantage compared with someone who could not disengage from their current activity. Arguably, the most biologically important (and in this sense, salient) stimuli we encounter regularly are faces (Ellis, 1981). Faces contain a large amount of 8 socially and evolutionarily relevant information such as, age, race, sex, attractiveness, emotion, and gaze direction. The importance of these stimuli is reflected in the existence of neural circuits dedicated to the processing of facial information (Farah, Wilson, Drain, & Tanaka, 1998; Perrett, Hietanen, Oram, & Benson, 1992). Furthermore there is evidence that the relationship between faces and attention is ‘special’ in the sense that faces draw attention to their location (Hershler & Hochstein, 2005) and are particularly difficult to ignore, even when they are not relevant to current goals (Jenkins, Lavie, & Driver, 2003). Thus, faces may be prioritised for processing by the attentional system compared to other visual stimuli. If the allocation of attention to faces is prioritised relative to other classes of visual stimuli it is reasonable to suppose that this allocation of attention might also depend on different characteristics of the face (e.g., emotional expression). This idea is supported by data which suggest that angry faces capture attention more efficiently than do happy or neutral faces (Öhman, Lundqvist, & Esteves, 2001). Furthermore, faces with gaze directed at the viewer (direct gaze) are thought to capture attention more efficiently than similar faces with averted gaze (von Grünau & Anston, 1995). These studies suggest that meaningful facial signals can influence the way in which attention is allocated to a face. Furthermore, once attention reaches the face, the specific signals that are being communicated can have very different implications for what attention does next. These signals are only understood fully with the ability to understand gaze direction (e.g., she is attracted to me; he is scared of something over there) and all have a signal-specific attention component attached to the viewer’s response (e.g., I should look back at the attractive person; I should also attend to the scary thing). 9 This thesis investigates the relationship between meaningful facial signals and visual attention. Throughout, the focus of the experiments in this thesis is on the role that eye gaze has in the allocation of attention to faces, both when they are free from emotional expression and when they express emotion. Given that such signals are so regularly encountered when interacting with other humans, their study will help illuminate the processes by which attention operates in such scenarios, something that has traditionally been neglected in the study of attention (Fox, 2005). This work also serves to highlight the need for general theories of visual attention to incorporate such data or risk fundamentally misrepresenting the nature of attention as it operates out-with the laboratory setting (Kingstone, Smilek, Ristic, Friesen, & Eastwood, 2003). Despite an increase in interest in this area over recent years there is still much that is not understood. For example, while eye gaze is thought to both draw (von Grünau & Anston, 1995) and shift (Langton & Bruce, 1999) attention, it is not known to what extent these effects are dependent on the stimulus features per se, or the relevance of those features to the observer’s goals. Two questions that arise from this are; 1) does direct gaze still draw attention if the observer’s task is to try and ignore it, and 2) does averted gaze still shift attention when it is irrelevant to the task? These questions are addressed in Chapters 2 and 3 respectively. Furthermore, work looking at how attention is allocated to emotional facial expressions has exclusively presented faces with eyes directed at the viewer (e.g., Öhman et al, 2001). This means that these previous studies have confounded the signal conveyed by emotional expression with the gaze signal that is also being communicated. As a result it is currently unclear whether any observed attentional effects related to the presentation of the face are a function of the emotional 10