ASSESSMENT OF FACIAL EXPRESSION ASYMMETRIES UTILIZING DIGITIZED IMAGE ANALYSIS AND IMPRESSIONISTIC RATINGS By CHARLES K. RICHARDSON A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE DEGREE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 1996 TABLE OF CONTENTS page ABSTRACT iv INTRODUCTION 1 Defining Emotion 1 Neuropsychology of Emotion 6 Evaluation of Nonverbal Emotional Signals H7 Valence Hypothesis Expression of Emotional Signals 14 Methodological Issues in Facial Expression Research... 15 Production of Facial Expressions 24 Spontaneous Expressions in Normals: Laterality Studies 25 • Spontaneous Expressions in Brain-Impaired Subjects.... 28 Posed Expressions in Brain-Impaired Subjects 30 Posed Expressions in Normals 31 Morphological Variation 37 Methodological Differences 41 GENERAL ISSUES ADDRESSED 56 HYPOTHESES ^4 METHOD 6^ Overview ^^ Sub ects 66 j Stimulus Generation 67 Stimulus Preparation 69 Equipment ^0 Research Assistants 72 Digitization of Pixel Intensities 74 Subjective Ratings 75 Raters ^6 Ratings Procedure ^8 RESULTS "79 Laterality of Emotional Expressions based on Entropy Score 80 Laterality of Emotional Expressions based on Asymmetry Score 82 ii Influence of Lighting on Expressive Asymmetries 85 Emotional Expression Laterality with Correction for Lighting ^^ Emotional Laterality based on Adjusted Asymmetry Score 91 Asymmetry Scores for Additional Emotional Expressions 94 Nonemotional Laterality based on Adjusted Entropy Score ^^ Nonemotional Laterality based on Adjusted Asymmetry Score 100 Adjusted Entropy Scores of Additional Expressions 101 Adjusted Asymmetry Scores of Additional Expressions... 105 Association between Emotional and Non-emotional Expressions 10^ Non-parametric Tests of Laterality 109 Subjective Ratings of Expressive Asymmetries 110 Effect of Lighting Bias on Subjective Ratings Ill Agreement between Digitized Data and Subjective Ratings 112 Vertical Focus of Subjective Ratings 113 Effect of Training on Subjective Ratings 115 DISCUSSION. 118 Asymmetries in the Lower Face: Valence Dependent 122 Asymmetries in the Upper Face: Right Upper Bias 129 Relationship between Digitized Data and Subjective Ratings 141 Methodological Issues and the Subjective Ratings 144 CONCLUSION AND FUTURE DIRECTIONS 150 Evidence Supporting the Valence Hypothesis 151 Evidence Supporting the Right Hemisphere Hypothesis... 153 Evidence Supporting the Facial Mobility Hypothesis 154 Multiple Factors Contributing to Expressive Biases.... 156 Methodological Issues and Future Directions 158 APPENDIX 162 REFERENCES 166 BIOGRAPHICAL SKETCH 181 111 Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy ASSESSMENT OF FACIAL EXPRESSION ASYMMETRIES UTILIZING DIGITIZED IMAGE ANALYSIS AND IMPRESSIONISTIC RATINGS By Charles K. Richardson December 1996 Chair: Dawn Bowers Cochair: Russell M. Bauer Major Department: Clinical and Health Psychology Facial expressions encompass a prominent aspect of emotional behavior. Prior research has generally found that the left hemiface is more expressive than the right hemiface while posing emotional expressions. There are three major proposals that account for expressive asymmetries and they are the Right Hemisphere, Valence and Facial Mobility hypotheses. Research reporting expressive asymmetries has primarily relied upon subjective ratings, while studies failing to replicate these findings have generally utilized a more objective measurement system. In an effort to find an "objective" method which has the sensitivity to uncover possible facial asymmetries, digitized image analysis has been proposed as an alternative approach. In this study, digitized image analysis was used IV to scrutinize videotaped images of posed facial expressions. Facial movement was inferred from changes in pixel intensities during facial expressions. The degree of movement for each hemiface was measured during emotional and nonemotional expressions. Subjective ratings were also obtained and compared with the digitized image analysis. The primary questions of interest were whether hemiface asymmetries in dynamic expressions would be present using digitized image analysis and to what degree are subjective ratings consistent with the results of the digitized data. Other major questions of interest were whether expressive biases conform to our current knowledge of neuroanatomy and which of the major competing hypotheses would be most consistent with the findings of the digitized analysis. The results indicated the presence of hemifacial expression asymmetries using digitized image analysis. The pattern of lateral biases differed depending upon the portion of the face examined. In the lower face, negative expressions were left-biased and the positive expression displayed a right-sided trend. The nonemotional expressions did not suggest a clear pattern of asymmetry. In the upper face, nearly all of the emotional and nonemotional expressions were right-biased. Expressive asymmetries were more consistent with the Facial Mobility hypothesis in the upper face and the Valence hypothesis in the lower face. In contrast, the subjective ratings indicated that no emotional expression was asymmetrical. There were significant methodological weaknesses associated with the subjective ratings in this study. The discussion focuses on the implications of the digitized data. VI INTRODUCTION Defining Emotion The study of emotion has been an integral aspect of psychology and philosophy since the beginnings of these disciplines. One of the most enduring contributions of the founding father of American psychology, William James, is his theory of emotions. Despite the attention emotion has received, a consensus has yet to be developed regarding the definition of emotion. Most definitions tend to include four major components: physiological arousal, motor activity, subjective feeling and an evaluative capacity (Borod, 1993; Bowers, Bauer and Heilman, 1993). Fehr and Russell (1984) noted, however, that attempts to incorporate these four components within a definition has failed to produce one which precludes other cognitive processes, such as attitudes or motives. Consequently, they concluded that a classical definition of emotion is likely unattainable. Despite the difficulty of delineating an airtight definition of emotion, vigorous scientific investigation has continued. In general, the question has been addressed by focusing on one or more of the aforementioned components of physiological arousal, motor activity, subjective feeling and evaluative ability. While there may be substantial 2 agreement concerning these features, the role of cognition is hotly debated. For example, Zajonc (1980) has proposed a distinct emotional system that does not require cognitive processes to be activated. He further asserts that the affective reactions occur more rapidly than the more analytical, cognitive processes. The ideas of Zajonc and other similarly minded theorists share some aspects of the James (1892) view of emotions which emphasizes the role of primitive perceptual analysis leading to autonomic and motor responses. Zajone's views (1980), however, differ from those of James (1892) in that subjective emotional experience is posited to occur almost simultaneously with the rapid affective perceptual system. James (1892) on the other hand, argued that , subjective feeling developed after one sensed a change in one's autonomic system. In this regard, James' (1892) theory is more consistent with the proposals that favor a cognitive role in emotional experience (Schacter and Singer, 1962; Lazarus, 1990). They suggest that there cannot be emotion without cognition, and thus, a separate entity of emotion is not possible. For instance, Schacter and Singer (1962) have asserted that cognitive attributions must exist prior to the subjective feeling of emotion, and it is through the feedback process between cognition, physiological arousal and motor activity that the subjective feeling of emotion is experienced. In support of their proposition, they reported an experiment in which subjects injected with epinephrine experienced emotion only if an emotional stimulus was present in the environment. They argued that one's physiological arousal must be attributed to an emotional stimulus in order for the subjective experience of emotion to develop (Schacter and Singer, 1962) Consistent with . this viewpoint is evidence suggesting that cognitive decisions are made more rapidly than ones requiring an affective response. In a review of a series of studies, Feyereisen (1989) concluded that when examining a set of stimuli, subjects are slower at making affective decisions than cognitive ones, a conclusion which contradicts Zajone's (1980) position that the affective system operates more rapidly than the cognitive system. On the other hand, there are several lines of evidence that are inconsistent with the cognitive perspective on emotion. For example, several studies have described seizure patients who experience the sensation of an emotional state, usually fear, at the initial stages of a seizure (Strauss, 1989) The fact that these patients . experience other emotions besides fear suggests that this phenomena is not the result of their sensing an impending seizure. Furthermore, seizure patients who have had their amygdala electrically stimulated have also described various emotional sensations (Halgren, Walter, Cherlow and Crandall, 4 1978; Gloor, Olivier, Quesney, Anderman and Horowitz, 1982). In addition, single cell recordings in the amygdala of primates have found that certain cells are particularly sensitive to emotional stimuli, or more specifically, emotional faces (Leonard, Rolls and Wilson, 1985) This . might provide a link between the evaluation and the experience of emotion (LeDoux, 1989) Taken together, these . findings suggest that there are neurons which are specialized in producing subjective emotional states and that cortical involvement may not be necessary for the experience of emotion. Lazarus (1990) has asserted that the conflict between the cognitive and non-cognitive viewpoints of emotion originates in a simplistic understanding of cognition. He disputes the commonly held notion that cognitions are solely the function of conscious processes. He argues, instead, that cognitions include even basic mental operations that are calculated without the awareness of the individual (Lazarus, 1990; Buck, 1993) In defense of his position, there exists substantial evidence of a dissociation between implicit and explicit learning on semantic memory tasks (Schacter, 1987; Squire, 1987). A dissociation of implicit and explicit memory indicates that one can develop a cognitive representation of a stimulus and not be aware of it.