LIE, LIAR, LANCE ARMSTRONG   Lie, Liar, Lance Armstrong A case study of automatic detection of deception using non-verbal facial and vocal social signals Steven van Leer Anr 584022 Master's Thesis Communication and Information Sciences Specialization: Business Communication & Digital Media Faculty of Humanities Tilburg University, Tilburg Supervisor: Prof. Dr. E.O. Postma Second supervisor: Dr. M. Postma August 2013 LIE, LIAR, LANCE ARMSTRONG 1     Preface Na mijn hbo-opleiding Technische Bedrijfskunde die ik in 2011 aan Avans Hogeschool te ’s- Hertogenbosch heb afgerond, wist ik dat hier mijn toekomst niet lag. Een tijdje heb ik nog getwijfeld; een ritme, geld, een baan en wellicht niet gelukkig met mijn dagelijkse werkzaamheden? Of zal ik toch verder studeren? Mijzelf verder ontwikkelen, nog even genieten van het studentenleven? Uiteindelijk heb ik gekozen om door te gaan studeren. Het was een uitdaging en een nieuwe stap. In het begin even wennen, maar ik heb er geen seconde spijt van gehad. Het was een mooie tijd. Ik wil alle mensen bedanken die mij hebben geholpen in deze tijd, maar ook de mensen die ik heb leren kennen. Familie, vrienden, mijn vriendin en ook zeker mijn studiegenoten. Zij hebben deze tijd gemaakt tot iets onvergetelijks. Dank jullie wel. Daarnaast wil ik uiteraard, en in het bijzonder mijn supervisors Prof. Dr. E.O. Postma en Dr. M. Postma bedanken die mij hebben begeleid tijdens het doen van dit onderzoek en het schrijven van deze scriptie. Dank jullie wel. Al met al heb ik naar mijn idee een goed onderzoek neergezet, een interessant verhaal verteld, en hopelijk de wetenschap een stapje verder geholpen. Ik wens u veel plezier met lezen, en dank u bij voorbaat dat u hier de tijd voor neemt. Steven, Tilburg, augustus 2013 LIE, LIAR, LANCE ARMSTRONG 2     Index PREFACE 1 ABSTRACT 3 INTRODUCTION 4 LITERATURE REVIEW 6 FACIAL EXPRESSIVE CUES TO DECEPTION 7 AUDITORY FEATURES CUES TO DECEPTION 8 CASE STUDY 8 METHOD 11 DATA COLLECTION 11 ANALYSIS 1: FACIAL FEATURES ANALYSIS 12 ANALYSIS 2: SPEECH ANALYSIS 12 ANALYSIS 3: TRAINING THE CLASSIFIER 12 RESULTS 14 FACIAL EXPRESSIONS 14 SPEECH CHARACTERISTICS 19 ACCURACY RATES OF THE TRAINED CLASSIFIER 19 EXPECTATION TESTING 20 Chin raising 20 Smiling 20 Lips presser 20 Pitch 20 Pitch variance 20 DISCUSSION 21 FACIAL EXPRESSIONS 21 AUDITORY FEATURES 21 IMPLICATIONS OF THE MACHINE LEARNING 22 RESTRICTIONS 22 CONCLUSION 24 FIGURES 25 TABLES 34 LITERATURE 41 LIE, LIAR, LANCE ARMSTRONG 3     Abstract Detecting human deception is difficult. Scientific studies showed humans to perform badly on the task of detecting lies in other humans. Distinguishing truthful from deceptive behavior is hampered by the fact that cues to deception differ from person to person. Social Signal Processing (SSP), the automatic analysis of nonverbal social signals with computers, may facilitate the discovery of deception cues. SSP software can process facial and vocal information and search for informative cues to deception, provided that sufficient samples of truthful and deceptive behavior are available. The recent case of Lance Armstrong, who admitted to have lied about his use of doping, provides a (relatively) unique sample of videotaped interviews of deceptive and truthful behaviors. The research question addressed in this thesis reads: to what extent is it possible to automatically detect deception using facial and vocal cues? To answer the research question, SSP methods were employed to analyze videotaped fragments of interviews with Lance Armstrong. The fragments were divided in two classes: truthful and deceptive. The fragments in the truthful class consisted of statements about his experiences with his illness. We assumed that these statements are truthful. Those in the deceptive class consisted of responses to questions about the use of doping. These responses are now established to be all deceptive. The SSP analysis focused on the automatic measurement of facial expressions (relying on so-called Facial Action Units) and vocal characteristics (measures of vocal pitch). The SSP measurements were statistically analyzed to determine which facial expressions and vocal characteristics were diagnostic for truthful or deceptive statements. For facial expressions, the intensity of a number of Facial Action Units was found to vary with the nature of the statements. Most notably, deceptive statements were accompanied by more frequent smiling and sad expressions, whereas truthful statements were often characterized by expressions of anger. The statistical analysis of the vocal measurements revealed an effect of pitch. Deceptive statements tended to have a higher pitch than truthful ones. Finally, the measurements were used to train a classifier on the task of distinguishing between truthful and deceptive fragments. The optimal combination of measurements made use of only one measurement; a specific Facial Action Unit, called Brow Lower (Facial Action Unit 4). With this measurement, on average 83,1% of the fragments could be classified correctly by the classifier when trained on a balanced set of truthful and deceptive measurements (chance level 50%). The conclusion reads that for the particular case under consideration, deception can be detected with an accuracy of 83,1%. Future work should determine on the extent to which this result generalizes to other cases. Keywords: Deception, Non-verbal behavior, Facial expressions, Auditory features, Automatic deception detection, Lance Armstrong LIE, LIAR, LANCE ARMSTRONG 4     Introduction Detecting deception is considered a tough task for human perceivers. According to several studies, plain human observers only achieve slightly above chance level (Akehurst et al. 1996; Ekman and O’Sullivan, 1991; Ekman, O’Sullivan & Frank 1999; Malone & DePaulo, 2001). Nevertheless, paying attention to the proper non-verbal cues, might result in a more accurate classifying of truthful and deceptive utterances. Facial and vocal expressions have been shown to provide such cues (Ekman, 1989; DePaulo, Stone & Lassiter, 1985; Zuckerman & Driver, 1985). Still, deception detection is a difficult task since expressed cues during deception differ from person to person. Therefore, it is useful to study the detection of deception in single persons. To date, only two studies researched the detection of deceptive behavior of one particular person. Vrij and Mann (2001) recorded video fragments in which a convict was examined on his allegations of murder. In this case there already was sufficient evidence to prove the suspects’ guilt, thus the truth was already known. This made the video material suitable to compare the deceptive and truthful statements. Results of the study showed detection rates of 57% on deceptive utterances and 70% on truthful utterances. Davis and Hadiks (1995) analyzed an interview with Saddam Hussein, focusing on his non-verbal behavior during truthful and deceptive utterances. They found that a certain pattern exists in ones non-verbal behavior when it is compared to certain subjects that are discussed. Though these studies have put great effort into detecting the deceit of one particular person, they are limited by several factors. Vrij and Mann’s research is limited because they only used human observers for deception detection while most humans only achieve chance level in detecting deception (Ekman, O’Sullivan & Frank 1999; Malone & DePaulo, 2001). The case study on Saddam Hussein was limited because it used manual coding of the non-verbal behavior. Although the coders may have been trained exceptionally well, there is a reasonable chance that they made mistakes or that they did not code systematically the same way. Next to this, both studies did not use software to predict deceit based on the results. Concluding, these previous studies have their restrictions, and should therefore be extended to get more knowledge on predicting deception based on uttered cues. The study reported in this thesis improves upon earlier work in two respects. First, cues are automatically coded by means of software, which should increase the objectivity. Second, the coded cues are automatically classified by means of machine learning. This should result in better insights in deception detection. Field research on non-verbal behavior during deception is hard to conduct (Mann, Vrij & Bull, 2002). As a result, past research almost exclusively conducted laboratory studies. Laboratory studies have the disadvantages of the low-stakes that are involved, resulting in little involvement of the participants. Furthermore, the lack of naturalness raises a major issue, as research showed a significant difference between a natural setting and a laboratory setting (Porter & Brinke, 2010; Vrij & Semin, 1996). The present study has a higher ecological validity compared to previous laboratory research, because it relies on video recordings of Lance Armstrong in more or less natural settings. LIE, LIAR, LANCE ARMSTRONG 5     The Lance Armstrong case undoubtedly involves high stakes as well as naturalness. Preliminary to his confessions of having used doping, in his interviews he denied ever having used performance-enhancing drugs. The competitive nature of professional cycling and the high stakes that are involved makes the Armstrong case highly suitable for the investigation of high-stake deception in a natural setting. LIE, LIAR, LANCE ARMSTRONG 6     Literature review As several studies have concluded, deception detection is difficult. Ekman and O’Sullivan (1991), for example, performed a study on deception detection, for which they compared the detection rate results from freshman students to those of police officers, lawyers and secret service agents. This study did not show significant differences in the accuracy of recognizing deception between these groups. Only Secret Service agents performed significantly better. Other participants only achieved chance level. Other studies also did not find significant differences between professional and non-professionals in detecting deceit. In all studies the performances on deception detecting were slightly above chance level (Akehurst et al. 1996; Ekman, O’Sullivan & Frank 1999; Kraut, 1980; Malone & DePaulo, 2001). During attempted deception detection, verbal and non-verbal cues are important for perceivers to identify deceit (Zuckerman, DePaulo & Rosenthal, 1981). However, it is important to note that verbal communication is a less accurate predictor of deceptive behavior. One reason for this is that the tone of voice is capable of expressing information that is not in accordance with the verbal content (Bugental, Henker & Whalen, 1976; Love, 1975; Shennum, 1980; Wietz, 1972 as cited in Zuckerman, DePaulo & Rosenthal, 1981). Contrary to that, non-verbal behavior is a part of communication which lends itself for accurately detecting deception (DePaulo, et al., 1996; Ekman, 2001; Zuckerman, DePaulo & Rosenthal). This might be due to the greater cognitive load one experiences during the expression of a lie. While telling a lie, people have to control their verbal and non-verbal behavior to avoid getting caught, which leads to unnatural behavior that can be detected (Ekman; Zuckerman, DePaulo & Rosenthal). Therefore, non-verbal behavior is an accurate indicator of deception. A study shows that all types of non-verbal behavior can in some way be predictors of deception (Nicholas et al. 2013). However, facial expressions and auditory features are the best predictors of deception (DePaulo, Stone & Lassiter, 1985; Zuckerman & Driver, 1985). Moreover, Ekman (1989) stated that the combination of facial expressions and auditory features “allows for highly accurate identification of deceptive behavior” (p. 71). Therefore, this study focuses on facial expressions and auditory features during deceptive behavior. Although detecting deception is difficult, studies show that uttered cues during deception leak some kind of information in their non-verbal behavior. This is called leakage (Ekman et al., 1991; Warren, Schertler & Bull, 2009). Ekman et al. show that this leakage expresses itself in facial expressions, body movements and/or vocal tones. This leakage might lead to the great reliability of detecting deception based on non-verbal behavior. Leakage in facial expressions can be distinguished into two categories; subtle expressions and micro-expressions. The subtle expressions are characterized as “fragments of otherwise suppressed or masked affect displays using only part of the normally associated musculature” (Warren, Schertler & Bull, 2009, p. 61). Micro-expressions are LIE, LIAR, LANCE ARMSTRONG 7     facial expressions, which typically last for short time intervals, between 1/ and 1/ of a second. This 5 25 makes them difficult to discover with the bare eye (Ekman & Friesen 1969; Frank & Ekman, 1997; Porter & Brinke, 2008). These two specific types of leakage may possibly lead to the detection of deceit by means of a research on facial expressions. DePaulo et al. (1982) proposed a fixed set of cues consisting of ten visual and nine auditory features, with which it should be capable of accurately recognizing deceptive behavior. Later research by DePaulo et al. (1997), Ennis, Vrij and Chance (2008) and Vrij and Mann (2004), though, showed that there are individual differences in cues during deceptive situations, and thus the fixed set of cues by DePaulo et al., is rejected. These differences form an additional barrier in detecting deception. Although there are differences between individuals, some cues occur more frequently during deception. Facial expressive cues to deception In order to universally and objectively code facial expressions, researchers developed the FACS (Facial Action Coding System) (Ekman & Friesen 1976; Ekman & Friesen, 1978; Ekman, Friesen & Hager, 2002). This system uses action units (AU’s). AU’s are facial muscles that consist of single facial muscles or groups of facial muscles that play a role in facial expressions (Ekman, Friesen & Hager). Numerous researches have focused on finding reliable facial cues during deception. O'Hair, Cody and McLaughlin (1981) found that there is less smiling during a prepared lie compared to during a spontaneous lie or a truthful utterance. Bond Kahler and Paolicelli (1985) and also DePaulo (1994) later confirmed this result. More importantly, DePaulo et al. (2003) conducted a meta-analysis on cues to deception (which is the most recent to date) in which they questioned whether or not there are general reliable indicators of deception. In their study, they reviewed 120 studies, and paid attention to verbal and non-verbal behavior in relation to deception. With regard to facial expressions, they came up with three measurable reliable indicators of deception. These are; pressed lips (AU 23, AU 24), chin raise (AU 17) and genuine smile (AU 6). The results of this study are found in the appendix, table A1. To automatically code facial expressions, a variety of software has been developed. Computer Expression Recognition Toolbox (CERT) is capable of detecting expressions based on the FACS and the six basic emotions (Happiness, Sadness, Anger, Fear, Surprise and Disgust) as proposed by Ekman (1992). CERT runs real time on a frame of 320 x 240 on 10 fps (Littlewort et al. 2011). The advantage of using a system like CERT is that it automatically codes the expressions. Next to that, it is an objective and accurate tool. This software makes use of different classifiers based on Gabor filters and Fourier transforms. As a result, CERT is capable of detecting faces and expressions based on AU’s (Shen & Bai, 2006). LIE, LIAR, LANCE ARMSTRONG 8     Auditory features cues to deception As stated earlier, besides facial expressions auditory cues are also accurate predictors of deception (DePaulo, Stone & Lassiter, 1985; Zuckerman & Driver, 1985). This conclusion is based on the difficulty of controlling vocal cues (Scherer, 1986). The non-verbal part of speech thus also leaks information leading to deception (Ekman et al. 1991; Warren, Schertler & Bull, 2009). Therefore, analyzing one’s speech next to analyzing the facial expressions might result in an improved deception detection mechanism. Similar to facial expressions, no fixed set of cues is available to detect deception in one’s speech, as results are not consistent across studies, and individual differences are present (Greene et. al. 1985; Matarazzo et. al. 1970; Motley, 1974; O'Hair, Cody, & McLaughlin, 1981). This might partly be caused by the difference in pitch between men and women, since a men’s voice generally has an overall lower pitch (Latinus & Belin, 2011). Nonetheless, just as in the case of facial expressions, some audible cues are recognized that roughly implicate deception as well. Many researchers have therefore focused on the non-verbal auditory part of deception. Rockwell, Buller and Burgoon (1997) found that deceivers tend to speak slower and have a higher intensity and a greater pitch variance in their voice. In addition to that, a higher pitch was noticed in a number of studies (Apple, Streeter & Kraus, 1979; Ekman, Friesen & Scherer, 1976; Streeter et al. 1977; Vrij, 1991; Vrij, 1995). In general, longer message duration and more speech hesitations or pauses are found as accurate indicators of deceptive behavior (Ekman, 1989; Ekman, Friesen, & Scherer; Ekman, O'Sullivan, Friesen, & Scherer, 1991; Streeter et al.; Zuckerman et al. 1981). Despite individual differences, studies showed significant differences between auditory cues during truthful and deceptive utterances, which make it possible to detect deception by means of an auditory analysis. The differences in auditory features during truthful and deceptive utterances may be the result of the arousing experience of lying (Barland & Raskin, 1975; Streeter et al., 1977). Again, the greater cognitive load plays a role in the accidental leakage (Ekman, 2001; Zuckerman, DePaulo & Rosenthal, 1981). Case study The explorative study that is presented in this thesis focuses on accurately recognizing deceptive behavior of one person in particular. In contrast to a great number of earlier studies (Ekman & Friesen, 1974; Harrison, Hwalek, Raney, & Fritz, 1978; Kraut, 1978), in this study features of deceptive behavior are identified by trained computer software. This study is conducted respecting these previous studies, as it tries to accurately distinguish deceptive from truthful utterances of one particular person. Furthermore, this study attempts to automatically predict deception in non-analyzed data by means of a trained software program. Previous research on deception detection lacks credibility on two major issues. First, the involved stakes are important to consider. Lies in which high stake lies are involved, are easier LIE, LIAR, LANCE ARMSTRONG 9     detectable due to emotional cues (Frank & Ekman, 1997). Further, DePaulo et al. (2003) show that cues of deception are easier to detect when the deceptive behavior is based on crimes, or when it is based on personal reasons rather then when it is set up for an incentive. Also, two other studies show that personally motivated liars are more likely betrayed by their non-verbal behavior than liars who are not personally motivated to succeed (Burgoon & Floyd, 2000; DePaulo et al., 1988). These liars without a personal motivation are better capable of controlling their facial expressions (DePaulo, Stone, & Lassiter, as cited in DePaulo). Second, the naturalness of the settings is important. As a majority of the research has been conducted in a laboratory setting, this lacks the factor of naturalness. However, Mann, Vrij and Bull (2002) found that it has been hard to conduct field research on the non- verbal behavior during deception, this case involves a more realistic setting compared to previous laboratory research, as in the Lance Armstrong case, both the high-stakes and a natural setting are involved. The Lance Armstrong case, as analyzed in this study, involves a video, in which he confesses to have used performance-enhancing substances during his cycling career. Preliminary to his confessions, he denied ever having used performance-enhancing drugs during all of his interviews. As numerous of these interviews in which he lied were videotaped, this creates unique opportunity to analyze the non-verbal deceptive behavior of the athlete while high stakes are involved. However, to detect deception, also fragments in which he tells the truth have to be analyzed in order to make a decent comparison between truthful and deceptive behavior. As, in the interviews, also truth-based stories are considered, the video material is suitable for comparison. In conclusion, it is possible to detect deceit by analyzing non-verbal behavior, in particular by analyzing facial and auditory cues. There are cues that, to some extent, might indicate deception. However, these cues are not a fixed set of cues due to individual differences. Nevertheless, the most important finding of this literature review is that deceptive behavior differs from truthful behavior. This literature review therefore leads to the following research question: RQ: To what extent is it possible to automatically detect deception using facial and vocal cues? Based on previous research, the following cues, as set in table 1, are expected to indicate deceptive behavior. Table 1 shows the expectations for this case study. Two types of non-verbal behavior are distinguished; the visual cues and the auditory cues. The second column shows the cues, which at large are expressed during deception. The third column shows the size of the effect. In the fourth column the sources are found. The fifth column considers the type of study of which the cues derive from. A distinction is made between meta studies, laboratory studies and a natural setting. The meta studies rely on different types of studies, the laboratory studies are experimental in nature, and consider low stakes. Opposed to the laboratory setting, one study considers a more natural setting in which high stakes are involved.