Data-Driven Facial Expression Analysis from Live Video by Wee Kiat, Tay A thesis submitted to the Victoria University of Wellington in partial fulfilment of the requirements for the degree of Master of Science In Computer Graphics. Victoria University of Wellington 2017 Abstract Emotion analytics is the study of human behavior by analyzing the responses when humans experience different emotions. In this thesis, we research into emotion analytics solutions using computer vision to detect emotions from facial expressions automatically using live video. Considering anxiety is an emotion that can lead to more serious conditions like anxiety disorders and depression, we propose 2 hypotheses to detect anxiety from facial expressions. One hypothesis is that the complex emotion “anxiety” is a subset of the basic emotion “fear”. The other hypothesis is that anxiety can be distinguished from fear by differences in head and eye motion. We test the first hypothesis by implementing a basic emotions detector based on facial action coding system (FACS) to detect fear from videos of anxious faces. When we discover that this is not as accurate as we would like, an alternative solution based on Gabor filters is implemented. A comparison is done between the solutions and the Gabor- based solution is found to be inferior. The second hypothesis is tested by using scatter graphs and statistical analysis of the head and eye motions of videos for fear and anxiety expressions. It is found that head pitch has signific ant differences between fear and anxiety. As a conclusion to the thesis, we implement a systems software using the basic emotions detector based on FACS and evaluate the software by comparing commercials using emotions detected from facial expressions of viewers. i Acknowledgements I would like to thank my supervisor Dr. Taehyun Rhee and my co- supervisors Dr. Harvey Ho and Prof. Neil Dodgson. It’s with their guidance, support and valuable advice that I can complete my master’s research and thesis. I would also like to thank my fellow postgraduate students from the computer graphics group for sharing their research and providing valuable feedback during our weekly group meetings. In addition, I would like to give special thanks to Auckland Bioengineering Institute for sponsoring the research grant to support my thesis. Finally, I would like to take this opportunity to express my gratitude to my brother and parents for their support and encouragement given to me throughout my life. Without them, it would have been possible for me to pursue my interests in computer graphics and undertake this master’s degree. iii Contents 1. Introduction ............................................................................................. 11 1.1 Motivation of thesis ...................................................................... 11 1.2 Objectives of thesis ........................................................................ 12 1.3 Research Methodology ................................................................. 13 1.3.1 Literature survey on solutions for detecting anxiety ..... 13 1.3.2 Propose hypothesis on facial expression of anxiety ....... 13 1.3.3 FACS-based solution......................................................... .14 1.3.4 Gabor-based solution ......................................................... 16 1.3.5 Detecting head and eye movement ................................. 18 1.3.6 System application for detecting emotions ..................... 19 1.4 Structure of thesis .......................................................................... 19 2. Background and Related Works ............................................................ 21 2.1 Basic Emotions ............................................................................... 21 2.2 Facial Action Coding System (FACS) .......................................... 22 2.3 Facial Expression of Anxiety ........................................................ 26 2.4 Detecting emotions from facial images ....................................... 28 2.4.1 FACS-based detection methods ....................................... 28 2.4.2 Gabor-based detection methods ...................................... 31 2.5 Beyond Basic Emotions ................................................................ 34 2.6 Affective Applications .................................................................. 36 2.6.1 Detecting emotional stress from facial expressions for driving safety ................................................................................. 36 2.6.2 Video Classification and Recommendation using Emotions ........................................................................................ .37 2.6.3 Predicting Movie Ratings from Audience Behavior .s.... 38 2.6.4 Intelligent Advertising Billboards ................................... 39 v vi CONTENTS 2.6.5 Predicting Online Media Effectiveness using Smile Response.......................................................................................... 40 3. Databases and Tools ................................................................................ 43 3.1 Facial Expression Databases ........................................................ .43 3.1.1 Japanese Female Facial Expression (JAFFE) ................... 43 3.1.2 Extended Cohn-Kanade Database (CK+) ........................ 44 3.1.3 Mind Reading DVD .......................................................... .46 3.1.4 Affectiva-MIT Facial Expression Dataset (AM-FED)..... 49 3.2 Tools Used ..................................................................................... 50 3.2.1 OpenFace ............................................................................ 51 3.2.2 Weka ................................................................................... 51 3.2.3 Matlab ................................................................................. 52 3.2.4 LibSVM ............................................................................... 53 3.2.5 WebRTC ............................................................................. 55 3.2.6 Kurento ............................................................................... 57 4. Detecting Emotions using Facial Action Units ..................................... 59 4.1 Existing solution and its limitations ............................................ 59 4.2 Proposed Solution and Implementation .................................... 60 4.3 Testing Methodology ................................................................... 63 4.3.1 How to select the test candidates .................................... 63 4.3.2 How to interpret the results ............................................. 66 4.4 Results ........................................................................................... .67 4.4.1 Classifier performance using CK+ Database .................. 67 4.4.2 Classifier performance using JAFFE Database ............... 70 4.4.3 Detecting fear from anxiety videos ................................. 71 4.5 Analysis and Discussion .............................................................. .71 5. Detecting Emotions using Gabor Filter ................................................. 75 5.1 Proposed Solution ......................................................................... 75 CONTENTS vii 5.2 Implementation ............................................................................... 77 5.3 Results ............................................................................................... 82 5.3.1 CK+ Database ...................................................................... 82 5.3.2 JAFFE Database ................................................................... 83 5.4 Analysis and discussion ................................................................. 85 6. Detecting Head and Eye Motion............................................................ 89 6.1 Method ........................................................................................... .89 6.2 Results ............................................................................................ .92 6.3 Analysis and discussion ............................................................... 97 7. Systems Implementation .......................................................................101 7.1 Introduction ................................................................................. 101 7.2 System Architecture .................................................................... 101 7.2.1 Overview .......................................................................... 101 7.2.2 Kurento Client ................................................................. 104 7.2.3 Kurento Media Server ...................................................... 106 7.2.4 Applications Server ......................................................... .107 7.3 Prototype ...................................................................................... 107 7.3.1 Specifications.................................................................... 107 7.3.2 Implementation ............................................................... 108 7.4 Evaluation of System .................................................................. 109 7.4.1 Procedure .......................................................................... 110 7.4.2 Evaluation methodology ................................................. 110 7.4.3 Test Results ....................................................................... 112 7.5 Analysis and discussion ............................................................. .115 8. Conclusion ..............................................................................................117 8.1 Summary and Findings .............................................................. .117 8.2 Limitations and Future Work ..................................................... 118