Markus Kächele Machine Learning Systems for Multimodal Affect Recognition Machine Learning Systems for Multimodal Affect Recognition Markus Kächele Machine Learning Systems for Multimodal Affect Recognition Markus Kächele Walzenhausen, Switzerland Dissertation at the Faculty of Engineering, Computer Sciences and Psychology, Ulm University, Germany, 2019 ISBN 978-3-658-28673-6 ISBN 978-3-658-28674-3 (eBook) https://doi.org/10.1007/978-3-658-28674-3 Springer Vieweg © Springer Fachmedien Wiesbaden GmbH, part of Springer Nature 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer Vieweg imprint is published by the registered company Springer Fachmedien Wiesbaden GmbH part of Springer Nature. The registered company address is: Abraham-Lincoln-Str. 46, 65189 Wiesbaden, Germany ForGerhard Acknowledgments Writing these final lines means that the work is almost finished. A work in whichtheeffortofthelastroughlyfiveyearsculminates. Theessenceofmy research formulated and written down. This work would have never been possiblewithoutthehelp,adviceandsupportofagreatdealofpeople. IwanttostartwithmysupervisorsPDDr.FriedhelmSchwenkerandProf.Dr. Gu¨ntherPalmfortheirinvaluablesupportduringthecourseofthePhD(but also already during my Diploma studies). Their knowledge, experience and keeneyesformistakesinequationshelpedmeagreatdealandwithoutthem, thisworkwouldneverexist. Atthispoint,mydeepgratitudegoestowardstheTransregionalCollaborative ResearchCentreSFB/TRR62Companion-TechnologyforCognitiveTechnicalSys- temsandtheLandesgraduiertenfo¨rderungBaden-Wu¨rttembergwhichfunded theresearchfoundinthisthesis,allowedmetopresentmyworkoninterna- tionalconferencesandsupportedmewithascholarship. Next,Iwouldliketothankmycolleaguesforthepleasanttimeintheinstitute. Besidesvaluabletechnicaldiscussions, theywerealsoavailableforbreaksto regain a clear head. Especially I would like to point out Thomas Bottesch, Hans-Georg Glo¨ckler, Viktor Kessler and Martin Schels for achievements in theaforementionedtasks. Finally,Iwouldliketothankmyfamilyandfriends.Icanhardlyimaginethis workbeingpossiblewithouttheirsupport. Theycontributedtothisworkby keepingmylifeoutsideofuniversityintactbutalsoprovidednecessarygentle pushs from time to time. They were also the ones who helped me through toughertimesduringtheyearsandmydeepgratitudebelongstothemfornot leavingmyside. Contents Acknowledgments vii Contents ix ListofFigures xiii ListofTables xvii Abstract xix 1 Introduction 1 1.1 Relatedwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 ThesisOutline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Classificationandregressionapproaches 7 2.1 MultilayerPerceptrons . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1 Filterlearning . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2 RandomForest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3 SupportVectorMachines . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.1 LatticesofSMONodes . . . . . . . . . . . . . . . . . . . . 17 2.4 Supportvectordomaindescription . . . . . . . . . . . . . . . . . 20 2.5 EchoStateNetworks . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.5.1 TheEchoStateProperty . . . . . . . . . . . . . . . . . . . 24 2.5.2 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.6 Fusionprinciples . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.7 Lossfunctionsandperformancemeasures . . . . . . . . . . . . . 29 3 ApplicationsandAffectivecorpora 31 3.1 Affectivecategories . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3 Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.3.1 AVEC2013/2014 . . . . . . . . . . . . . . . . . . . . . . . 35 3.3.2 RECOLA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.3.3 EmotiW . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.3.4 BioVidHeatpaindatabase . . . . . . . . . . . . . . . . . . 37 x Contents 3.3.5 Cohn-Kanadedatabaseoffacialexpressions . . . . . . . 38 3.4 Afewwordsoncorpusdesignandannotation . . . . . . . . . . 39 3.4.1 AnnotationandTools. . . . . . . . . . . . . . . . . . . . . 39 3.4.2 Theannotator . . . . . . . . . . . . . . . . . . . . . . . . . 44 4 ModalitiesandFeatureextraction 47 4.1 Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.1.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.2 Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.2.1 Pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.2.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.3 Bio-Physiology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.3.1 Electromyography . . . . . . . . . . . . . . . . . . . . . . 55 4.3.2 Electrocardiogram . . . . . . . . . . . . . . . . . . . . . . 58 4.3.3 Skinconductancelevel . . . . . . . . . . . . . . . . . . . . 59 4.4 Metainformation . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.4.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5 Machinelearningfortheestimationofaffectivedimensions 63 5.1 Discreteestimation . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.1.1 Classimbalancing. . . . . . . . . . . . . . . . . . . . . . . 63 5.1.2 Importanceestimationoflocalizedfeaturedescriptorscom- putedoverthefacialregion . . . . . . . . . . . . . . . . . 65 5.1.3 Ensemblemethods . . . . . . . . . . . . . . . . . . . . . . 66 5.1.4 Experimentalvalidation . . . . . . . . . . . . . . . . . . . 69 5.1.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.2 Continuousestimation . . . . . . . . . . . . . . . . . . . . . . . . 78 5.2.1 Pre-Processing . . . . . . . . . . . . . . . . . . . . . . . . . 78 5.2.2 Cascadedregressionarchitectures . . . . . . . . . . . . . 79 5.2.3 Annotationdelayandscaling . . . . . . . . . . . . . . . . 87 5.2.4 Post-Processing . . . . . . . . . . . . . . . . . . . . . . . . 87 5.2.5 Protolabelsandtheperformancemeasure . . . . . . . . . 88 5.2.6 Experimentalvalidation . . . . . . . . . . . . . . . . . . . 89 5.2.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6 Adaptationandpersonalizationofclassifiers 107 6.1 Personalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.1.1 Metainformationbasedmeasures . . . . . . . . . . . . . 108 6.1.2 Distancebasedmeasures . . . . . . . . . . . . . . . . . . . 108 6.1.3 Machinelearningbasedmeasures . . . . . . . . . . . . . 109 6.2 Confidencelearning. . . . . . . . . . . . . . . . . . . . . . . . . . 110 6.2.1 Trainingthebaseregressor. . . . . . . . . . . . . . . . . . 110 6.2.2 Theconfidenceestimation . . . . . . . . . . . . . . . . . . 111 Contents xi 6.2.3 Sampleselection. . . . . . . . . . . . . . . . . . . . . . . . 112 6.2.4 Retrainingandprediction . . . . . . . . . . . . . . . . . . 113 7 Experimentalvalidationofpainintensityestimation 115 7.1 Estimationofdiscretepainlevels . . . . . . . . . . . . . . . . . . 115 7.2 Personalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 7.2.1 Multi-classexperiments . . . . . . . . . . . . . . . . . . . 117 7.2.2 Regressionexperiments . . . . . . . . . . . . . . . . . . . 120 7.3 Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 7.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 8 ExperimentalvalidationofMethodologicaladvancements 131 8.1 SMOLattices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 8.2 MajorityclassawareSVDD . . . . . . . . . . . . . . . . . . . . . 134 9 Discussion 137 9.1 Whatistheproblemwithcorpusdesign? . . . . . . . . . . . . . 137 9.2 Whatisthegroundtruth?. . . . . . . . . . . . . . . . . . . . . . . 138 9.3 Affectrecognitioninthewild . . . . . . . . . . . . . . . . . . . . 139 10 Conclusion 141 10.1 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 11 Summaryofmajorcontributions 145 Appendices 147 A Appendix 147 A.1 Personalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 A.1.1 Multi-class . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 A.1.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 A.1.3 Painthresholdvs.paintolerance . . . . . . . . . . . . . . 154 A.2 AVEC2014 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 A.3 Continuouspainintensityestimation . . . . . . . . . . . . . . . . 158 Referencesrelatedtotheauthor 159 Bibliography 165 List of Figures 1.1 ProcessingpipelineformachinelearningapplicationsinHCI. 3 2.1 Multilayerperceptronwithasinglehiddenlayer. . . . . . . . . 8 2.2 Parametricfilter. . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 ExamplesofdecisionboundariesgivenbytheRandomForest algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.4 Decisiontreewithtwonodes. . . . . . . . . . . . . . . . . . . . 15 2.5 Support Vector Machine with maximum margin in a linearly separablecase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6 TwopossibleSMOlatticearchitectures.. . . . . . . . . . . . . . 18 2.7 AnEchoStateNetwork(ESN)withreservoir,inputandoutput layers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.8 EarlyfusionwithanESNasclassification/regressionmodule. 27 2.9 LatefusionscenarioinvolvingseveralEchoStateNetworksas baseclassifiers/regressors. . . . . . . . . . . . . . . . . . . . . . 28 2.10 MidlevelfusionusingEchoStateNetworks. . . . . . . . . . . . 28 3.1 Valence-arousal-dominancespace. . . . . . . . . . . . . . . . . 32 3.2 IllustrationoftheGenevawheelofemotions. . . . . . . . . . . 33 3.3 RecordingsituationasfoundintheRECOLAcorpus. . . . . . 36 3.4 Painstimulationduringexperiment. . . . . . . . . . . . . . . . 38 3.5 ExampleofarecordingfromtheCohn-Kanadedataset. . . . . 39 3.6 ValenceannotationoftheRECOLAcorpus. . . . . . . . . . . . 41 3.7 Theannotationtoolgtrace. . . . . . . . . . . . . . . . . . . . . . 42 3.8 Theweb-basedannotationtoolAnnemo.. . . . . . . . . . . . . 43 3.9 TheMATLABbasedannotationtoolCarma. . . . . . . . . . . . 43 3.10 TheATLAStoolallowsvisualizationandsemi-supervisedan- notation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.1 Audiosampleofspokenspeech.. . . . . . . . . . . . . . . . . . 48 4.2 Exampleofaglottalflowanditstemporalderivative. . . . . . 50 4.3 Resultsoffacedetectionandlandmarklocalization. . . . . . . 52 4.4 Processofaligningtheface. . . . . . . . . . . . . . . . . . . . . 52