Handbook of Face Recognition Stan Z. Li Anil K. Jain Editors Handbook of Face Recognition With 210 Illustrations StanZ.Li AnilK.Jain CenterforBiometricsResearchandTesting& DepartmentofComputerScience NationalLabofPatternRecognition &Engineering InstituteofAutomation MichiganStateUniversity ChineseAcademyofSciences EastLansing,MI48824-1226 Beijing100080 USA China [email protected] [email protected] LibraryofCongressCataloging-in-PublicationData Handbookoffacerecognition/editors,StanZ.Li&AnilK.Jain. p.cm. Includesbibliographicalreferencesandindex. ISBN0-387-40595-X(alk.paper) 1. Humanfacerecognition(Computerscience I. Li,S.Z.,1958– II. Jain, AnilK.,1948– TA1650.H36 2004 006.4′2—dc22 2004052453 ISBN0-387-40595-X Printedonacid-freepaper. ©2005SpringerScience+BusinessMedia,Inc. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, Inc., 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in con- nectionwithreviewsorscholarlyanalysis.Useinconnectionwithanyformofinfor- mation storage and retrieval, electronic adaptation, computer software, or by similar ordissimilarmethodologynowknownorhereafterdevelopedisforbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinionastowhetherornottheyaresubjecttoproprietaryrights. PrintedintheUnitedStatesofAmerica. (MP) 9 8 7 6 5 4 3 2 1 SPIN10946602 springeronline.com Preface Facerecognitionhasalargenumberofapplications,includingsecurity,personverification,In- ternetcommunication,andcomputerentertainment.Althoughresearchinautomaticfacerecog- nitionhasbeenconductedsincethe1960s,thisproblemisstilllargelyunsolved.Recentyears have seen significant progress in this area owing to advances in face modeling and analysis techniques. Systems have been developed for face detection and tracking, but reliable face recognitionstilloffersagreatchallengetocomputervisionandpatternrecognitionresearchers. There are several reasons for recent increased interest in face recognition, including ris- ing public concern for security, the need for identity verification in the digital world, and the needforfaceanalysisandmodelingtechniquesinmultimediadatamanagementandcomputer entertainment. Recent advances in automated face analysis, pattern recognition, and machine learninghavemadeitpossibletodevelopautomaticfacerecognitionsystemstoaddressthese applications. Thisbookwaswrittenbasedontwoprimarymotivations.Thefirstwastheneedforhighly reliable,accuratefacerecognitionalgorithmsandsystems.Thesecondwastherecentresearch in image and object representation and matching that is of interest to face recognition re- searchers. Thebookisintendedforpractitionersandstudentswhoplantoworkinfacerecognitionor whowanttobecomefamiliarwiththestate-of-the-artinfacerecognition.Italsoprovidesref- erencesforscientistsandengineersworkinginimageprocessing,computervision,biometrics andsecurity,Internetcommunications,computergraphics,animation,andthecomputergame industry. The material fits the following categories: advanced tutorial, state-of-the-art survey, andguidetocurrenttechnology. The book consists of 16 chapters, covering all the subareas and major components nec- essary for designing operational face recognition systems. Each chapter focuses on a specific topicorsystemcomponent,introducesbackgroundinformation,reviewsup-to-datetechniques, presentsresults,andpointsoutchallengesandfuturedirections. Chapter1introducesfacerecognitionprocessing,includingmajorcomponentssuchasface detection,tracking,alignment,andfeatureextraction,anditpointsoutthetechnicalchallenges ofbuildingafacerecognitionsystem.Weemphasizetheimportanceofsubspaceanalysisand learning,notonlyprovidinganunderstandingofthechallengesthereinbutalsothemostsuc- VI Preface cessfulsolutionsavailablesofar.Infact,mosttechnicalchaptersrepresentsubspacelearning- basedtechniquesforvariousstepsinfacerecognition. Chapter 2 reviews face detection techniques and describes effective statistical learning methods. In particular, AdaBoost-based learning methods are described because they often achieve practical and robust solutions. Techniques for dealing with nonfrontal face detection arediscussed.Resultsarepresentedtocompareboostingalgorithmsandotherfactorsthataf- fectfacedetectionperformance. Chapters 3 and 4 discuss face modeling methods for face alignment. These chapters de- scribe methods for localizing facial components (e.g., eyes, nose, mouth) and facial outlines and for aligning facial shape and texture with the input image. Input face images may be ex- tractedfromstaticimagesorvideosequences,andparameterscanbeextractedfromtheseinput imagestodescribetheshapeandtextureofaface.Theseresultsarebasedlargelyonadvances intheuseofactiveshapemodelsandactiveappearancemodels. Chapters5and6covertopicsrelatedtoilluminationandcolor.Chapter5describesrecent advances in illumination modeling for faces. The illumination invariant facial feature repre- sentationisdescribed;thisrepresentationimprovestherecognitionperformanceundervarying illuminationandinspiresfurtherexplorationsofreliablefacerecognitionsolutions.Chapter6 deals with facial skin color modeling, which is helpful when color is used for face detection andtracking. Chapter7providesatutorialonsubspacemodelingandlearning-baseddimensionreduction methods,whicharefundamentaltomanycurrentfacerecognitiontechniques.Whereasthecol- lectionofallimagesconstituteshighdimensionalspace,imagesoffacesresideinasubspaceof thatspace.Facialimagesofanindividualareinasubspaceofthatsubspace.Itisofparamount importance to discover such subspaces so as to extract effective features and construct robust classifiers. Chapter 8 addresses problems of face tracking and recognition from a video sequence of images. The purpose is to make use of temporal constraints present in the sequence to make trackingandrecognitionmorereliable. Chapters 9 and 10 present methods for pose and illumination normalization and extract effective facial features under such changes. Chapter 9 describes a model for extracting illu- mination invariants, which were previously presented in Chapter 5. Chapter 9 also presents a subregionmethodfordealingwithvariationinpose.Chapter10describesarecentinnovation, calledMorphableModels,forgenerativemodelingandlearningoffaceimagesunderchanges inilluminationandposeinananalysis-by-synthesisframework.Thisapproachresultsinalgo- rithmsthat,inasense,generalizethealignmentalgorithmsdescribedinChapters3and4tothe situationwherethefacesaresubjecttolargechangesinilluminationandpose.Inthiswork,the three-dimensionaldataoffacesareusedduringthelearningphasetotrainthemodelinaddition tothenormalintensityortextureimages. Chapters 11 and 12 provide methods for facial expression analysis and synthesis. The analysispart,Chapter11,automaticallyanalyzesandrecognizesfacialmotionsandfacialfea- turechangesfromvisualinformation.Thesynthesispart,Chapter12,describestechniqueson three-dimensionalfacemodelingandanimation,facelightingfromasingleimage,andfacial expressionsynthesis.Thesetechniquescanpotentiallybeusedforfacerecognitionwithvary- ing poses, illuminations, and facial expressions. They can also be used for human computer interfaces. Preface VII Chapter13reviews27publiclyavailabledatabasesforfacerecognition,facedetection,and facial expression analysis. These databases provide a common ground for development and evaluation of algorithms for faces under variations in identity, face pose, illumination, facial expression,age,occlusion,andfacialhair. Chapter14introducesconceptsandmethodsforfaceverificationandidentificationperfor- manceevaluation.ThechapterfocusesonmeasuresandprotocolsusedinFERETandFRVT (facerecognitionvendortests).Analysisofthesetestsidentifiesadvancesofferedbystate-of- the-arttechnologiesforfacerecognition,aswellasthelimitationsofthesetechnologies. Chapter 15 offers psychological and neural perspectives suggesting how face recognition might go on in the human brain. Combined findings suggest an image-based representation thatencodesfacesrelativetoaglobalaverageandevaluatesdeviationsfromtheaverageasan indicationoftheuniquepropertiesofindividualfaces. Chapter 16 describes various face recognition applications, including face identification, security,multimediamanagement,andhuman-computerinteraction.Thechapteralsoreviews manyfacerecognitionsystemsanddiscussesrelatedissuesinapplicationsandbusiness. Acknowledgments Anumberofpeoplehelpedinmakingthisbookareality.VincentHsu,DirkColbry,Xiaoguang Lu,KarthikNandakumar,andAnoopNamboodiriofMichiganStateUniversity,andShiguang Shan,ZhenanSun,ChenghuaXuandJiangweiLioftheChineseAcademyofScienceshelped proofread several of the chapters. We also thank Wayne Wheeler and Ann Kostant, editors at Springer,fortheirsuggestionsandforkeepingusonschedulefortheproductionofthebook. ThishandbookprojectwasdonepartlywhenStanLiwaswithMicrosoftResearchAsia. December2004 StanZ.Li Beijing,China AnilK.Jain EastLansing,Michigan Contents Chapter1.Introduction StanZ.Li,AnilK.Jain ..................................................... 1 Chapter2.FaceDetection StanZ.Li................................................................ 13 Chapter3.ModelingFacialShapeandAppearance TimCootes,ChrisTaylor,HaizhuangKang,VladimirPetrovic´ ...................... 39 Chapter4.ParametricFaceModelingandTracking Jo¨rgenAhlberg,FadiDornaika .............................................. 65 Chapter5.IlluminationModelingforFaceRecognition RonenBasri,DavidJacobs.................................................. 89 Chapter6.FacialSkinColorModeling J.BirgittaMartinkauppi,MattiPietika¨inen ..................................... 113 ColorPlatesforChapters6and15 ........................................................................ 137 Chapter7.FaceRecognitioninSubspaces GregoryShakhnarovich,BabackMoghaddam ................................... 141 Chapter8.FaceTrackingandRecognitionfromVideo RamaChellappa,ShaohuaKevinZhou ........................................ 169 Chapter9.FaceRecognitionAcrossPoseandIllumination RalphGross,SimonBaker,IainMatthews,TakeoKanade ......................... 193 Chapter10.MorphableModelsofFaces SamiRomdhani,VolkerBlanz,CurzioBasso,ThomasVetter ...................... 217 X Contents Chapter11.FacialExpressionAnalysis Ying-LiTian,TakeoKanade,JeffreyF.Cohn .................................... 247 Chapter12.FaceSynthesis ZichengLiu,BainingGuo .................................................. 277 Chapter13.FaceDatabases RalphGross.............................................................. 301 Chapter14.EvaluationMethodsinFaceRecognition P.JonathonPhillips,PatrickGrother,RossMicheals ............................. 329 Chapter15.PsychologicalandNeuralPerspectivesonHumanFaceRecognition AliceJ.O’Toole........................................................... 349 Chapter16.FaceRecognitionApplications ThomasHuang,ZiyouXiong,ZhenqiuZhang ................................... 371 Index................................................................... 391