Advances in Computer Vision and Pattern Recognition Forfurthervolumes: www.springer.com/series/4205 Antonio Robles-Kelly (cid:2) Cong Phuoc Huynh Imaging Spectroscopy for Scene Analysis AntonioRobles-Kelly CongPhuocHuynh NationalICTAustralia NationalICTAustralia Canberra,ACT Canberra,ACT Australia Australia SeriesEditors Prof.SameerSingh Dr.SingBingKang ResearchSchoolofInformatics MicrosoftResearch LoughboroughUniversity MicrosoftCorporation Loughborough Redmond,WA UK USA ISSN2191-6586 ISSN2191-6594(electronic) AdvancesinComputerVisionandPatternRecognition ISBN978-1-4471-4651-3 ISBN978-1-4471-4652-0(eBook) DOI10.1007/978-1-4471-4652-0 SpringerLondonHeidelbergNewYorkDordrecht LibraryofCongressControlNumber:2012951401 ©Springer-VerlagLondon2013 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer. PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations areliabletoprosecutionundertherespectiveCopyrightLaw. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub- lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect tothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) To mywife,forherloveand helpover the years. To mychildren,for thejoythey brought intomylife.Andtomyparents and sisters, fortheircontinuousand unconditionalsupport. AntonioRobles-Kelly To myparents and brother, fortheirendless loveand continuingsupport. CongPhuocHuynh Foreword Current Computer Vision systems, developed to solve many and varied problems insceneunderstandingandobjectrecognition,typicallyusethethreecolourchan- nelsdesignedtomimicthethree-colourvisionsystemofhumans.However,human vision is complex and involves significant perceptual learning, knowledge acqui- sition and, in general, post retinal neural processing to arrive at the type of inter- pretations of objects, scenes and events that predominate our perception. Indeed, oneofthemoststrikingaspectsofhumanvisualperceptionistheabilitytoappar- ently“directly”perceivetheworldaroundasmaterials,objectsandphysicalevents “out there” and not, purely, as images. The challenge for Computer Vision is to create elegant and efficient ways to achieve these goals and, as this book shows, suchachievementsarepossiblewiththeuseofnewcamerasystemsthatgobeyond the fixed three colour channels. Indeed the evolution of Imaging Spectroscopy (in remotesensingcalledHyperspectralImaging),capableofimagingspectralinforma- tionbeyondthevisibleregionandatspectralresolutionsfar greaterthanthefixed threecolourchannels,offersgreatpotentialandnewwaysofextractingthephysical propertiesofmaterials,objects,scenesandilluminants,muchmorepowerfullythan thestandardthreecoloursystemscanpossiblydo. This book is a landmark and timely contribution in this direction as it offers, forthefirsttime,detaileddescriptionsandanalysisofthesenewcamerasandhow they can be used to extract physical properties of what is being sensed in elegant and efficient ways relevant to material identification, object recognition and scene understanding. ImagingSpectroscopyalsoenablesthecreationofvisionsystems“ondemand” thatbestsuitspecificenvironmentsandapplicationswellbeyondthesystemsavail- ableincurrentfixedthree-coloursystemcameras.Itisalsotheidealcamerasystem foradvancingtheareaofComputationalPhotographyandmovingComputerVision awayfromtheimage,perse,towhatisbeingimaged. InthismonographAntonioandConghavebroughttogethertheirworkinImag- ingSpectroscopyoverthepastdecadetoresultinabookthatwillbecomeastandard forthearea.Welldone. VictoriaResearchLaboratory TerryCaelli NationalICTAustralia(NICTA) vii Preface The vast majority of ground-based image capture (e.g. for general photography, surveillance and industrial machine vision) is currently performed using RGB fil- ters on imaging sensors. This is particularly true for digital photography which, sinceitsinceptioninthe1980s,hasalmostexclusivelyemployedtrichromaticcam- erasbasedonCMOSorCCDsensorsusingBayerarrays,three-CCDcamerasora set of stacked photodiodes with a layer for each colour channel. Despite their dif- ferences,allofthesesensortypesinvolvecapturinglightatthewavelengthsofthe threeadditiveprimarycolours(red,greenandblue).Thesebandsareusedbecause they are a close approximation to the bands to which the human eye is sensitive. Therefore,theyarewellsuitedtocapturingphotosandvideosforimmediatehuman perception. Nowadays,theimagedataisnotonlyusedforimmediatehumanperceptionas originally intended, but also for a wide range of processing, contributing to cam- erautility,qualityandperformance.Overthepastfiveyears,manydigitalcameras have featured integrated circuits and firmware for sophisticated image processing (suchasCanon’sDIGICandNikon’sExpeedchips).Thesechipsinitiallyperformed functionssuchasautomaticfocus,exposureandwhitebalance.Morerecently,such chips have performed a wider range of higher-level scene analysis features such asfacedetection,smiledetectionandobjecttracking.Suchsceneanalysisfeatures provide a high value to professional and consumer camera users and are typically significant selling points in particular models. Many industrial cameras now also includehigh-levelsceneanalysisfunctionssuchaspeoplecountingforsurveillance andobjectrecognitionforindustrialmachinevision. In practise, every scene comprises a rich tapestry of light sources, material re- flectance, lighting and other photometric effects due to object curvature and shad- ows.Despitebeingreasonablyeffectiveforsceneanalysis,trichromatic(i.e.RGB) technologydoeshavelimitsinitssceneanalysiscapabilities.Forexample,acamera with an RGB sensor cannot determine the constituent material of an object in the scene.Similarly,cameraswithRGBsensorscannot,ingeneral,deliverphotometric invariantscharacteristictoamaterialandindependentofthelightingcondition,for robusttracking,identificationandrecognitiontasks. ix x Preface Inthisbook,weexploretheopportunities,applicationareasandchallengescon- cerningtheuseofimagingspectroscopyasameansforsceneunderstanding.This isimportant,sincesceneanalysisinthescopeofimagingspectroscopyinvolvesthe ability to robustly encode material properties, object composition and concentra- tions of primordial components. The combination of spatial and spectral informa- tionpromisesavastnumberofapplicationpossibilities.Forinstance,spectroscopic scene analysis can enable advanced capabilities for surveillance by permitting ob- jectsto betracked basedontheir composition.In computationalphotography,im- age colours may be enhanced taking into account each specific material type in the scene. For food security, health and precision agriculture the analysis of spec- troscopicimagescanbethebasisforthedevelopmentofnon-intrusivediagnostic, monitoringandsurveyingtools. The ability to combine spatial and compositional information of the scene re- quiressolvingseveraldifficultproblems.Withtheseproblemssolved,spectroscopic sceneanalysisoffersthepossibilityofperformingshapeanalysisfromasingleview for non-diffuse surfaces (Huynh and Robles-Kelly 2009), recovering photometric invariantsandmaterial-specificsignatures(FuandRobles-Kelly2011a),recovering the illuminant power spectrum (Huynh and Robles-Kelly 2010a) and visualising digitalmedia(Kimetal.2010). With the availability of imaging spectroscopy in ground-based cameras, it will nolongerbenecessarytolimitthecameradatacapturedtothreeRGBcolourchan- nels. Hyperspectral cameras offer an alternative number and range of bands that provide the best trade-off between functionality, performance and cost for a par- ticular market segment or application need. Rather than having the same spectra capturedasdisplayed,itwillbepracticaltodecouplethem,capturingarichspectral representation, performing processing on this representation and then rendering it intrichromaticformwhenneeded. Theuseoftheinformation-richrepresentationofthescenethatspectralimaging providesis,byitself,anewapproachtosceneanalysiswhichmakesuseofthespec- tralsignaturesandtheircontextsoastoprovideabetterunderstandingofmaterials andobjectsthroughouttheimage.Theuseofpolarisationandreflectiontorecover objectprofilesakintothoseinphaseimagingcandelivernovelmethodscapableof recovering an optimal representation of the scene which captures shape, material, objectprofilesandphotometricparameterssuchasindexofrefraction. Furthermore,thehighdimensionalityinherentinspectroscopydataimpliesthat these algorithms may not be exclusive to imaging spectroscopy, but could also be appliedtotheprocessingofotherhigh-dimensionaldata.Thus,thesemethodsmay beextendibletomanyothersensingtechnologiesbeyondjustspectralimagery. References Fu,Z.,&Robles-Kelly,A.(2011).Discriminantabsorptionfeaturelearningformaterialclassifi- cation.IEEETransactionsonGeoscienceandRemoteSensing,49(5),1536–1556. Preface xi Huynh,C.P.,&Robles-Kelly,A.(2010).Asolutionofthedichromaticmodelformultispectral photometricinvariance.InternationalJournalofComputerVision,90(1),1–27. Huynh,C.P.,&Robles-Kelly,A.(2009).Simultaneousphotometricinvarianceandshaperecovery. InIEEEinternationalconferenceoncomputervision. Kim,S.J.,Zhuo,S.,Deng,F.,Fu,C.W.,&Brown,M.(2010).Interactivevisualizationofhy- perspectralimagesofhistoricaldocuments.IEEETransactionsonVisualizationandGraphics, 16(6),1441–1448. Acknowledgements We wish to thank Professor Terry Caelli, Professor Edwin Hancock and Mr. Bill Simpson-Young for their support, impartial assessment and constructive criticism onthematerialinthisbook.ProfessorTerryCaelliwasinstrumentalinourefforts towardswritingthisdocument.Withouthiscontinuingsupport,someoftheresearch presented here would have never been accomplished. The insights that Professor Edwin Hancock provided into reflectance modelling and polarisation have greatly influencedthisbook.Mr.BillSimpson-Youngprovidedavaluableinsightintothe commercialuseandapplicationsofimagingspectroscopy. Wewouldliketoexpressoursincereappreciationtoourcollaborators.Inpartic- ular,wethankDr.ZhouyuFu,Dr.RanWei,Mr.LinGuandMs.SejutiRahmanfor theirpatienceanddedicationtotheteamworkthatdeliveredsomeoftheresultspre- sentedhere.Wealsoextendourappreciationtoallourcolleaguesfromthecomputer visiongroupatNICTA. Finally,wewouldliketoacknowledgeNICTA,1andinparticularitsCEO,Hugh Durrant-Whyte, for fostering this technology. We also thank the ANU and the UNSW@ADFAforallowingustosupervisetheirgraduatestudents. 1NICTAisfundedbytheAustralianGovernmentasrepresentedbytheDepartmentofBroadband, CommunicationsandtheDigitalEconomyandtheAustralianResearchCouncilthroughtheICT CentreofExcellenceprogram. xiii