High Dynamic Range Video From Acquisition to Display and Applications High Dynamic Range Video From Acquisition to Display and Applications Edited by Frédéric Dufaux CNRS, Télécom ParisTech Patrick Le Callet University of Nantes Rafał K. Mantiuk University of Cambridge Marta Mrak British Broadcasting Corporation AMSTERDAM (cid:129) BOSTON (cid:129) HEIDELBERG (cid:129) LONDON NEW YORK (cid:129) OXFORD (cid:129) PARIS (cid:129) SAN DIEGO SAN FRANCISCO (cid:129) SINGAPORE (cid:129) SYDNEY (cid:129) TOKYO Academic Press is an imprint of Elsevier AcademicPressisanimprintofElsevier 125LondonWall,London,EC2Y5AS,UK TheBoulevard,LangfordLane,Kidlington,OxfordOX51GB,UK ©2016ElsevierLtd.Allrightsreserved. Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,electronicormechanical,including photocopying,recording,oranyinformationstorageandretrievalsystem,withoutpermissioninwritingfromthepublisher. Detailsonhowtoseekpermission,furtherinformationaboutthePublisher’spermissionspoliciesandourarrangements withorganizationssuchastheCopyrightClearanceCenterandtheCopyrightLicensingAgency,canbefoundatour website:www.elsevier.com/permissions. ThisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightbythePublisher(otherthanasmaybe notedherein). Notices Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchandexperiencebroadenourunderstanding, changesinresearchmethods,professionalpractices,ormedicaltreatmentmaybecomenecessary. Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgeinevaluatingandusinganyinformation, methods,compounds,orexperimentsdescribedherein.Inusingsuchinformationormethodstheyshouldbemindfuloftheir ownsafetyandthesafetyofothers,includingpartiesforwhomtheyhaveaprofessionalresponsibility. Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,oreditors,assumeanyliabilityforanyinjury and/ordamagetopersonsorpropertyasamatterofproductsliability,negligenceorotherwise,orfromanyuseoroperationof anymethods,products,instructions,orideascontainedinthematerialherein. LibraryofCongressCataloging-in-PublicationData AcatalogrecordforthisbookisavailablefromtheLibraryofCongress BritishLibraryCataloguing-in-PublicationData AcataloguerecordforthisbookisavailablefromtheBritishLibrary ISBN:978-0-08-100412-8 ForinformationonallAcademicPresspublications visitourwebsiteathttps://www.store.elsevier.com/ Contributors M.A.Abebe Technicolor,Cesson-Sévigné,France N.Adami UniversityofBrescia,Brescia,Italy D.Agrafiotis UniversityofBristol,Bristol,UnitedKingdom L.Albani BarcoFIMI,Saronno(VA),Italy M.A.Ali UniversityofToronto,Toronto,ON,Canada P.Andrivon Technicolor,Cesson-Sevigne,France A.Artusi UniversityofGirona,Girona,Spain A.Badano USFoodandDrugAdministration,SilverSpring,MD,UnitedStates R.Boitard IRISA,Rennes;Technicolor,Cesson-Sevigne,France S.Bonfiglio BarcoFIMI,Saronno(VA),Italy P.Bordes Technicolor,Cesson-Sevigne,France K.Bouatouch IRISA,Rennes,France R.Brémond ParisEstUniversity,Marne-la-Vallée,France D.R.Bull UniversityofBristol,Bristol,UnitedKingdom A.Chalmers UniversityofWarwick,Coventry,UnitedKingdom R.Cozot IRISA,Rennes,France M.P.DaSilva UniversityofNantes,Nantes,France xix xx Contributors S.Daly DolbyLaboratories,Inc.,SanFrancisco,CA,UnitedStates N.-T.Dang ParisEstUniversity,Marne-la-Vallée,France F.DeSimone TelecomParisTech,CNRSLTCI,Paris,France K.Debattista UniversityofWarwick,Coventry,UnitedKingdom F.Dufaux TelecomParisTech,CNRSLTCI,Paris,France G.Eilertsen LinköpingUniversity,Norrköping,Sweden S.Forchhammer TechnicalUniversityofDenmark,Bristol,Denmark E.François Technicolor,Cesson-Sevigne,France J.Froehlich DolbyLaboratories,Inc.,SanFrancisco,CA,UnitedStates O.Gallo NVIDIAResearch,SantaClara,CA,UnitedStates G.Guarnieri UniversityofTrieste,Trieste,Italy F.Guerrini UniversityofBrescia,Brescia,Italy S.Hajisharif LinköpingUniversity,Norrköping,Sweden B.Karr KennedySpaceCenter,FL,UnitedStates;UniversityofWarwick,Coventry, UnitedKingdom J.Korhonen TechnicalUniversityofDenmark,Bristol,Denmark J.Kronander LinköpingUniversity,Norrköping,Sweden T.Kunkel DolbyLaboratories,Inc.,SanFrancisco,CA,UnitedStates M.-C.Larabi UniversityofPoitiers,Poitiers,France Contributors xxi S.Lasserre Technicolor,Cesson-Sevigne,France P.LeCallet UniversityofNantes,Nantes,France F.LeLéannec Technicolor,Cesson-Sevigne,France C.Lèbre Binocle,Paris,France R.Leonardi UniversityofBrescia,Brescia,Italy O.Letz Binocle,Paris,France C.Loscos UniversityofReimsChampagne-Ardenne,Reims,France S.Mann UniversityofToronto;RotmanSchoolofManagementCDL,Toronto,ON,Canada;Meta, RedwoodCity,CA,UnitedStates C.Mantel TechnicalUniversityofDenmark,Bristol,Denmark R.K.Mantiuk UniversityofCambridge,Cambridge;BangorUniversity,Bangor,UnitedKingdom I.Martin UniversityofGirona,Girona,Spain S.Miller DolbyLaboratories,Inc.,SanFrancisco,CA,UnitedStates M.Narwaria UniversityofNantes,Nantes,France M.Okuda UniversityofKitakyushu,Kitakyushu,Japan R.R.Orozco UniversityofGirona,Girona,Spain T.Pouli Technicolor,Cesson-Sévigné,France Y.Pupulin Binocle,Paris,France G.Ramponi UniversityofTrieste,Trieste,Italy xxii Contributors E.Reinhard Technicolor,Cesson-Sévigné,France T.Richter UniversityofStuttgart,Stuttgart,Germany T.Ritschel MPIInformatik;SaarlandUniversity,Saarbrücken,Germany U.Seger RobertBoschGmbH,Leonberg,Germany P.Sen UniversityofCalifornia,SantaBarbara,CA,UnitedStates X.Shu McMasterUniversity,Hamilton,ON,Canada J.Unger LinköpingUniversity,Norrköping,Sweden G.Valenzise TelecomParisTech,CNRSLTCI,Paris,France C.Villa ParisEstUniversity,Marne-la-Vallée,France X.Wu McMasterUniversity,Hamilton,ON,Canada Y.Zhang UniversityofBristol,Bristol,UnitedKingdom Editor Biographies Frédéric Dufaux is a Centre National de la Recherche Scientifique (CNRS) research director at TelecomParisTech.HeisalsotheEditor-in-ChiefofSignalProcessing:ImageCommunication. Frédéric received his M.Sc. in physics and his Ph.D. in electrical engineering from the Swiss FederalInstituteofTechnologyinLausannein1990and1994,respectively.Hehasmorethan20years ofexperienceinresearch,previouslyholdingpositionsattheSwissFederalInstituteofTechnologyin Lausanne,EmitallSurveillance,Genimedia,Compaq,DigitalEquipment,MIT,andBellLabs.Hehas beeninvolvedinthestandardizationofdigitalvideoandimagingtechnologies,beingamemberofthe MPEGandJPEGcommittees.HeistherecipientoftwoISOawardsforhiscontributions.Frédéricisa fellowofIEEE.HewasViceGeneralChairofthe2014InternationalConferenceonImageProcessing. HeisanelectedmemberoftheIEEEImage,Video,andMultidimensionalSignalProcessingTechnical CommitteeandtheIEEEMultimediaSignalProcessingTechnicalCommittee.HeisalsoChairofthe European Association for Signal Processing (EURASIP) Special Area Team on Visual Information Processing. His research interests include image and video coding, distributed video coding, 3D video, high dynamicrange imaging,visual quality assessment, videosurveillance,privacyprotection,image and videoanalysis,multimediacontentsearchandretrieval,andvideotransmissionoverwirelessnetworks. He is the author or coauthor of more than 120 research publications and holds 17 patents issued or pending. PatrickLe Callet is Professorat the UniversityofNantes/PolytechNantesand leadsresearchatthe InstitutdeRechercheenCommunicationsetCybernétiquedeNantes(IRCCyN)/CentreNationaldela RechercheScientifique(CNRS)laboratory. He received both an M.Sc. and a Ph.D. in image processing from École Polytechnique de l’Université de Nantes. He was also a student at École Normale Superieure de Cachan, where he sattheaggrégation(competitiveexamination)inelectronicsoftheFrenchnationaleducationsystem. He worked as an assistant professor from 1997 to 1999 and as a full-time lecturer from 1999 to 2003 in the Department of Electrical Engineering of the Technical Institute of the University of Nantes. Since 2003 he has taught at École Polytechnique de l’Université de Nantes (Engineering School) in the Electrical Engineering Department and the Computer Science Department, where is now a full professor. Since 2006, he has been the head of the Image and Video Communication Laboratory at CNRS IRCCyN, a group of more than 35 researchers. He is mostly engaged in research dealing with the application of human vision modeling in image and video processing. His interests currently center on 3D image and video quality assessment, watermarking techniques, and visual attention modeling and applications. He is the coauthor of more than 200 publications and communications and the co-inventor named on 13 international patents on these topics. He also cochairs within the Video Quality Expert Group, the HDR Group, and 3DTV activities. He is currently serving as an associate editor of IEEE Transactions on Image Processing, IEEE Transactions on Circuit System and Video Technology, EURASIP Journal on Image and Video Processing,andSPIEElectronicImaging. xxiii xxiv Editor Biographies Rafał K. Mantiuk is a senior lecturer in the Computer Laboratory, University of Cambridge (UK). He received his Ph.D. from the Max Planck Institute for Computer Science (Germany, 2006), was a postdoctoral researcher at the University of British Columbia (Canada) and a lecturer at Bangor University(UK).HehaspublishednumerousjournalarticlesandconferencepaperspresentedatACM SIGGRAPH,Eurographics,IEEEComputerVisionandPatternRecognition,andSPIEHumanVision and Electronic Imagingconferences, has been awarded severalpatents, and was recognizedwith the HeinzBillingAward(2006).Hehasmadecontributionstothefieldofhighdynamicrangeimagingin theareasofvideocompression,tonemapping,andqualityassessment.Inhisworkheinvestigateshow knowledgeofthehumanvisualsystemandperceptioncanbeincorporatedwithincomputergraphics and imagingalgorithms.His recentinterests focuson the designof imagingalgorithmsthatadaptto human visual performance and viewing conditions in order to deliver the best images given limited resources,suchasbandwidth,computationtime,anddisplaycontrast. MartaMrak receivedDipl. Ing.andM.Sc. in electronicengineeringfromthe Universityof Zagreb, Croatia, and a Ph.D. from Queen Mary University of London, UK. Before joining the Research and Development Department at the BBC in 2010 to work on video compression research and the H.265/HEVC standardization, she was a postdoctoral researcher at the University of Surrey and QueenMaryUniversityofLondon.In2002,shewasawardedaGermanAcademicExchangeService (DAAD) scholarship for video compression research at the Heinrich Hertz Institute, Germany. She hascoauthoredmorethan100articles,bookchapters,andstandardizationcontributions,andcoedited the bookHigh-QualityVisualExperience(Springer,2010).Shehasbeeninvolvedin severalprojects fundedbyEuropeanandUKresearchcouncilsinrolesrangingfromresearchertoscientificcoordinator. She is a member of the Multimedia Signal Processing Technical Committee of the IEEE, a senior memberoftheIEEE,andanareaeditorofSignalProcessing:ImageCommunication,andhasbeena guesteditorofseveralspecialissuesinrelevantjournalsinthefield. Preface Producing truly realistic video is widely seen as the holy grail toward further improving the quality of experiencefor endusersof multimediaservices. Additionally,numerousprofessionalapplications canbemadepossibleonlybytheuseofvideosignalswithpropertiesbeyondthewidespread,butvery limited,parameterspace,suchasstandarddynamicrange.Severalnewapproacheshavebeenproposed and often successfully implemented in the history of the applications that use video data. Currently investigateddirectionsincludehighspatialresolution,highframerates,widecolorgamutandhigh-bit- depthrendering. The human visual system is able to perceive a wide range of colors and luminous intensities, as present in outdoor scenes in everyday life, ranging from bright sunshine to dark shadows. However, currenttraditionalimagingtechnologiescannotcapturenorreproducesuchabroadrangeofluminance. Theobjectiveofhighdynamicrange(HDR)imagingistoovercometheselimitations,henceleadingto morerealisticvideocontentandagreatlyenhanceduserexperience. HDRappliedtostillimageshasbeenanactivefieldofresearchanddevelopmentformanyyears, especiallyforphotography.However,itsextensiontovideocontenthasbeenconsideredonlyrecently. Thanks to rapid technological advancements, commercial HDR video cameras and HDR monitors are becoming available. Nevertheless, the effective deploymentof HDR video technologies involves redefiningcommoninterfacesforend-to-endcontentdelivery,whichinturnentailsmanytechnicaland scientificchallengesforbothacademiaandindustry. ThisbookprovidesanoverviewofrecentlyresearchedtechnologicaldirectionsforenablingHDR videoinkeyapplicationareas.Morespecifically,itcoversthestateoftheart,discussestheeffectiveness of various techniques, reviews some of the standardization efforts during the time of writing, and exploresnewresearchdirections. By providing a broad coverage, including general and advanced topics, the book should be of interest to a large readership with different backgrounds and expectations. Target readers include researchers,students,engineers,practitioners,andmanagers.Thisbookhasbeeneditedwiththeaim forittobethefirstreferenceonmanytopicsthatencompassHDRvideo,andtobearelevanthandbook forthoseinvolvedorinterestedinthisfield. The aim is to cover numerous aspects required to establish HDR video systems, ranging from content acquisition to display technologies. Although several applications that use HDR video technologies are discussed, the range of topics covered provides a solid base for the establishment ofnewHDRapplicationareas. Thisbookis composedofsix parts. Afteranintroductionto thebackgroundandfundamentalsof HDR imaging, Part I addresses the problem of HDR content acquisition and production. Different approachesare describedto captureHDR video,includingthe use of multiple synchronizedsensors, spatial multiplexing of the sensor response, merging of multiple images captured with different exposuretimes,andcapturingmultiviewHDRvideo.TheimpactofHDRoncinematographicshooting isalsodiscussed. xxv