Temporal coherency in video tone mapping Ronan Boitard To cite this version: Ronan Boitard. Temporal coherency in video tone mapping. Graphics [cs.GR]. Université Rennes 1, 2014. English. ￿NNT: 2014REN1S060￿. ￿tel-01112432v2￿ HAL Id: tel-01112432 https://theses.hal.science/tel-01112432v2 Submitted on 7 Mar 2015 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. ANN(cid:201)E 2014 THÈSE / UNIVERSITÉ DE RENNES 1 sous le sceau de l’Université Européenne de Bretagne pour le grade de DOCTEUR DE L’UNIVERSITÉ DE RENNES 1 Mention : Informatique École doctorale Matisse présentée par Ronan B OITARD préparée à l’unité de recherche IRISA Rennes Bretagne Atlantique Thèse soutenue à Rennes le 16 Octobre 2014 Temporal devantlejurycomposéde: Coherency in Rafał MANTIUK Ass. Professor,Univ. ofBangor / Rapporteur Video Tone Sumanta PATTANAIK Ass. Professor,Univ. ofCentralFlorida / Rapporteur Alan CHALMERS Mapping Professor,Univ. ofWarwick / Examinateur Frédéric DUFAUX ResearchDirector,TélécomParisTech / Examinateur Luce MORIN Professor,INSARennes / Examinateur Rémi COZOT Ass. Professor,Univ. ofRennes1 / Examinateur Dominique THOREAU Researcher,Technicolor / Examinateur Kadi BOUATOUCH Professor,Univ. ofRennes1 / Directeurdethèse Abstract One of the main goals of digital imagery is to improve the capture and the reproduction of real or synthetic scenes on display devices with restricted capabilities. Standard im- agery techniques are limited with respect to the dynamic range that they can capture and reproduce. High Dynamic Range (HDR) imagery aims at overcoming these limita- tionsbycapturing, representinganddisplayingthephysicalvalueoflightmeasuredina scene. However, current commercial displays will not vanish instantly hence backward compatibility between HDR content and those displays is required. This compatibility is ensured through an operation called tone mapping that retargets the dynamic range of HDR content to the restricted dynamic range of a display device. Although many tone mapping operators exist, they focus mostly on still images. The challenges of tone mapping HDR videos are more complex than those of still images since the tempo- ral dimensions is added. In this work, the focus was on the preservation of temporal coherency when performing video tone mapping. Two main research avenues are in- vestigated: the subjective quality of tone mapped video content and their compression e(cid:30)ciency. Indeed, tone mapping independently each frame of a video sequence leads to tem- poral artifacts. Those artifacts impair the visual quality of the tone mapped video se- quence and need to be reduced. Through experimentations with HDR videos and Tone Mapping Operators (TMOs), we categorized temporal artifacts into six categories. We testedvideotonemappingoperators(techniquesthattakeintoaccountmorethanasin- gle frame) for the di(cid:27)erent types of temporal artifact and we observed that they could handle only three out of the six types. Consequently, we designed a post-processing technique that adapts to any tone mapping operator and reduces the three types of artifact not dealt with. A subjective evaluation reported that our technique always preserves or increases the subjective quality of tone mapped content for the sequences and TMOs tested. The second topic investigated was the compression of tone mapped video content. So far, work on tone mapping and video compression focused on optimizing a tone map curve to achieve high compression ratio. These techniques changed the rendering of the video to reduce its entropy hence removing any artistic intent or constraint on the (cid:28)nal results. That is why, we proposed a technique that reduces the entropy of a tone mapped video without altering its rendering. Our method adapts the quantization to increase the correlation between successive frames. Results showed an average bit-rate reduction under the same PSNR ranging from 5.4% to 12.8%. Acknowledgments First of all, I would like to thank my three supervisors: Kadi Bouatouch, RØmi Cozot and Dominique Thoreau. Each of you helped me with di(cid:27)erent aspects of this thesis and were complimentary of each other. Thank you for the support, fruitful discussions and most importantly all the time you spent advising me. I would also like to thanks my colleagues from Technicolor and the FRSense team to provide a work place that is more than just a place to work. I will de(cid:28)nitely miss the co(cid:27)ee breaks, lunch breaks and any type of breaks we had. I am really glad to have met you all. Many thanks as well to Adrien Gruson, Mickaºl RibardiŁre and Ricardo MarquŁs for designing and rendering the computer generated sequences so badly needed for all the tests. A special thanks to all the members and participants of the COST IC1005 action. It has been a pleasure to attend to all those meetings, training schools and workshops and to meet so many wonderful people. Finally,Iwouldliketothankallmyfriendsforalltheweek-ends,trips,musicfestival andAmaryllisnights. AlthoughMondaysweretoughsometimes,theseactivitiesallowed me to keep the balance between work and social life. Contents Contents 3 1 Introduction 5 1.1 Tone Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.4 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 Background in High Dynamic Range Imaging 11 2.1 Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.1 Light . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.2 Human Vision System . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1.3 Colorimetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 HDR Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.1 Computer Graphics Content . . . . . . . . . . . . . . . . . . . . . 16 2.2.2 Radiance File Format . . . . . . . . . . . . . . . . . . . . . . . . 16 2.2.3 OpenEXR File Format . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2.4 Real-World HDR capture . . . . . . . . . . . . . . . . . . . . . . 17 2.3 Display of HDR/LDR Content . . . . . . . . . . . . . . . . . . . . . . . 18 2.3.1 Display Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.2 HDR to LDR conversion . . . . . . . . . . . . . . . . . . . . . . . 21 2.3.3 HDR Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.3.4 LDR to HDR Conversion . . . . . . . . . . . . . . . . . . . . . . 31 2.4 Video Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.4.1 HEVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.4.2 HDR Backward Compatible Video Compression . . . . . . . . . . 34 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3 Video Tone Mapping 37 3.1 Temporal Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.1.1 Flickering Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.1.2 Temporal Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.1.3 Temporal Brightness Incoherency . . . . . . . . . . . . . . . . . . 40 3.1.4 Temporal Object Incoherency . . . . . . . . . . . . . . . . . . . . 41 1 2 Contents 3.1.5 Temporal Hue Incoherency . . . . . . . . . . . . . . . . . . . . . 43 3.2 Video TMOs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.2.1 Global Temporal Filtering . . . . . . . . . . . . . . . . . . . . . . 45 3.2.2 Local Temporal Filtering . . . . . . . . . . . . . . . . . . . . . . 48 3.2.3 Detection and Reduction of Artifacts by Post-Processing . . . . . 48 3.3 Temporal Artifacts caused by Video TMOs . . . . . . . . . . . . . . . . 49 3.3.1 Temporal Contrast Adaptation . . . . . . . . . . . . . . . . . . . 49 3.3.2 Ghosting Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4 Global Brightness Coherency 53 4.1 Temporal Contrast and Number of Tonal Levels . . . . . . . . . . . . . . 53 4.2 Brightness Coherency Post-Processing . . . . . . . . . . . . . . . . . . . 56 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.3.1 Temporal Brightness Coherency . . . . . . . . . . . . . . . . . . . 58 4.3.2 Temporal Object Coherency . . . . . . . . . . . . . . . . . . . . . 60 4.3.3 Video Fade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.4 Limitations of the BC Post-Processing . . . . . . . . . . . . . . . . . . . 61 4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5 Zonal Brightness Coherency 65 5.1 Global and Local Post-Processing . . . . . . . . . . . . . . . . . . . . . . 65 5.2 Zonal Brightness Coherency Post-Processing . . . . . . . . . . . . . . . . 66 5.2.1 Frame Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.2.2 Video Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.2.3 Applying BC to video zones . . . . . . . . . . . . . . . . . . . . . 70 5.2.4 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.3.1 Spatial Contrast and Brightness Coherency . . . . . . . . . . . . 74 5.3.2 Local Brightness Coherency . . . . . . . . . . . . . . . . . . . . . 76 5.3.3 Hue Coherency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 5.3.4 Subjective Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 81 5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6 Compression of Tone Mapped Video Contents 85 6.1 Video Tone Mapping and Video Compression . . . . . . . . . . . . . . . 85 6.2 Temporal Coherency and Inter-Prediction . . . . . . . . . . . . . . . . . 88 6.3 Compression of Tone Mapped Video Sequence . . . . . . . . . . . . . . . 89 6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 7 Motion-Guided Quantization for Video Tone Mapping 95 7.1 Quantization in Video Tone Mapping . . . . . . . . . . . . . . . . . . . . 95 7.2 Motion-Guided Quantization . . . . . . . . . . . . . . . . . . . . . . . . 96 7.2.1 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Contents 3 7.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.3.1 Quantization Loss . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.3.2 Compression E(cid:30)ciency . . . . . . . . . . . . . . . . . . . . . . . . 100 7.3.3 Improving Performances . . . . . . . . . . . . . . . . . . . . . . . 101 7.3.4 Denoising Application . . . . . . . . . . . . . . . . . . . . . . . . 102 7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8 Conclusion 105 8.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 RØsumØ en Fran(cid:231)ais 109 Travaux de la thŁse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 A HDR Sequences 115 A.1 Max Planck Institute fur Informatiks (MPII) Sequences . . . . . . . . . 115 A.2 OpenFootage Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 A.3 IRISA Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 A.4 Nevex Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 A.5 Link(cid:246)ping Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Bibliography 131