IMMERSIVE VIDEO TECHNOLOGIES This page intentionally left blank IMMERSIVE VIDEO TECHNOLOGIES Editedby GIUSEPPEVALENZISE UniversitéParis-Saclay,CNRS,CentraleSupélec Laboratoiredessignauxetsystèmes Gif-sur-Yvette,France MARTINALAIN HuaweiIrelandResearchCentre Dublin,Ireland EMINZERMAN STCResearchCenter MidSwedenUniversity Sundsvall,Sweden CAGRIOZCINAR SamsungR&DUK London,UnitedKingdom AcademicPressisanimprintofElsevier 125LondonWall,LondonEC2Y5AS,UnitedKingdom 525BStreet,Suite1650,SanDiego,CA92101,UnitedStates 50HampshireStreet,5thFloor,Cambridge,MA02139,UnitedStates TheBoulevard,LangfordLane,Kidlington,OxfordOX51GB,UnitedKingdom Copyright©2023ElsevierInc.Allrightsreserved. MATLAB®isatrademarkofTheMathWorks,Inc.andisusedwithpermission. TheMathWorksdoesnotwarranttheaccuracyofthetextorexercisesinthisbook. Thisbook’suseordiscussionofMATLAB®softwareorrelatedproductsdoesnotconstituteendorsementor sponsorshipbyTheMathWorksofaparticularpedagogicalapproachorparticularuseoftheMATLAB®software. Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,electronicor mechanical,includingphotocopying,recording,oranyinformationstorageandretrievalsystem,without permissioninwritingfromthepublisher.Detailsonhowtoseekpermission,furtherinformationaboutthe Publisher’spermissionspoliciesandourarrangementswithorganizationssuchastheCopyrightClearanceCenter andtheCopyrightLicensingAgency,canbefoundatourwebsite:www.elsevier.com/permissions. ThisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightbythePublisher(other thanasmaybenotedherein). Notices Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchandexperiencebroadenour understanding,changesinresearchmethods,professionalpractices,ormedicaltreatmentmaybecomenecessary. Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgeinevaluatingandusingany information,methods,compounds,orexperimentsdescribedherein.Inusingsuchinformationormethodsthey shouldbemindfuloftheirownsafetyandthesafetyofothers,includingpartiesforwhomtheyhaveaprofessional responsibility. Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,oreditors,assumeanyliability foranyinjuryand/ordamagetopersonsorpropertyasamatterofproductsliability,negligenceorotherwise,or fromanyuseoroperationofanymethods,products,instructions,orideascontainedinthematerialherein. ISBN:978-0-323-91755-1 ForinformationonallAcademicPresspublications visitourwebsiteathttps://www.elsevier.com/books-and-journals Publisher:MaraConner AcquisitionsEditor:TimPitts EditorialProjectManager:HowiM.DeRamos ProductionProjectManager:PremKumarKaliamoorthi CoverDesigner:VickyPearsonEsser TypesetbyVTeX Contents Listofcontributors xi Preface xvii PART1 Foundations 1. Introductiontoimmersivevideotechnologies 3 MartinAlain,EminZerman,CagriOzcinar,andGiuseppeValenzise 1.1. Introduction 3 1.2. Whatisimmersion? 4 1.3. Immersivevideo 8 1.4. Challengesintheprocessingandcontentdeliverypipeline 13 1.5. Discussionandperspectives 20 References 21 PART2 Omnidirectionalvideo 2. Acquisition,representation,andrenderingofomnidirectionalvideos 27 ThomasMaugey 2.1. Introduction 27 2.2. Acquisitionandprojectionmodel 29 2.3. Sphererepresentation 36 2.4. Rendering 43 2.5. Towardsmulti-viewacquisition 44 References 45 3. Streaminganduserbehaviorinomnidirectionalvideos 49 SilviaRossi,AlanGuedes,andLauraToni 3.1. Introduction 49 3.2. Streamingpipeline:evolutiontowardsODV 53 3.3. System-centricstreaming 58 3.4. TheroleoftheuserinODV 62 3.5. User-centricODVstreaming 71 3.6. Conclusionsandperspectives 75 References 77 v vi Contents 4. Subjectiveandobjectivequalityassessmentforomnidirectionalvideo 85 SimoneCroci,AshutoshSingla,StephanFremerey,AlexanderRaake,andAljosaSmolic 4.1. Introduction 85 4.2. Subjectivequalityassessment 88 4.3. Short-termvideoassessment 91 4.4. Long-termvideoassessment 96 4.5. Tile-basedstreaming 97 4.6. Objectivequalityassessment 98 4.7. VIVA-Q:omnidirectionalvideoqualityassessmentbasedonplanarVoronoipatchesand visualattention 108 4.8. Conclusion 113 References 114 5. Omnidirectionalvideosaliency 123 Fang-YiChao,FedericaBattisti,PierreLebreton,andAlexanderRaake 5.1. Introduction 123 5.2. Collectinguserdata 126 5.3. Userbehavioranalysis 128 5.4. Modeling 130 5.5. Metrics 144 5.6. Conclusionandoutlook 150 References 152 PART3 Lightfields Preliminariesonlightfields 161 6. Acquisitionoflightfieldimages&videos 163 ThorstenHerfet,KelvinChelli,andMikaelLePendu 6.1. Plenopticcameras 164 6.2. Lightfieldgantries 168 6.3. Cameraarrays 169 References 171 7. Lightfieldrepresentation 173 ThorstenHerfet,KelvinChelli,andMikaelLePendu 7.1. Spacedomainrepresentation 173 7.2. Frequencydomainrepresentation 182 Contents vii 7.3. Depth-basedrepresentation 188 7.4. Alternativerepresentations 195 References 198 8. Compressionoflightfields 201 MilanStepanov,GiuseppeValenzise,andFrédéricDufaux 8.1. Introduction 201 8.2. Transform-basedmethods 201 8.3. Prediction-basedmethods 207 8.4. JPEGPleno 214 8.5. Deeplearning-basedmethods 218 8.6. Performanceanalysis 221 8.7. Conclusionsandperspectives 222 References 223 9. Lightfieldprocessingformediaapplications 227 JoachimKeinert,LauraFink,FlorianGoldmann,MuhammadShahzebKhanGul, TobiasJaschke,NicoPrappacher,MatthiasZiegler,MichelBätz,andSiegfriedFößel 9.1. Lightfieldprocessingchainoverview 227 9.2. Imageacquisitionandgeometriccameracalibration 229 9.3. Depthreconstruction 230 9.4. Interactivelightfieldprocessing 232 9.5. Lightfieldrendering 233 9.6. Real-timerenderingingameengines 236 9.7. Neuralrenderingand3Dreconstruction 243 9.8. Conclusionandopenchallenges 255 References 255 10. Qualityevaluationoflightfields 265 AliAkandPatrickLeCallet 10.1. Introduction 265 10.2. CharacteristicsofLF-relateddistortions 266 10.3. Subjectivequalityassessment 270 10.4. Objectivequalityassessment 277 10.5. Conclusion 283 References 284 viii Contents PART4 Volumetricvideo 11. Volumetricvideo–acquisition,interaction,streamingandrendering 289 PeterEisert,OliverSchreer,IngoFeldmann,CorneliusHellge,andAnnaHilsmann 11.1. Creationofvolumetricvideo 290 11.2. Animatingandinteractingwithvolumetricvideo 299 11.3. Streamingvolumetricvideo 310 11.4. Conclusions 320 References 321 12. MPEGimmersivevideo 327 PatrickGarus,MartaMilovanovic´,JoëlJung,andMarcoCagnazzo 12.1. MIVdescriptionandprofiles 329 12.2. TMIVdescription 331 12.3. Commontestconditions 335 12.4. EvaluationofMIVwithdifferentconfigurationsand2Dcodecs 338 12.5. Conclusion 351 References 354 13. Pointcloudcompression 357 GiuseppeValenzise,MauriceQuach,DongTian,JiahaoPang,andFrédéricDufaux 13.1. Introduction 357 13.2. Basictoolsforpointcloudcompression 359 13.3. MPEGPCCstandardization 364 13.4. Learning-basedtechniques 372 13.5. Conclusionsandperspectives 379 References 380 14. Codingofdynamic3Dmeshes 387 Jean-EudesMarvie,MajaKrivokuc´a,andDanilloGraziosi 14.1. Introduction 387 14.2. Meshfundamentals 390 14.3. Staticmeshes 397 14.4. Constant-connectivitymeshsequences 402 14.5. Variable-connectivitymeshsequences 411 14.6. Conclusionandfuturedirections 416 References 417 Contents ix 15. Volumetricvideostreaming 425 IreneViolaandPabloCesar 15.1. Theoreticalapproachestovolumetricvideostreaming 426 15.2. Volumetricvideostreamingsystems 434 15.3. Conclusion 437 References 438 16. Processingofvolumetricvideo 445 SihengChenandJinZeng 16.1. Introduction 445 16.2. Restoration 446 16.3. Semanticsegmentation 454 16.4. Objectdetection 461 16.5. Chapterataglance 465 References 465 17. Computational3Ddisplays 469 JingyuLiu,FangchengZhong,ClaireMantel,SørenForchhammer,andRafałK.Mantiuk 17.1. Introduction 469 17.2. Perceptualconsiderations 470 17.3. Computational3Ddisplays 475 17.4. Summaryanddiscussion 492 18. Subjectiveandobjectivequalityassessmentforvolumetricvideo 501 EvangelosAlexiou,YanaNehmé,EminZerman,IreneViola,GuillaumeLavoué,AliAk, AljosaSmolic,PatrickLeCallet,andPabloCesar 18.1. Subjectivequalityassessment 503 18.2. Objectivequalityassessment 522 18.3. Conclusion 539 References 544 PART5 Applications 19. MRinvideoguidedliversurgery 555 RafaelPalomar,RahulPrasannaKumar,CongcongWang,EgidijusPelanis,and FaouziAlayaCheikh 19.1. Introduction 555 19.2. Medicalimagespre-processing 558 19.3. 3Dlivermodeling 562