3D VISUAL COMMUNICATIONS 3D VISUAL COMMUNICATIONS Guan-Ming Su DolbyLabs,California,USA Yu-Chi Lai NationalTaiwanUniversityofScienceandTechnology,Taiwan Andres Kwasinski RochesterInstituteofTechnology,NewYork,USA Haohong Wang TCLResearchAmerica,California,USA A John Wiley & Sons, Ltd., Publication Thiseditionfirstpublished2013 2013JohnWileyandSonsLtd Registeredoffice JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UnitedKingdom Fordetailsofourglobaleditorialoffices,forcustomerservicesandforinformationabouthowtoapplyforpermissionto reusethecopyrightmaterialinthisbookpleaseseeourwebsiteatwww.wiley.com. TherightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththeCopyright, DesignsandPatentsAct1988. Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inany formorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbytheUK Copyright,DesignsandPatentsAct1988,withoutthepriorpermissionofthepublisher. Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynotbeavailable inelectronicbooks. Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrandnamesand productnamesusedinthisbookaretradenames,servicemarks,trademarksorregisteredtrademarksoftheirrespective owners.Thepublisherisnotassociatedwithanyproductorvendormentionedinthisbook.Thispublicationisdesigned toprovideaccurateandauthoritativeinformationinregardtothesubjectmattercovered.Itissoldontheunderstanding thatthepublisherisnotengagedinrenderingprofessionalservices.Ifprofessionaladviceorotherexpertassistanceis required,theservicesofacompetentprofessionalshouldbesought. LibraryofCongressCataloging-in-PublicationData Su,Guan-Ming. 3Dvisualcommunications/Guan-MingSu,Yu-ChiLai,AndresKwasinski,HaohongWang. pagescm Includesbibliographicalreferencesandindex. ISBN978-1-119-96070-6(cloth) 1. Multimediacommunications. 2. Three-dimensionaldisplaysystems. I. Lai,Yu-Chi. II. Kwasinski,Andres. III. Wang,Haohong,1973- IV. Title. V. Title:Threedimensionalvisualcommunications. TK5105.15.S832013 006.7–dc23 2012031377 AcataloguerecordforthisbookisavailablefromtheBritishLibrary. ISBN:978-1-119-96070-6 Typesetin10/12ptTimesbyLaserwordsPrivateLimited,Chennai,India Contents Preface ix About the Authors xiii 1 Introduction 1 1.1 Why 3D Communications? 1 1.2 End-to-End 3D Visual Ecosystem 3 1.2.1 3D Modeling and Representation 5 1.2.2 3D Content Creation 6 1.2.3 3D Video Compression 7 1.2.4 3D Content Delivery 8 1.2.5 3D Display 9 1.2.6 3D QoE 9 1.3 3D Visual Communications 10 1.4 Challenges and Opportunities 11 References 15 2 3D Graphics and Rendering 17 2.1 3DTV Content Processing Procedure 19 2.2 3D Scene Representation with Explicit Geometry – Geometry Based Representation 22 2.2.1 Surface Based Representation 23 2.2.2 Point Based Representation 37 2.2.3 Point Based Construction 38 2.2.4 Point Based Compression and Encoding for Transmission 38 2.2.5 Point Based Rendering: Splatting 39 2.2.6 Volumetric Representation 40 2.2.7 Volumetric Construction 40 2.2.8 Volumetric Compression and Encoding for Transmission 41 2.2.9 Volumetric Rendering 42 2.3 3D Scene Representation without Geometry – Image-Based Representation 43 2.3.1 Plenoptic Function 43 2.3.2 Single Texture Representation 46 2.3.3 Multiple Texture Representation 48 2.3.4 Image Based Animation 51 vi Contents 2.4 3D Scene Representation with Implicit Geometry – Depth-Image-Based Representation 51 2.4.1 History of Depth-Image-Based Representation 52 2.4.2 Fundamental Concept Depth-Image-Based Representation 53 2.4.3 Depth Construction 56 2.4.4 Depth-Image-Based Animation 57 References 57 3 3D Display Systems 63 3.1 Depth Cues and Applications to 3D Display 63 3.1.1 Monocular Depth Cues 63 3.1.2 Binocular Depth Cues 64 3.2 Stereoscopic Display 65 3.2.1 Wavelength Division (Color) Multiplexing 65 3.2.2 Polarization Multiplexing 69 3.2.3 Time Multiplexing 69 3.3 Autostereoscopic Display 71 3.3.1 Occlusion-Based Approach 71 3.3.2 Refraction-Based Approach 75 3.4 Multi-View System 78 3.4.1 Head Tracking Enabled Multi-View Display 79 3.4.2 Automultiscopic 79 3.5 Recent Advances in Hologram System Study 83 References 84 4 3D Content Creation 85 4.1 3D Scene Modeling and Creation 85 4.1.1 Geometry-Based Modeling 86 4.1.2 Image-Based Modeling 86 4.1.3 Hybrid Approaches 87 4.2 3D Content Capturing 87 4.2.1 Stereo Camera 87 4.2.2 Depth Camera 88 4.2.3 Multi-View Camera 88 4.2.4 3D Capturing with Monoscopic Camera 89 4.3 2D-to-3D Video Conversion 101 4.3.1 Automatic 2D-to-3D Conversion 103 4.3.2 Interactive 2D-to-3D Conversion 111 4.3.3 Showcase of 3D Conversion System Design 112 4.4 3D Multi-View Generation 125 References 126 5 3D Video Coding and Standards 129 5.1 Fundamentals of Video Coding 129 5.2 Two-View Stereo Video Coding 142 5.2.1 Individual View Coding 142 5.2.2 Inter-View Prediction Stereo Video Coding 143 Contents vii 5.3 Frame-Compatible Stereo Coding 144 5.3.1 Half-Resolution Frame-Compatible Stereo Coding 144 5.3.2 Full-Resolution Frame-Compatible Layer Approach 146 5.4 Video Plus Depth Coding 148 5.5 Multiple View Coding 156 5.6 Multi-View Video Plus Depth (MVD) Video 160 5.7 Layered Depth Video (LDV) 163 5.8 MPEG-4 BIFS and AFX 165 5.9 Free-View Point Video 166 References 167 6 Communication Networks 171 6.1 IP Networks 171 6.1.1 Packet Networks 171 6.1.2 Layered Network Protocols Architecture 172 6.2 Wireless Communications 174 6.2.1 Modulation 175 6.2.2 The Wireless Channel 177 6.2.3 Adaptive Modulation and Coding 191 6.3 Wireless Networking 193 6.4 4G Standards and Systems 193 6.4.1 Evolved Universal Terrestrial Radio Access Network (E-UTRAN) 195 6.4.2 Evolved Packet Core (EPC) 200 6.4.3 Long Term Evolution-Advance (LTE-A) 201 6.4.4 IEEE 802.16 – WiMAX 202 References 203 7 Quality of Experience 205 7.1 3D Artifacts 205 7.1.1 Fundamentals of 3D Human Visual System 205 7.1.2 Coordinate Transform for Camera and Display System 206 7.1.3 Keystone Distortion 211 7.1.4 Depth-Plane Curvature 212 7.1.5 Shear Distortion 212 7.1.6 Puppet-Theater Effect 213 7.1.7 Cardboard Effect 215 7.1.8 Asymmetries in Stereo Camera Rig 216 7.1.9 Crosstalk 217 7.1.10 Picket-Fence Effect and Lattice Artifacts 217 7.1.11 Hybrid DCT Lossy Compression Artifact 218 7.1.12 Depth Map Bleeding and Depth Ringing 219 7.1.13 Artifacts Introduced by Unreliable Communication Networks 219 7.1.14 Artifacts from New View Synthesis 219 7.1.15 Summary of 3D Artifacts 220 7.2 QoE Measurement 220 7.2.1 Subjective Evaluations 222 7.2.2 2D Image and Video QoE Measurement 226 viii Contents 7.2.3 3D Video HVS Based QoE Measurement 235 7.2.4 Postscript on Quality of Assessment 246 7.3 QoE Oriented System Design 247 7.3.1 Focus Cues and Perceptual Distortions 247 7.3.2 Visual Fatigue 249 References 250 8 3D Video over Networks 259 8.1 Transmission-Induced Error 259 8.2 Error Resilience 267 8.3 Error Concealment 270 8.4 Unequal Error Protection 275 8.5 Multiple Description Coding 279 8.6 Cross-Layer Design 282 References 286 9 3D Applications 289 9.1 Glass-Less Two-View Systems 289 9.1.1 Spatially Multiplexed Systems 290 9.1.2 Temporally Multiplexed Systems 290 9.2 3D Capture and Display Systems 291 9.3 Two-View Gaming Systems 294 9.4 3D Mobile 298 9.4.1 HTC EVO 3D 298 9.4.2 Mobile 3D Perception 299 9.5 Augmented Reality 302 9.5.1 Medical Visualization 304 9.5.2 Mobile Phone Applications 306 References 309 10 Advanced 3D Video Streaming Applications 313 10.1 Rate Control in Adaptive Streaming 313 10.1.1 Fundamentals of Rate Control 313 10.1.2 Two-View Stereo Video Streaming 318 10.1.3 MVC Streaming 318 10.1.4 MVD Streaming 319 10.2 Multi-View Video View Switching 321 10.3 Peer-to-Peer 3D Video Streaming 325 10.4 3D Video Broadcasting 328 10.5 3D Video over 4G Networks 329 References 331 Index 335 Preface AstheAvatar3Dmovieexperienceswepttheworldin2010,3Dvisualcontenthasbecome themosteye-catchingspotintheconsumerelectronicsproducts.This3Dvisualwavehas spreadto3DTV,Blu-ray,PC,mobile,andgamingindustries,asthe3Dvisualsystempro- vides sufficient depth cues for end users to acquire better understanding of the geometric structureofthecapturedscenes,andnonverbalsignalsandcuesinvisualconversation.In addition,3Dvisualsystemsenableobserverstorecognizethephysicallayoutandlocation for each object with immersive viewing experiences and natural user interaction, which also makes it an important topic for both academic and industrial researchers. Living in an era of widespread mobility and networking, where almost all consumer electronic devices are endpoints of the wireless/wired networks, the deployment of 3D visual representation will significantly challenge the network bandwidth as well as the computational capability of terminal points. In other words, the data volume received in an endpoint required to generate 3D views will be many times that of a single view in a 2D system, and hence the new view generation process sets a higher requirement for the endpoint’s computational capability. Emerging 4G communication systems fit very well into the timing of 3D visual communications by significantly improving the bandwidth as well as introducing many new features designed specifically for high-volume data communications. In this book, we aim to provide comprehensive coverage of major theories and prac- tices involved in the lifecycle of a 3D visual content delivery system. The book presents technologiesusedinanend-to-end3Dvisualcommunicationsystem,includingthefunda- mentalsof3Dvisualrepresentation,thelatest3Dvideocodingtechniques,communication infrastructure and networks in 3D communications, and 3D quality of experience. Thisbooktargetsprofessionalsinvolvedintheresearch,design,anddevelopmentof3D visual coding and 3D visual transmission systems and technologies. It provides essential reading for students, engineers, and academic and industrial researchers. This book is a comprehensivereferenceforlearningallaspectsof3Dgraphicsandvideocoding,content creation and display, and communications and networking. Organization of the book This book is organized as three parts: • principles of 3D visual systems: 3D graphics and rending, 3D display, and 3D content creation are all well covered x Preface • visual communication: fundamental technologies used in 3D video coding and communication system, and the quality of experience. There are discussions on various 3D video coding formats and different communication systems, to evaluate the advantages of each system • advances and applications of 3D visual communication Chapter 1 overviews the whole end-to-end 3D video ecosystem, in which we cover key componentsinthepipeline:the3Dsourcecoding,pre-processing,communicationsystem, post-processing, and system-level design. We highlight the challenges and opportunities for 3D visual communication systems to give readers a big picture of the 3D visual content deployment technology, and point out which specific chapters relate to the listed advanced application scenarios. 3D scene representations are the bridging technology for the entire 3D visual pipeline fromcreationtovisualization.Different3Dscenerepresentationsexhibitdifferentcharac- teristics and the selections should be chosen according to the requirement of the targeted applications. Various techniques can be categorized according to the amount of geomet- ric information used in the 3D representation spectrum; at one extreme is the simplest form via rendering without referring to any geometry, and the other end uses geometrical description. Both extremes of the technology have their own advantages and disadvan- tages. Therefore, hybrid methods, rendering with implicit geometries, are proposed to combine the advantages and disadvantages of both ends of the technology spectrum to better support the needs of stereoscopic applications. In Chapter 2, a detailed discussion about three main categories for 3D scene representations is given. InChapter3,weintroducethedisplaytechnologiesthatallowtheenduserstoperceive 3D objects. 3D displays are the direct interfaces between the virtual world and human eyes and these play an important role in reconstructing 3D scenes. We first describe the fundamentals of the human visual system (HVS) and discuss depth cues. Having this background,weintroducethesimplestscenariotosupportstereoscopictechnologies(two- viewonly)withaidedglasses.Then,thecommonstereoscopictechnologieswithoutaided glasses are presented. Display technologies to support multiple views simultaneously are addressed to cover the head-tracking-enabled multi-view display, occlusion-based and reflection-based multi-view system. At the end of this chapter, we will briefly discuss the holographic system. InChapter4,welookat3Dcontentcreationmethods,from3Dmodelingandrepresen- tation, capturing, 2D to 3D conversion and, to 3D multi-view generation. We showcase three practical examples that are adopted in industrial 3D creation process to provide a clear picture of how things work together in a real 3D creation system. It has been observed that 3D content has significantly higher storage requirements compared to their 2D counterparts. Introducing compression technologies to reduce the requiredstoragesizeandalleviatetransmissionbandwidthisveryimportantfordeploying 3D applications. In Chapter 5, we introduce 3D video coding and related standards. We will first cover the fundamental concepts and methods used in conventional 2D video codecs, especially the state-of-the-art H.264 compression method and the recent devel- opment of next generation video codec standards. With common coding knowledge, we first introduce two-view video coding methods which have been exploited in the past decade. Several methods, including individual two-view coding, simple inter-view pre- diction stereo video coding, and the latest efforts on frame-compatible stereo coding, are Preface xi presented.Researchonthedepthinformationtoreconstructthe3Dscenehasbroughtsome improvements and the 3D video coding can benefit from introducing depth information into the coded bit stream. We describe how to utilize and compress the depth information in the video-plus-depth coding system. Supporting multi-view video sequence compres- sion is an important topic as multi-view systems provide a more immersive viewing experience. We will introduce the H.264 multiple view coding (MVC) for this particu- lar application. More advanced technologies to further reduce the bit rate for multi-view systems, such as the multi-view video plus depth coding and layered depth video coding system, are introduced. At the end of this chapter, the efforts on the 3D representation in MPEG-4, such as binary format for scenes (BIFS) and animation framework extension (AFX), are presented. The ultimate goal for 3D video system, namely, the free viewpoint system, is also briefly discussed. In Chapter 6, we present a review of the most important topics in communication networks that are relevant to the subject matter of this book. We start by describing the mainarchitectureofpacketnetworkswithafocusonthosebasedontheInternetprotocol (IP)networks.Herewedescribethelayeredorganizationofnetworkprotocols.Afterthis, weturnourfocustowirelesscommunications,describingthemaincomponentsofdigital wireless communications systems followed by a presentation of modulation techniques, the characteristics of the wireless channels, and adaptive modulation and coding. These topics are then applied in the description of wireless networks and we conclude with a study of fourth generation (4G) cellular wireless standards and systems. To make 3D viewing systems more competitive relative to 2D systems, the quality of experience (QoE) shown from 3D systems should provide better performance than from 2D systems. Among different 3D systems, it is also important to have a systematic way to compare and summarize the advances and assess the disadvantages. In Chapter 7, we discuss the quality of experience in 3D systems. We first present the 3D artifacts which may be induced throughout the whole content life cycle: content capture, content creation, content compression, content delivery, and content display. In the second part, we address how to measure the quality of experience for 3D systems subjectively and objectively. With those requirements in mind, we discuss the important factors to design a comfortable and high-quality 3D system. Chapter 8 addresses the main issue encountered when transmitting 3D video over a channel: that of dealing with errors introduced during the communication process. The chapter starts by presenting the effects of transmission-induced errors following by a discussion of techniques to counter these errors, such as the error resilience, error con- cealment,unequalerrorprotection,andmultipledescriptioncoding.Thechapterconcludes with a discussion of cross-layer approaches. Developing 3D stereoscopic applications has become really popular in the software industry.3Dstereoscopicresearchandapplicationsareadvancingrapidlyduetothecom- mercial need and the popularity of 3D stereoscopic products. Therefore, Chapter 9 gives a short discussion of commercially available products and technologies for application development. The discussed topics include commercially available glass-less two-view systems, depth adaptation capturing and displaying systems, two-view gaming systems, mobile 3D systems and perception, and 3D augmented reality systems. In the final chapter, we introduce the state-of-the-art technologies for delivering compressed 3D content over communication channels. Subject to limited bandwidth