www.it-ebooks.info Learning Virtual Reality Developing Immersive Experiences and Applications for Desktop, Web and Mobile Tony Parisi Boston www.it-ebooks.info Learning Virtual Reality by Tony Parisi Copyright © 2015 Tony Parisi. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles ( http://safaribooksonline.com ). For more information, contact our corporate/ institutional sales department: 800-998-9938 or [email protected] . Editor: Meg Foley Proofreader: FILL IN PROOFREADER Production Editor: FILL IN PRODUCTION EDI‐ Indexer: FILL IN INDEXER TOR Interior Designer: David Futato Copyeditor: FILL IN COPYEDITOR Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest January -4712: First Edition Revision History for the First Edition 2015-07-06: First Release See http://oreilly.com/catalog/errata.csp?isbn=0636920038467 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning Virtual Reality, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the author(s) have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author(s) disclaim all responsibil‐ ity for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 063-6-920-03846-7 [FILL IN] www.it-ebooks.info Table of Contents 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 What is Virtual Reality? 9 Stereoscopic Displays 9 Motion Tracking Hardware 11 Input Devices 12 Computing Platforms 13 Virtual Reality Applications 14 Chapter Summary 16 2. Virtual Reality Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 The Oculus Rift 19 The Oculus Rift is a stereoscopic display with built-in head motion tracking sensors. It straps to the head, allowing hands-free operation. The Rift is a peripheral: it attaches to a computer: Mac, Linux or Windows; desktop or laptop. The Rift is tethered, with a cable running to the computer. At the moment the Rift is quite bulky-- but that will most certainly change with the newer models being designed as we speak. The DK1 20 The DK2 21 “Crescent Bay” 23 Setting Up Your Oculus Rift 24 Samsung Gear VR: Deluxe, Portable Virtual Reality 25 Anticipating these issues, Oculus has also produced a much lighter-weight mobile solution. Through a partnership with Samsung, Oculus technology has been incorporated into , a revolutionary headset that combines Oculus optics (those barrel distortion lenses) with new head tracking technology, iii www.it-ebooks.info placed in a custom headset that houses a mobile phone with a high- resolution display. The Oculus Mobile SDK 27 Google Cardboard: Low-Cost VR for Smart Phones 27 Stereo Rendering and Head Tracking with Cardboard VR 29 Cardboard VR Input 30 Developing for Google Cardboard 30 Chapter Summary 30 3. Going Native: Developing for Oculus Rift on the Desktop. . . . . . . . . . . . . . . . . . . . . . . . . 33 In programming our first VR application, we will explore the following core concepts: 3D Graphics Basics 34 3D: A Definition 34 3D Coordinate Systems 34 Meshes, Polygons and Vertices 35 Materials, Textures and Lights 36 Transforms and Matrices 37 Cameras, Perspective, Viewports and Projections 39 Stereoscopic Rendering 40 Unity3D: The Game Engine for the Common Man 41 Setting up the Oculus SDK 43 Setting Up Your Unity Environment for Oculus Development 44 Your First VR Application 45 Building and Running the Application 48 Walking Through the Code 50 Chapter Summary 54 4. Going Mobile: Developing for Gear VR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 The Gear VR User Interface and Oculus Home 57 Using the Oculus Mobile SDK 57 Setting up the Android SDK 58 Generating an Oculus Signature File 58 Setting Up Your Device for USB Debugging 59 Developing for Gear VR Using Unity3D 59 Setting Up Your Unity3D Environment 60 A Simple Unity3D Sample 61 Handling Touchpad Events 68 Implementing Gaze Tracking 70 Deploying Applications for Gear VR 73 Chapter Summary 73 iv | Table of Contents www.it-ebooks.info 5. WebVR: Browser-Based Virtual Reality in HTML5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 The Story of WebVR 76 The WebVR API 78 Supported Browsers and Devices 78 Querying for VR Devices 79 Setting Up VR Fullscreen Mode 81 Head Tracking 82 Creating a WebVR Application 82 Three.js: A JavaScript 3D Engine 83 A Full Example 83 Tools and Techniques for Creating Web VR 91 WebVR Engines and Development Tools 92 Using Unity3D and Unreal for WebVR Development 93 Open Source Libraries and Frameworks 94 95 WebVR and the Future of Web Browsing 95 Chapter Summary 96 6. VR Everywhere: Google Cardboard for Low-Cost Mobile Virtual Reality. . . . . . . . . . . . . 99 This chapter covers how to build applications for Cardboard VR. There are actually several ways to do this: Cardboard Basics 101 Supported Devices and Operating Systems 101 Headset Manufacturers 101 Cardboard Applications 103 Input Devices for Cardboard 105 Cardboard Stereo Rendering and Head Tracking 107 Developing with the Cardboard SDK for Android 108 Setting Up the Environment 109 Walking Through the Code 110 Developing with the Cardboard SDK for Unity 115 Setting up the SDK 115 Building Treasure Hunt for Unity 116 A Walk-Through of the Unity Code 119 Developing Cardboard Applications Using HTML5 and a Mobile Browser 122 Setting up the WebVR Project 123 The JavaScript Cardboard Code 123 Chapter Summary 126 Table of Contents | v www.it-ebooks.info www.it-ebooks.info CHAPTER 1 Introduction Virtual Reality is a medium with tremendous potential. The ability to be transported to other places, to be fully immersed in experiences, and to feel like you’re really there-- present-- opens up unimagined ways to interact and communicate. Until recently, virtual reality was out of reach for the average consumer due to cost and other factors. However, advances in the technology over the last few years have set the stage for a mass market revolution that could be as influential as the introduction of television, the Internet, or the smartphone. Virtual reality-- VR for short-- comprises a collection of technologies: 3D displays, motion tracking hardware, input devices, software frameworks, and development tools. While consumer-grade VR hardware is young and evolving, a handful of plat‐ forms have emerged as go-to choices, including Oculus Rift, Samsung Gear VR, and Google Cardboard. Each delivers a different level of VR experience, at a different price point, with varying degrees of in-your-hands portability. Software to create and display consumer virtual reality is also coming together rap‐ idly. The Unity3D and Unreal game engines, popular for making desktop and mobile games, have become tools of choice for native VR development. And the web is not far behind: WebGL and 3D JavaScript frameworks like Three.js and Babylon.js are providing a path for creating open source, browser-based virtual reality experiences for desktop and mobile operating systems. It’s an exciting time! With so much energy going into development, and so much con‐ sumer interest, VR just might be the next big wave of computer technology. In this book, we explore the hardware, software, application techniques and interface design challenges encountered by today’s virtual reality creator. Virtual reality is still early. It’s a lot like the wild west, and you are a pioneer. The landscape may be fraught with unknowns, even dangers-- but we push on, driven by the promise of a better life. Let’s take a peek at this new frontier. 7 www.it-ebooks.info Figure 1-1 shows a screen shot of the now-famous Tuscany VR demo, created by the team at Oculus VR to show off their hardware. Put on the Oculus Rift and launch the demo. You are on the grounds of a Tuscan estate, looking at a beautiful villa. Clouds drift lazily across the sky. You hear birds chirping, and the sound of waves lapping gently against a shore. You move through the scene, video game-style using the w, a, s and d keys on your keyboard (known to gamers as the “WASD keys”). If you play a lot of PC games, this is nothing new. But now, turn your head: looking up, down, and behind, you can see the entire estate. You are there, immersed in a virtual world that completely sur‐ rounds you. Walk forward, into the villa, and take a look around. Walk out, up to the edge of the property and see the lake below. For a few moments at least, you forget that you are not actually in this other place. You’re present. This feeling of total immersion, of being somewhere else, experiencing something else entirely, is what we are striving for with virtual reality. And this is where our journey begins. Figure 1-1. Tuscany VR Demo by the Oculus VR Team 8 | Chapter 1: Introduction www.it-ebooks.info What is Virtual Reality? Reality is merely an illusion, albeit a very persistent one. —Albert Einstein Virtual Reality has one goal: to convince you that you are somewhere else. It does this by tricking the human brain-- in particular the visual cortex and parts of the brain that perceive motion. A variety of technologies conspire to create this illusion, includ‐ ing: • Stereoscopic Displays. Also known as 3D displays, or head mounted displays (HMDs). These displays use a combination of multiple images, realistic optical distortion, and special lenses to produce a stereo image that our eyes interpret as having three-dimensional depth. • Motion Tracking Hardware. Gyroscopes, accelerometers and other low-cost components are used in virtual reality hardware to sense when our bodies move and our heads turn, so that the application can update our view into the 3D scene. • Input Devices. Virtual reality is creating the need for new types of input devices beyond the keyboard and mouse, including game controllers and hand- and body-tracking sensors that can recognize motion and gestures. • Desktop and Mobile Platforms. This includes the computer hardware, operat‐ ing systems, software to interface to the devices, frameworks and engines that run applications, and software tools for building them. Without all four of the above components, it is hard to achieve a fully immersive vir‐ tual reality experience. We will dive into the details throughout the book; for now let’s take a quick look at each. Stereoscopic Displays The main ingredient in virtual reality is a persistent 3D visual representation of the experience that conveys a sense of depth. To create this depth, virtual reality hardware systems employ a 3D display, also known as a stereoscopic display or head mounted display. For years, one of the biggest impediments to consumer-grade virtual reality was an affordable stereoscopic display that is light and comfortable enough to be worn for an extended period. This situation changed dramatically when the team from Oculus VR created the Oculus Rift. First introduced in 2012, the Rift was a breakthrough in VR hardware featuring a stereoscopic display and a head-tracking sensor built into a lightweight headset that could be purchased as a development kit for a few hundred What is Virtual Reality? | 9 www.it-ebooks.info