Light Field Appearance Manifolds by Chris Mario Christoudias Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering and Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY August 2004 0 Massachusetts Institute of Technology 2004. All rights reserved. Author............ ............. .................... Department of Electrical Engineering and Computer Science August 23, 2004 i /I Certified by.... Trevor Darrell Associate Professor of Electrical Engineering and Computer Science Thesis Supervisor Accepted by... Arthur C. Smith Chairman, Department Committee on Graduate Students MASSACHUSETTS INSTITUTE OF TECHNOLOGY OCT 2 8 2004 LIBRARIES BARKER 2 Light Field Appearance Manifolds by Chris Mario Christoudias Submitted to the Department of Electrical Engineering and Computer Science on August 23, 2004, in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering and Computer Science Abstract Statistical shape and texture appearance models are powerful image representations, but previously had been restricted to 2D or 3D shapes with smooth surfaces and lambertian reflectance. In this thesis we present a novel 4D appearance model using image-based rendering techniques, which can represent complex lighting conditions, structures, and surfaces. We construct a light field manifold capturing the multi-view appearance of an object class and extend previous direct search algorithms to match new light fields or 2D images of an object to a point on this manifold. When matching to a 2D image the reconstructed light field can be used to render unseen views of the object. Our technique differs from previous view-based active appearance models in that model coefficients between views are explicitly linked, and that we do not model any pose variation within the deformable model at a single view. It overcomes the limitations of polygonal based appearance nodels and uses light fields that are acquired in real-time. Thesis Supervisor: Trevor Darrell Title: Associate Professor of Electrical Engineering and Computer Science 3 4 Acknowledgments Firstly, I would like to thank my advisor, Trevor Darrell, for his endless support and guidance. Without him this thesis would not have been possible. In the short time I have been at MIT I have learned a lot from him. He is a very creative and gifted scientist and a caring person. I am very fortunate and grateful to be his student. I am grateful to the members of the Vision Interface Group for their support. In particular, I would like to thank Greg Shakhnarovich, Ali Rahimi, Kevin Wilson and David Demirdjian for their advice and encouragement. I would like to thank Louis-Philippe Morency for taking ine under his wing when I first came to MIT. I also would like to acknowledge him for all of his help throughout the work of this thesis'. He has always been there for me as a fellow co-worker and more importantly as a friend. I also would like to thank my good friend Neal Checka for all of his support. I would like to thank my undergraduate advisor, Peter Meer, for introducing me to research and the field of computer vision. Without him I would not be where I am today. I gratefully acknowledge the Kanellakis fellowship for their generous funding dur- ing my first year at MIT. I also want to thank the participants of our data collection. Finally, I would like to thank my parents John and Aphrodite Christoudias. sister Tina Christoudias and my family for all of their love and support. They have given me so much to be grateful for and I love them deeply. Part of the work presented in this thesis is jointly published in the 8th European Conference on Computer Vision. See [7] for details. 5 6 Contents 1 Introduction 21 1.1 Related Work ........ ............................... 25 1.2 O utline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2 Image Warping and Morphing 29 2.1 Image Warping ........ .............................. 30 2.1.1 inage Interpolation . . . . . . . . . . . . . . . . . . . . . . . . 31 2.1.2 Reverse vs. Foward Warping .................. . 34 2.2 Im age M orphing ......... .............. ....... 35 2.3 Beier and Neely . . . . . .. .. . . . . . . . . . . . . . . . . . . . . . 37 2.4 Piecewise Image Warping ........................ . 40 3 Statistical Shape and Texture Appearance Models 43 3.1 Shape and Texture Appearance Models . . . . . . . . . . . . . . . . . 47 3.2 Multidimensional Morphable Models . . . . . . . . . . . . . . . . . . 49 3.3 Active Appearance Models . . . . . . . . . . . . . . . . . . . . . . . . 53 4 Image-Based Rendering: The Light Field 63 4.1 The Plenoptic Function . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.2 Light Field Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3 The Unstructured Luinigraph . . . . . . . . . . . . . . . . . . . . . . 69 5 Light Field Morphing 77 5.1 Light Field Morphing Algorithm . . . . . . . . . . . . . . . . . . . . . 79 7 5.1.1 Warping Aligned Objects . . . . . . . . . . . . . . . . . . . . . 82 5.2 Metamorphosis Between Multiple Objects . . . . . . . . . . . . . . . 83 5.3 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.4 Exam ples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6 4D Shape and Texture Appearance Manifolds 95 6.1 Light Field Shape and Texture . . . . . . . . . . . . . . . . . . . . . . 97 6.1.1 Automatic Shape Acquisition: Optical Flow . . . . . . . . . . 99 6.2 Light Field Appearance Manifolds . . . . . . . . . . . . . . . . . . . . 100 6.3 Optical Flow Based Model Matching . . . . . . . . . . . . . . . . . . 102 6.3.1 Matching to a Light Field . . . . . . . . . . . . . . . . . . . . 103 6.3.2 Matching to an Image . . . . . . . . . . . . . . . . . . . . . . 103 6.4 Model Matching via Direct Search . . . . . . . . . . . . . . . . . . . . 104 6.4.1 Matching to a Light Field . . . . . . . . . . . . . . . . . . . . 104 6.4.2 Matching to an Image . . . . . . . . . . . . . . . . . . . . . . 105 6.4.3 Jacobian Computation for Matching an Image . . . . . . . . . 109 6.5 Automatic Pose Estimation Algorithm . . . . . . . . . . . . . . . . . 110 7 Experiments and Results 113 7.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . .. . . 114 7.2 Comparison to a View-Based AAM . . . . . . . . . . . . . . . . . . . 115 7.3 Matching to a Light Field . . . . . . . . . . . . . . . . . . . . . . . . 118 7.3.1 Optical Flow Based Model . . . . . . . . . . . . . . . . . . . . 118 7.3.2 Feature-point Based Model . . . . . . . . . . . . . . . . . . . . 119 7.4 Matching to an Image . . . . . . . . . . . . . . . . . . . . . . . . .. 119 7.4.1 Automatic Pose Estimation . . . . . . . . . . . . . . . . . . . 120 7.4.2 Optical Flow Based Model . . . . . . . . . . . . . . . . . . . . 121 7.4.3 Feature-point Based Model . . . . . . . . . . . . . . . . . . . . 122 7.5 Algorithm Implementation and Performance . . . . . . . . . . . . . . 123 8 8 Discussion and Future Work 129 8.1 Contributions ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 8.2 Applications . ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 8.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 9 10