Photo Forensics Photo Forensics Hany Farid The MIT Press Cambridge, Massachusetts London, England (cid:13)c2016 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechani- cal means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. This book was set in LATEX by the author. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Names: Farid, Hany, author. Title: Photo Forensics / Hany Farid. Description: Cambridge, MA : The MIT Press, [2016] | Includes bibliographical references and index. Identifiers: LCCN 2016015126 — ISBN 9780262035347 (hardcover : alk. paper) Subjects: LCSH: Photography, Handworked. | Photography–Retouching. | Photographs–Identification. | Photographs–Inspection. | Forensic sciences. Classification: LCCTR147.F362016|DDC770–dc23LCrecordavailableathttps://lccn.loc.gov/2016015126 10 9 8 7 6 5 4 3 2 1 Contents Preface vii 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 Physics-Based . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Cast Shadows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Attached Shadows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3 Shadow (Pen)umbra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4 Specularities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.5 Lighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.6 Reflections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3 Geometric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.1 Vanishing Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.2 Planar Rectification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.3 Measuring Distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 3.4 3-D Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4 Optical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119 4.1 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 4.2 Lens Distortion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 4.3 Lens Flare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5 Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .153 5.1 Color Filter Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 5.2 Sensor Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.3 Photo Response Non-Uniformity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 vi Contents 6 File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187 6.1 Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 6.2 JPEG Quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 6.3 Thumbnails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 6.4 JPEG Markers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7 Pixel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .217 7.1 Multi-JPEG Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 7.2 Cloning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 7.3 Re-sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 8 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .267 8.1 Golden Eagle Snatches Kid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 8.2 Footballer Momentarily Shrunk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 8.3 The Lee Harvey Oswald Backyard Photo. . . . . . . . . . . . . . . . . . . . . . . . . 275 9 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .285 9.1 Bayesian Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 9.2 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 9.3 Discrete Cosine Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 9.4 Expectation Maximization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 9.5 Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 9.6 Least-Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 9.7 Perspective Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 Bibliography 315 Index 321 Preface In the beginning: I first got interested in the question of photo authenticity in the most unlikely of ways. Nearly twenty years ago, I was waiting in line at a library, when I noticed an enormous bookonacart. Outofidlecuriosity,Ipickedthebookup. ItwastheFederalRulesofEvidence,and thumbing through it I came across Article X. Contents of Writing, Recordings, and Photographs, Rule 1001. In the article is the statement that a photograph is admissible as evidence if it is an original. The rule seemed straightforward, until I read the definition of original: (d) An “original” of a writing or recording means the writing or recording itself or any counterpart intended to have the same effect by the person who executed or issued it. For electronically stored information, “original” means any printout – or other output readablebysight–ifitaccuratelyreflectstheinformation. An“original”ofaphotograph includes the negative or a print from it. Iwasstruckthatthedefinitionof“original”includedsuchavaguestatementas“... orotheroutput readable by sight.” At the time, the digital revolution was still in its early days, digital cameras werestillanovelty, andphoto-editingsoftwarelikeAdobePhotoshopwasrelativelyprimitive. Nev- ertheless, the trajectory of digital advances was clear and it seemed likely that such advances would greatly complicate the introduction of photographic evidence in a court of law. A first idea: This led me to start thinking about how one might authenticate a digital image. I’m a little embarrassed to say that nearly two years passed before I had my first solid idea on how to tackle the question of photo authentication. I was goofing around in Photoshop splicing the head of a friend onto the body of another person. My friend’s head was too small to fit on the body and so I had to increase its size. As I performed this simple operation, I realized that Photoshop must use some type of interpolation to fill in the missing pixels and this interpolation would introduce correlations between neighboring pixels (Section 7.3). A graduate student, Alin Popescu, and I found a way to quantify and detect these correlations, and we submitted this work for publication. A reviewer of our paper made an insightful comment that provided the fodder for our second forensic technique. This second technique exploited the fact that digital cameras do not record all of the pixels needed to generate a complete 3-channel color image. Instead, cameras record a subset of the required pixels and fill in the rest by interpolating them from their neighboring recorded viii Preface pixels. This led us to realize that every digital image has a hidden but discoverable pattern of pixel correlations, and, further, that this pattern will be disturbed if something is added to or removed from the image (Section 5.1). A controversial photo inspired our third forensic technique. The photo was a composite of then presidentialhopefulSenatorJohnKerryandactressandanti-waractivistJaneFondasharingastage at an anti-war rally. This photo was intended to draw attention to Kerry’s involvement in the anti- war movement following his service in Vietnam. The pattern of illumination on the faces of Kerry andFondaseemedinconsistentwiththemsharinganoutdoorstage. Verifyingthisinconsistencyled Kimo Johnson and I to develop a series of forensic techniques for measuring the properties of the surrounding illumination to detect photo composites (Section 2.5). An emerging picture: As we gained traction on the problem of photo authentication, we started tothinkabouttheentireimagingpipeline–thephysicsandgeometryoftheinteractionoflightwith the physical 3-D world (Chapters 2 and 3), the way light passes through a camera lens (Chapter 4), theconversionoflighttopixelvaluesintheelectronicsensor(Chapter5),thepackagingofthesepixel values into a digital image file (Chapter 6), and the pixel-level artifacts introduced by photo-editing software (Chapter 7). Bycarefullymodelingthepathoflightduringimagecreation,wediscoveredphysical,geometric, and statistical regularities in images that are disrupted during the creation of a fake. This allowed us to develop a suite of photo forensic techniques, each based on characterizing part of the image formationprocess, quantifyingtheregularitiesthatarisebecauseofthisprocess, andthendetecting deviations from these expected regularities. This book: This book provides the intuition and background, as well as the mathematical and algorithmic details needed to understand, implement, and utilize a variety of photo forensic tech- niques. I wrote this book for students, researchers, and practitioners interested in digital forensics. I assume a certain amount of basic mathematical background (calculus, linear algebra, and basic probability theory). I also assume some experience with digital image processing. If you are a bit rusty in these fields, you may find the material in Chapter 9 helpful. Throughout this book I use both photographic and computer-generated images to illustrate various forensic techniques. The computer-generated images were created using a physically-based renderer that produces accurate facsimiles of real-world photographic images. Using the renderer allowed me to create images of uncluttered scenes viewed from an ideal vantage point. Throughout this book I use images downloaded from Flickr in which the photographer gave permission for the re-use of their image. I have credited each such image with the photographer’s Flickr user name. Thetechniquesdescribedinthisbookweredevelopedwiththeinvaluablehelpofmanywonderful students and colleagues: Giulia Boato, Mary Bravo, Tiago Carvalho, Valentina Conotter, Olivia Holmes, Kimo Johnson, Eric Kee, Siwei Lyu, James O’Brien, Senthil Periaswamy, Alin Popescu, Weihong Wang, Emily Whiting, and Jeffrey Woodward. This book was masterfully edited by Mary Bravo who imbued every sentence with her special brand of clarity and conciseness. Preface ix Wordsofadvice: Lastly,Ihaveafewwordsofadviceforanyonewhousestheseforensictechniques in a law enforcement or legal setting: work carefully and slowly. Double-, triple-, quadruple-check yourwork. Donotallowpreconceivednotionstoaffectwhatshouldbeapurelydata-drivenanalysis. Understandandrespecttheassumptionsrequiredbyananalysis. Whenyouareunsure,refrainfrom drawing conclusions.