Justin Ferguson Dan Kaminsky Jason Larsen Luis Miras Walter Pearce This page intentionally left blank Elsevier, Inc., the author(s), and any person or fi rm involved in the writing, editing, or production (collectively “Makers”) of this book (“the Work”) do not guarantee or warrant the results to be obtained from the Work. There is no guarantee of any kind, expressed or implied, regarding the Work or its contents. The Work is sold AS IS and WITHOUT WARRANTY. You may have other legal rights, which vary from state to state. In no event will Makers be liable to you for damages, including any loss of profi ts, lost savings, or other incidental or consequential damages arising out from the Work or its contents. Because some states do not allow the exclusion or limitation of liability for consequential or incidental damages, the above limitation may not apply to you. You should always use reasonable care, including backup and other appropriate precautions, when working with computers, networks, data, and fi les. Syngress Media®, Syngress®, “Career Advancement Through Skill Enhancement®,” “Ask the Author UPDATE®,” and “Hack Proofi ng®,” are registered trademarks of Elsevier, Inc. “Syngress: The Defi nition of a Serious Security Library”™, “Mission Critical™,” and “The Only Way to Stop a Hacker is to Think Like One™” are trademarks of Elsevier, Inc. Brands and product names mentioned in this book are trademarks or service marks of their respective companies. KEY SERIAL NUMBER 001 HJIRTCV764 002 PO9873D5FG 003 829KM8NJH2 004 BAL923457U 005 CVPLQ6WQ23 006 VBP965T5T5 007 HJJJ863WD3E 008 2987GVTWMK 009 629MP5SDJT 010 IMWQ295T6T PUBLISHED BY Syngress Publishing, Inc. Elsevier, Inc. 30 Corporate Drive Burlington, MA 01803 Reverse Engineering Code with IDA Pro Copyright © 2008 by Elsevier, Inc. All rights reserved. Printed in the United States of America. Except as permitted under the Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher, with the exception that the program listings may be entered, stored, and executed in a computer system, but they may not be reproduced for publication. Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 ISBN 13: 978-1-59749-237-9 Publisher: Andrew Williams Technical Editor: Dan Kaminsky Project Manager: Anne McGee Page Layout and Art: SPi For information on rights, translations, and bulk sales, contact Matt Pedersen, Commercial Sales Director and Rights, at Syngress Publishing; email [email protected]. This page intentionally left blank About IOActive Established in 1998, IOActive has successfully positioned itself as an industry leader in the Northwest’s computer security community, where it specializes in infrastructure assessment services, application security services, managed services, incident response services, and education services. The company has helped various Fortune 500 organizations with services ranging from enterprise risk management to independent technical validations of security hardware and a wide range of applications. It has also been commissioned to work on IT disaster recovery and business continuity planning for major insurance companies, state organizations and energy companies. IOActive’s consultants are members and active contributors to local and nationally recognized computer security organizations such as SANS, Agora, CRIME, ISSA, CTIN, WSA, HoneyNet Research Alliance, OWASP, and the University of Washington Information Assurance School. v Technical Editor and Contributing Author Dan Kaminsky is the Director of Penetration Testing for IOActive. Previously of Cisco and Avaya, Dan has been operating professionally in the security space since 1999. He is best known for his “Black Ops” series of talks at the well respected Black Hat Briefi ngs conferences. He is also the only speaker who has attended and spoken at every single “Blue Hat” Microsoft internal training event. Dan focuses on design level fault analysis, particularly against massive-scale network applications. Dan regularly collects detailed data on the health of the worldwide Internet, and recently used this data to detect the worldwide proliferation of a major rootkit. Dan is one of the few individuals in the world to combine both technical expertise with executive level consulting skills and prowess. vi Contributing Authors Justin Ferguson is a security consultant and researcher at IOActive. He is involved with helping Fortune 500 companies understand and mitigate risk introduced in complex software computing environments via the Application Security Practice at IOActive. Justin has over six years experience working as a reverse engineer, source code auditor, malware analyst, and enterprise security analyst for industries ranging from fi nancial institutions to the federal government. I would like to thank my father, Bruce Dennis Ferguson, who was a great man; I regret never having apologized to you nor allowing you to see the man your son has become. I would like to thank all of the blue collar union workers from Boston who worked themselves to the bone to make sure their children had a better life. No mention of these men would be complete if I neglected the women who stood by their sides and saw them through each day; you all truly are beautiful. I’d like to take a moment to remember everyone from the South End and Brockton/South Shore who didn’t make it and for those still struggling; continue on with the belief that unearned suffering is redemptive. Saint Jude, pray for us all. Jason Larsen has penetrated and owned some of the most integral systems on the planet. His career began when he was at Idaho State University and detected Internet-wide stealth scanning. He was awarded two scholarships in order to support his research into and creation of detection systems, including authorship of one of the fi rst Intrusion Prevention Systems that actually blocked penetration. Mr. Larsen has been unable to publish most of his work due to national security concerns. His work for the Department of Energy through the Idaho National Laboratories allowed him to develop even more elegant solutions to the security problems of major SCADA and PCS systems. His security work has benefi ted hundreds of clients among several industries, including US and foreign. I’d like to dedicate this book to the infi nite patience and understanding of The Girlfriend. Thank you for the quiet nods when listening to the latest problem and the occasional push out the door to get some sunlight. Every geek should be required to have a permanent tattooed companion. vii Luis Miras is an independent security researcher. He has worked for both security product vendors and leading consulting fi rms. His interests include vulnerability research, binary analysis, and hardware/software reversal. In the past, he has worked in digital design and embedded programming. He has presented at CanSecWest, Black Hat, CCC Congress, XCon, REcon, DefCon, and other conferences worldwide. When he isn’t heads down in IDA or a circuit board, you will likely fi nd him boarding down some sweet powder. I dedicate this book to my parents and brothers. I would like to thank Don Omar, Sister Nancy, and Nas for providing the coding soundtrack. I would like to send greetz to all my friends and let them know that, yes, I’m alive and no longer MIA. Thanks to Sebastian “topo” Muniz for the IDA discussions and bouncing ideas. Walter Pearce provides application security and penetration testing services for IOActive, and is a regular contributor to the ongoing research and development of advanced tools that automate IT security testing and p rotective functions. His career began at 12, and his fi rst professional role was as the operator of a data center cluster for an online retailer, which led to Senior Programming Engineer positions at fi nancial service fi rms and institutions. During his time in the fi nance industry, Walter specialized in the conception of internal threats and designed mitigations to reduce incidence of such events. Mr. Pearce is often requested by clients to provide expert application security services involving a variety of platforms and languages. To Becca, Mom, David. Love ya all. viii Contents Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 An Overview of Code Debuggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chapter 2 Assembly and Reverse Engineering Basics . . . . . . . . . . . . . . . . . .7 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Assembly and the IA-32 Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 The Stack, the Heap and Other Sections of a Binary Executable . . . . . . . . . . . . 19 IA-32 Instruction Set Refresher and Reference . . . . . . . . . . . . . . . . . . . . . . . . 24 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Chapter 3 Portable Executable and Executable and Linking Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Portable Executable Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Executable and Linking Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Chapter 4 Walkthroughs One and Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Following Execution Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Reversing What the Binary Does . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 The Processing Subroutine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Chapter 5 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Debugging Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Breakpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Hardware Breakpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Software Breakpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Using Breakpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Single Stepping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Watches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Debugging in IDA Pro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 ix