Mastering Malware Analysis Second Edition A malware analyst's practical guide to combating malicious software, APT, cybercrime, and IoT attacks Alexey Kleymenov Amr Thabet BIRMINGHAM—MUMBAI Mastering Malware Analysis Second Edition Copyright © 2022 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Group Product Manager: Mohd Riyan Khan Publishing Product Manager: Mohd Riyan Khan Content Development Editor: Adrija Mitra Technical Editor: Nithik Cheruvakodan Copy Editor: Safis Editing Project Coordinator: Ashwin Kharwa Proofreader: Safis Editing Indexer: Manju Arasan Production Designer: Ponraj Dhandapani Marketing Coordinator: Ankita Bhonsle First published: June 2019 Second edition: September 2022 Production reference: 1010922 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. 978-1-80324-024-4 www.packt.com I dedicate this book to my family and friends – your continuous support means so much to me. – Alexey Kleymenov To my family. – Amr Thabet Co n t r i b u t o r s About the authors Alexey Kleymenov started working in the information security industry in his second year at university and now has more than 14 years of practical experience at several international cybersecurity companies. He is a malware analyst and software developer who is passionate about reverse engineering, automation, and research. Alexey has taken part in numerous investigations analyzing all types of malicious samples, has developed various systems to perform threat intelligence activities in the IT, OT, and IoT sectors, and has authored several patents. Alexey is a member of the (ISC)² organization and holds the CISSP certification. Finally, he is a founder of the RE and More project, teaching people all over the world how to perform malware analysis in the most efficient way. I would like to deeply thank all my family, especially my beloved mom, Olga, and wife, Anastasia, for all your love and support. Big thanks to Amr, who turned this project into enjoyable cooperative work. I’m much obliged to the Packt team for addressing all of our inquiries, and to the readers and reviewers for their invaluable feedback. Finally, thanks to everyone who contributed to my personal development, served as an inspiration, or was next to me when I needed them. Amr Thabet is a malware researcher and an incident handler with over 10 years of experience. He has worked in several Fortune 500 companies, including Symantec and Tenable. Currently, he is the founder of MalTrak, providing real-world in-depth training in malware analysis, incident response, threat hunting, and red teaming to help the next generation of cybersecurity enthusiasts to build their careers in cybersecurity. Amr is also a speaker and trainer at some of the top security conferences all around the world, including Blackhat, DEFCON, Hack In Paris, and VB Conference. He was also featured in Christian Science Monitor for his work on Stuxnet. I’d like to thank my parents for helping me and believing in me throughout this journey. And a big thanks to my book partner, friend, and former colleague, Alexey. Without his expertise, hard work, and dedication, this book wouldn’t have come to light. We put our experience, expertise, and hearts into this work and we really hope it changes your life and your career as this knowledge once changed ours. About the reviewer Ahmed Neil is a well-known thought leader in the cybersecurity domain whose work focuses on information security, threat hunting, threat intelligence, malware analysis, and digital forensics. He also has a passion for academic research in the field of cybersecurity. He holds an MSc in computer forensics and is currently working at IBM as a cybersecurity engineer (operations). Table of Contents Preface xvii Part 1: Fundamental Theory 1 Cybercrime, APT Attacks, and Research Strategies 3 Why malware analysis? 4 APT attack 13 Malware analysis in collecting threat intelligence 4 Zero-day attack 13 Malware analysis in incident response 5 Fileless malware 14 Malware analysis in threat hunting 5 Choosing your analysis strategy 14 Malware analysis in creating detections 6 Understand your audience 14 Exploring types of malware 6 Answer your audience’s questions 15 A short history of malware development 6 Define your goals 16 Malware categories 7 Avoid unnecessary technical details 16 Naming conventions 10 Example structures 16 Typical analysis workflow 18 The MITRE ATT&CK framework explained 10 Setting up the environment 18 Basic terminology 10 Choosing the virtualization software 19 Enterprise Matrix 11 Safety features 19 APT and zero-day attacks and fileless Summary 21 malware 13 viii Table of Contents 2 A Crash Course in Assembly and Programming Basics 23 Basics of informatics 24 Basics 53 Numeral systems 24 The instruction set 54 Basic data units and data types 25 Diving deep into PowerPC 56 Bitwise operations 26 Basics 56 Architectures and their assembly 30 The instruction set 58 Registers 30 Covering the SuperH assembly 59 Memory 31 Basics 60 Instructions (CISC and RISC) 32 The instruction set 60 Becoming familiar with x86 (IA-32 Working with SPARC 62 and x64) 34 Basics 62 Registers 34 The instruction set 63 The instruction structure 36 Moving from assembly to high-level The instruction set 38 programming languages 64 Arguments, local variables, and calling conventions (in x86 and x64) 42 Arithmetic statements 64 If conditions 66 Exploring ARM assembly 45 While loop conditions 68 Basics 47 Instruction sets 49 Summary 68 Basics of MIPS 52 Part 2: Diving Deep into Windows Malware 3 Basic Static and Dynamic Analysis for x86/x64 71 Working with the PE header structure 72 Static linking 81 Why PE? 72 Dynamic linking 82 Exploring PE’s structure 73 Dynamic link libraries 82 PE+ (x64 PE) 78 Application programming interface (API) 83 PE header analysis tools 79 Using PE header information for Static and dynamic linking 80 static analysis 84 Table of Contents ix How to use the PE header for incident handling 84 Setting labels and comments 104 How to use a PE header for threat hunting 85 Differences between OllyDbg and x64dbg 104 PE loading and process creation 87 Debugging malicious services 105 Basic terminology 87 What is a service? 105 Process creation step by step 91 Attaching to services 107 PE file loading step by step 92 Essentials of behavioral analysis 109 WOW64 processes 93 File operations 109 Basics of dynamic analysis using Registry operations 110 OllyDbg and x64dbg 94 Process operations 111 Debugging tools 95 WinAPIs 111 How to analyze a sample with OllyDbg 97 Network activity 112 Types of breakpoints 100 Sandboxes 112 Modifying the program’s execution 102 Summary 114 List strings, APIs, and cross-references 104 4 Unpacking, Decryption, and Deobfuscation 115 Exploring packers 116 Technique 3 – monitoring memory allocated spaces for unpacked code 130 Exploring packing and encrypting tools 116 Technique 4 – in-place unpacking 132 Identifying a packed sample 117 Technique 5 – searching for and transferring Technique 1 – using static signatures 118 control to OEP 133 Technique 2 – evaluating PE section names 118 Technique 6 – stack restoration-based 133 Technique 3 – using stub execution signs 119 Dumping the unpacked sample and Technique 4 – detecting a small import table 119 fixing the import table 134 Automatically unpacking packed Dumping the process 134 samples 120 Fixing the import table 135 Technique 1 – the official unpacking process 120 Identifying simple encryption Technique 2 – using OllyScript with OllyDbg 121 algorithms and functions 137 Technique 3 – using generic unpackers 121 Types of encryption algorithms 137 Technique 4 – emulation 121 Basic encryption algorithms 139 Technique 5 – memory dumps 122 Identifying encryption functions in Manual unpacking techniques 123 disassembly 140 Technique 1 – memory breakpoint on String search detection techniques for simple execution 123 algorithms 141 Technique 2 – call stack backtracing 126 Identifying the RC4 encryption algorithm 143