A BEHAVIOURAL STUDY IN RUN- TIME ANALYSIS ENVIRONMENTS AND DRIVE-BY- DOWNLOAD ATTACKS MOHAMMAD ALLY REHAZ PUTTAROO A thesis submitted in partial fulfilment of the requirements of The University of West London for the degree of Doctor of Philosophy in Cyber Security and Operating Systems July 2017 Acknowledgements The process of undertaking this PhD has provided a life changing process of learning and development for me. It would not have been possible to undertake the PhD without the guidance, support and experience of many people. Firstly, I would like to thank my supervisor and mentor, Professor Peter Komisarczuk. Peter was exactly what every student who is crazy enough to start a PhD needed. With his invaluable expertise within this subject, there was never a time where Peter did not thoroughly support and provide me with direction. This direction was greatly appreciated during the difficult periods during my PhD. I am truly lucky to have had met and been able to work under the guidance of such an exemplary gentleman and Professor. To put into perspective, he kept me as his PhD student years after having left the original institution and never failed to uphold but the highest standards of supervisory and support. Thank you, Peter, there are no words to describe your kindness and dedication for what has been one hell of a journey. My parents have been a constant source of strength, motivation and support during my studies: from mild encouragement to listening to me worry about my progress throughout the journey. They were there when experiments failed, and environments broke, providing me with a constant stream of positivity and perseverance. Without them, it would have proved impossible to successfully complete this thesis. Thank you for being supportive and providing me the means to take part of this (insane) journey. I would like to thank my lovely sister Madina for all the joys she brings to me along with Mazmoon and Munira. 1 My beloved wife Imrana deserves an award for the endless emotional support and for the countless hours proof-reading my thesis and learning about honeypots just to accommodate my endless monologues, debates, doubts and reviews. I would also like to extend my sincerest gratitude towards the supervisory team: Dr. Renato Amorim who has one of the sharpest eyes when it comes to reviewing transcripts whilst also being an amazing co-author who has read through this thesis countlessly and helped tremendously in the betterment. Professor Thomas Roth- Berghofer has also been a constant source of feedback and review. These were key individuals in the refinement of the studies. The feedback from Anastasia, Christian and Junaid at the internal school board was very helpful and finding thesis reviewers that are willing to go through an entire thesis is a rare commodity. I would like to thank them for the time and effort put into the betterment of the work. I extend my thanks towards the University of West London for the scholarship and sponsorship that it provided over the years and the team at the graduate school. This is extended to the IT department for accommodating my needs and experiments. It must have been terrifying for an IT department to be told that malicious website analysis needed to take place on a network and without the accommodation required, this research would not have been possible. The Honeynet project has been an indirect contributor for this work by inspiring motivation for research focus in this field. In particular, I would like to thank the Polish CERT (CERT Polska) for their various security projects, which led to this research. 2 For my parents, who have firmly believed in education from the very first few years of my life and everyone who have supported me get through the struggles PhD journey and grow as a person. 3 Abstract In the information age, the growth in availability of both technology and exploit kits have continuously contributed in a large volume of websites being compromised or set up with malicious intent. The issue of drive-by-download attacks formulate a high percentage (77%) of the known attacks against client systems. These attacks originate from malicious web-servers or compromised web-servers and attack client systems by pushing malware upon interaction. Within the detection and intelligence gathering area of research, high-interaction honeypot approaches have been a long- standing and well-established technology. These are however not without challenges: analysing the entirety of the world wide web using these approaches is unviable due to time and resource intensiveness. Furthermore, the volume of data that is generated as a result of a run-time analysis of the interaction between website and an analysis environment is huge, varied and not well understood. The volume of malicious servers in addition to the large datasets created as a result of run-time analysis are contributing factors in the difficulty of analysing and verifying actual malicious behaviour. The work in this thesis attempts to overcome the difficulties in the analysis process of log files to optimise malicious and anomaly behaviour detection. The main contribution of this work is focused on reducing the volume of data generated from run-time analysis to reduce the impact of noise within behavioural log file datasets. This thesis proposes an alternate approach that uses an expert lead approach to filtering benign behaviour from potentially malicious and unknown behaviour. Expert lead filtering is designed in a risk-averse method that takes into account known benign and expected behaviours before filtering the log file. Moreover, the approach relies upon behavioural investigation as well as potential for 4 system compromisation before filtering out behaviour within dynamic analysis log files. Consequently, this results in a significantly lower volume of data that can be analysed in greater detail. The proposed filtering approach has been implemented and tested in real-world context using a prudent experimental framework. An average of 96.96% reduction in log file size has been achieved which is transferable to behaviour analysis environments. The other contributions of this work include the understanding of observable operating system interactions. Within the study of behaviour analysis environments, it was concluded that run-time analysis environments are sensitive to application and operating system versions. Understanding key changes in operating systems behaviours within Windows is an unexplored area of research yet Windows is currently one of the most popular client operating system. As part of understanding system behaviours for the creation of behavioural filters, this study undertakes a number of experiments to identify the key behaviour differences between operating systems. The results show that there are significant changes in core processes and interactions which can be taken into account in the development of filters for updated systems. Finally, from the analysis of 110,000 potentially malicious websites, typical attacks are explored. These attacks actively exploited the honeypot and offer knowledge on a section of the active web-based attacks faced in the world wide web. Trends and attack vectors are identified and evaluated. 5 Contents Acknowledgements .................................................................................................... 1 Abstract ...................................................................................................................... 4 Chapter 1 – Introduction ........................................................................................... 21 1.1 Motivation ....................................................................................................... 21 1.2 Research goals ............................................................................................... 23 1.3 Contributions................................................................................................... 24 1.3.1 Methodology in developing the expert driven behavioural filters .............. 24 1.3.2 Identification of differences in behavioural manifestations in different versions operating systems. .............................................................................. 25 1.3.3 Observed active exploits and malicious behaviours attacking honeypots 26 1.4 Thesis organisation ......................................................................................... 26 1.5 Publications .................................................................................................... 29 Chapter 2 – Literature review ................................................................................... 31 2.1 Introduction to drive-by-download and research area ..................................... 31 2.1.1 Drive-by-download attacks ....................................................................... 31 2.1.2 Detection, intelligence gathering parties and technologies ...................... 37 2.1.3 Collecting drive-by-download samples ..................................................... 40 2.2 Methods of analysing drive-by-downloads ...................................................... 41 2.2.1 Low-interaction ......................................................................................... 42 2.2.2 High-interaction ........................................................................................ 43 2.2.3 Hybrid-interaction ..................................................................................... 45 6 2.2.4 Chosen approach and rationale ............................................................... 47 2.3 Malware .......................................................................................................... 48 2.3.1 Malware analysis ...................................................................................... 48 2.3.2 Research around malware analysis ......................................................... 51 2.3.3 Malware packers ...................................................................................... 53 2.3.4 Differentiating drive-by-download and malware analysis ........................ 55 2.5 Instruments for drive-by-download collection and analysis ............................. 56 2.5.1 Capture-HPC ............................................................................................ 56 2.5.2 Cuckoo sandbox ...................................................................................... 58 2.6 Related work ................................................................................................... 59 2.6.1 Malicious Domain name servers .............................................................. 59 2.6.2 Systems monitoring state change ............................................................ 59 2.6.3 JavaScript research ................................................................................. 62 2.6.4 Current trends in client honeypot research .............................................. 63 2.6.5 Applicability of machine learning .............................................................. 64 2.6.6 Clustering approaches ............................................................................. 65 2.6.7 Classification approaches ........................................................................ 66 2.6.8 Closely related work ................................................................................. 68 2.7 Gap identification and research questions ...................................................... 70 2.7.1 Behaviour filtering in behaviour analysis environments ............................ 71 2.7.2 Expert driven filtering and exclusion lists ................................................. 72 7 2.7.3 Windows 7 applicability and operating system behaviours ...................... 76 2.7.4 Research scope and summary ................................................................. 77 Chapter 3 – Methodology and experiment design .................................................... 82 3.1 Safe, prudent and valid malware experimentation .......................................... 86 3.1.1 Correctness in malware experimentation ................................................. 88 3.1.2 Realism in malware experimentation ....................................................... 90 3.1.3 Transparency in malware experimentation............................................... 92 3.1.4 Safety in malware experimentation .......................................................... 93 3.1.5 Conclusion ............................................................................................... 95 3.2 Drive-by-download resource gathering ........................................................... 97 3.2.1 Creation of the data corpus ...................................................................... 99 3.2.2 Justification for the creation of the malware data corpus ....................... 101 3.2.3 Sample size evaluation .......................................................................... 102 3.3 Experimental setup ....................................................................................... 104 3.3.1 Experimental methodology justification .................................................. 105 3.3.2 Capture-BAT .......................................................................................... 106 3.3.3 Duration of malicious webpage analysis ................................................ 107 3.4 Testing the validity of experimental setup ..................................................... 111 3.5 Alternate resource gathering evaluations ..................................................... 111 3.6 Roundup ....................................................................................................... 111 Chapter 4 – Expert driven behaviour filter development ......................................... 115 8 4.1 Background .................................................................................................. 116 4.2 Research requirements ................................................................................ 117 4.2.1 Limitations of Capture-BAT within the Windows 7 environment ............. 118 4.2.2 Identification of behavioural vectors within Windows 7 .......................... 119 4.2.3 Exclusion list creation rationale .............................................................. 120 4.2.4 Aims and goals of exclusion lists ............................................................ 123 4.3 Methodology used in developing expert driven behavioural filters ................ 124 4.3.1 Decision making process behind classifications of behaviour ................ 127 4.3.2 Processing behavioural log files to synthesise expert driven behavioural filters ............................................................................................................... 128 4.4 Inside an exclusion list .................................................................................. 130 4.4.1 Typical Windows 7 native system calls .................................................. 133 4.4.2 System behaviours: The Windows 7 file system behaviours .................. 136 4.4.3 System behaviours: The Windows 7 processes’ behaviours.................. 142 4.4.4 System behaviours: The Windows 7 registry behaviours ....................... 148 4.4.5 Differences between Windows 7 with Windows XP system calls ........... 157 4.5 The development cycle of exclusion lists ...................................................... 161 4.5.1 Environment set up ................................................................................ 162 4.5.3 Created exclusion lists within a Windows 7 application profile ............... 166 4.5.4 Limitations of behavioural filters ............................................................. 167 4.7 Contributions to knowledge and key findings ............................................ 168 Chapter 5 – Behaviour analysis environment experiments .................................... 171 9