Universita(cid:18) degli Studi di Napoli \Federico II" Scuola Politecnica e delle Scienze di Base Dottorato di Ricerca in Ingegneria Informatica ed Automatica Ciclo XXVIII May 2016 Tesi di Dottorato Automated GUI Testing Techniques for Android Applications Tutori: Candidato: Amatucci Nicola Prof. Anna Rita Fasolino Prof. Por(cid:12)rio Tramontana Abstract Mobile devices are integral parts of our daily lives; a little computer in our pocket has became a faithful assistant both for work than for amusement. The availability of mobile applications (commonly referred as apps) has made more and more useful bringing these devices with us everyday. The number of such applications in these years has faced a tremendous growth due to the market attractiveness [106]; according to Forbes1, by 2017 more than 270 billion mobile applications will be downloaded worldwide. The quality of a mobile application is a major concern for developers, users and application stores [32]. According to a survey conducted by SmartBear2 from October to December 2013 nearly 50% of consumers will delete a mobile app if they encounter a bug. So, testing mobile applications to prevent the occurrence of software exceptions in production can be considered one of the key factor in(cid:13)uencing its quality together with the market response. As today, in literature many techniques have been presented aiming at testing mobile applications. In particular, many of them have been presented in the context of GUI Testing. The research activity described in this thesis is focused on proposing novel techniques and tools in the (cid:12)eld of Automated GUI Testing for Mobile Applica- tions. In particular, the work is targeted to the Android Operating System, that currently is the dominating operating system in the mobile devices market [54], although the results can be generalized to other mobile platforms. 1http://blogs-images.forbes.com/niallmccarthy/(cid:12)les/2014/10/Giant-social-apps Forbes.jpg 2https://smartbear.com/news/news-releases/the-state-of-mobile-testing-2014/ Acknowledgements Non sono bravo a scrivere i ringraziamenti, sono piu(cid:18) propenso a farli di persona, ma ci provo cercando di non lasciar fuori nessuno, perch(cid:19)e, diciamo la verit(cid:18)a, se non fosse stato per le tante persone che mi hanno accompagnato in questo viaggio, probabilmente non sarei qui adesso a scrivere queste parole. Ringrazio i miei tutor, la professoressa Anna Rita Fasolino ed il professor Por- (cid:12)rio Tramontana, che, insieme all’ingegner Domenico Amal(cid:12)tano, qualche anno fa mi hanno accordato un’enorme (cid:12)ducia concedendomi il privilegio di far parte di questo gruppo di ricerca. Senza di loro non sarei qui e non sarei potuto crescere cos(cid:18)(cid:16) tanto sia professionalmente che personalmente. Sono stati un punto di rifer- imento costante in questo percorso, mi hanno guidato ed aiutato e spero di non aver deluso troppo le loro aspettative e di aver dato loro qualche soddisfazione. Sono davvero delle persone eccezionali; se non fosse stato per loro non sarei qui e li ringrazio dal profondo del mio cuore. Ringrazio tutti i ragazzi che in questi anni hanno frequentato il Laborato- rio 4.04 e che lo frequentano ancora. Senza di loro, questo periodo non sarebbe stato cos(cid:18)(cid:16) bello, allegro e produttivo. Ringrazio tutti quelli che ho conosciuto qui all’Universit(cid:18)a di Napoli, professori, ricercatori, dottorandi, collaboratori, person- ale amministrativo e non, tesisti, studenti e tutte le persone che in questi anni ho incontrato. Alcuni di loro li ringrazio particolarmente e, forse, non li ringrazier(cid:18)o mai abbastanza; questo pensiero(cid:18)e dedicato in particolar modo a loro e soprattutto a quelli che posso chiamare amici. Voglio ringraziare i miei genitori e mia nonna che mi hanno incoraggiato, sup- portato, sostenuto e sopportato giorno per giorno e continuano a sostenermi. Gra- zie mille davvero per tutto quello che avete fatto in questi anni, da quando sono nato; senza di voi nulla sarebbe stato possibile. Ringrazio i miei suoceri, i miei cognati e le mie cognate che mi hanno sempre aiutato e sostenuto in ogni momento bello o brutto e ora fanno parte della mia famiglia. Ringrazio tutti gli amici e parenti che, anche se forse in questi ultimi anni ho trascurato un po’, sono sempre nei miei pensieri e nel mio cuore. In(cid:12)ne, con tutto il mio cuore, voglio ringraziare mia moglie Liliana, che ho avuto la fortuna di sposare quasi un anno fa, e che in questi anni mi ha davvero donatotantoamoreecheamocontuttomestesso; ancheneimomentididifficolt(cid:18)a(cid:18)e stata sempre accanto a me e lo(cid:18)e ancora oggi, per fortuna. Aspettiamo un bambino o una bambina (ancora non lo sappiamo) e questo oltre a darmi immensa gioia, mi d(cid:18)a forza e coraggio di affrontare tutto quello che verra(cid:18). Grazie, in(cid:12)nitamente. Dedico a voi questa tesi, come dedico a voi la mia vita, ogni giorno. Spero di non aver dimenticato nessuno in questi miei, seppur sintetici, ringrazi- amenti3. Grazie a tutti, Nicola 3Ci tengo a precisare che tutte le persone citate in queste pagina hanno svolto un ruolo fondamentale nella stesura della tesi, ma che ogni errore o imprecisione (cid:18)e imputabile soltanto a me A Liliana, la donna che amo, al frutto del nostro amore. Contents 1 Introduction 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Thesis Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Background & Related Work 6 2.1 The Android Operating System . . . . . . . . . . . . . . . . . . . . 6 2.1.1 The Android Framework Architecture . . . . . . . . . . . . . 7 2.1.2 Core Concepts of the Android Platform . . . . . . . . . . . . 9 2.1.2.1 Inter Process Communication (IPC) . . . . . . . . 10 2.1.2.2 Android Activities . . . . . . . . . . . . . . . . . . 11 2.1.2.3 Android User Interface Elements . . . . . . . . . . 14 2.1.2.4 Android Fragments . . . . . . . . . . . . . . . . . . 14 2.1.2.5 Android Services . . . . . . . . . . . . . . . . . . . 16 2.1.2.6 Android Platform Security . . . . . . . . . . . . . . 17 2.1.3 Android Testing Fundamentals . . . . . . . . . . . . . . . . 18 2.1.4 Android Testing Tools . . . . . . . . . . . . . . . . . . . . . 19 2.2 GUI Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.1 GUI Testing Android Applications . . . . . . . . . . . . . . 23 i Contents ii 3 Android Ripper 31 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.2.2 AndroidRipper Model . . . . . . . . . . . . . . . . . . . . . 36 3.2.3 AndroidRipper Driver . . . . . . . . . . . . . . . . . . . . . 38 3.2.4 AndroidRipper Test Case . . . . . . . . . . . . . . . . . . . 40 3.2.5 AndroidRipper Service . . . . . . . . . . . . . . . . . . . . . 41 3.2.6 AndroidRipper Master-Slave Interaction . . . . . . . . . . . 42 3.3 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.1 Subject Application . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.3.3 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3.3.1 Application Preconditions . . . . . . . . . . . . . . 46 3.3.3.2 Considered Testing Techniques . . . . . . . . . . . 46 3.3.3.3 Testing Environment . . . . . . . . . . . . . . . . . 48 3.3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.3.5 Source Code Coverage Analysis . . . . . . . . . . . . . . . . 51 3.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4 Considering Context Events in Event-Based Testing of Mobile Applications 56 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.2 Event-Based Testing Techniques for Mobile Apps . . . . . . . . . . 58 4.3 Techniques for Event-Patterns Based Testing . . . . . . . . . . . . . 61 4.4 Implementing Event-Based Testing in the Android Platform . . . . 65 4.5 A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Contents iii 5 AGRippin: a novel search based testing technique for Android applications 70 5.1 Search Based Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.2 The AGRippin Tecnique . . . . . . . . . . . . . . . . . . . . . . . . 71 5.2.1 Representation . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.2.2 Crossover . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.2.3 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 5.2.4 Fitness Evaluation . . . . . . . . . . . . . . . . . . . . . . . 76 5.2.5 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 5.2.6 Combination Technique . . . . . . . . . . . . . . . . . . . . 80 5.3 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.3.1 Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.3.2 Experiment Environment and Setup . . . . . . . . . . . . . . 82 5.3.3 Results and Discussions . . . . . . . . . . . . . . . . . . . . 84 6 Exploiting the saturation effect in automatic random testing of android applications 88 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.2 Monkey Fuzz Testing Tools . . . . . . . . . . . . . . . . . . . . . . 92 6.3 The Testing Process . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.3.1 The Testing Process Implementation . . . . . . . . . . . . . 95 6.3.1.1 Constants and Variables . . . . . . . . . . . . . . . 98 6.3.1.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . 99 6.3.2 The Testing Infrastructure . . . . . . . . . . . . . . . . . . . 99 6.4 An exploratory study . . . . . . . . . . . . . . . . . . . . . . . . . . 102 6.5 Experimentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.5.1 Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 6.5.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Contents iv 6.5.3 Experimental Procedure . . . . . . . . . . . . . . . . . . . . 110 6.5.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 6.5.4.1 Saturation results . . . . . . . . . . . . . . . . . . . 111 6.5.4.2 Effectiveness results . . . . . . . . . . . . . . . . . 112 6.5.5 Lessons learned . . . . . . . . . . . . . . . . . . . . . . . . . 114 7 A parallel and distributed implementation of GUI Ripping Tech- niques 118 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 7.2 A parallel implementation of Android Ripper . . . . . . . . . . . . . 119 7.3 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 7.3.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . 121 7.3.2 Variables & Measures . . . . . . . . . . . . . . . . . . . . . . 123 7.3.2.1 Independent Variables . . . . . . . . . . . . . . . . 123 7.3.2.2 Dependent Variables . . . . . . . . . . . . . . . . . 124 7.3.3 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . 124 7.3.3.1 Objects of the Experiment . . . . . . . . . . . . . . 124 7.3.3.2 Experimental Procedure . . . . . . . . . . . . . . . 125 7.3.4 Results & Discussion . . . . . . . . . . . . . . . . . . . . . . 125 8 Conclusions & Future Work 129 List of Figures 2.1 Android Framework Architecture . . . . . . . . . . . . . . . . . . . 7 2.2 Activity States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3 Callback invocation sequences . . . . . . . . . . . . . . . . . . . . . 13 2.4 Android UI Input Controls . . . . . . . . . . . . . . . . . . . . . . . 14 2.5 Fragment Adaptability . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6 Fragment Life cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.7 Android Testing Framework Overview . . . . . . . . . . . . . . . . 19 2.8 Contributions about Android GUI Testing per Year . . . . . . . . . 24 3.1 GUI Testing Automation Mechanisms . . . . . . . . . . . . . . . . . 32 3.2 Overview of Android Ripper . . . . . . . . . . . . . . . . . . . . . . 35 3.3 Package Diagram of Android Ripper . . . . . . . . . . . . . . . . . 37 3.4 Conceptual Model of a GUI Interface . . . . . . . . . . . . . . . . . 37 3.5 AndroidRipper Driver Package . . . . . . . . . . . . . . . . . . . . . 39 3.6 AndroidRipper Test Case Package . . . . . . . . . . . . . . . . . . . 41 3.7 AndroidRipper Service Package . . . . . . . . . . . . . . . . . . . . 42 3.8 AndroidRipper Master-Slave Interaction . . . . . . . . . . . . . . . 43 3.9 Excerpt of the initialized user interface . . . . . . . . . . . . . . . . 45 3.10 GUI Tree inferred by T1 . . . . . . . . . . . . . . . . . . . . . . . . 51 3.11 GUI Tree inferred by T4 . . . . . . . . . . . . . . . . . . . . . . . . 51 3.12 Trend of the LOCs Coverage for Trolly . . . . . . . . . . . . . . . . 54 v