SpringerBriefs in Computer Science Series Editors Stan Zdonik Peng Ning Shashi Shekhar Jonathan Katz Xindong Wu Lakhmi C. Jain David Padua Xuemin Shen Borko Furht V. S. Subrahmanian Martial Hebert Katsushi Ikeuchi Bruno Siciliano For furthervolumes: http://www.springer.com/series/10028 Scott Tilley Tauhida Parveen • Software Testing in the Cloud Migration and Execution 123 Scott Tilley TauhidaParveen Department of Computer Sciences Independent Consultant Florida InstituteofTechnology Melbourne,FL Melbourne,FL USA USA Additionalonlinematerialavailableunderwww.stitc.organdwww.springer.com/978-3-642-32121-4 ISSN 2191-5768 ISSN 2191-5776 (electronic) ISBN 978-3-642-32121-4 ISBN 978-3-642-32122-1 (eBook) DOI 10.1007/978-3-642-32122-1 SpringerHeidelbergNewYorkDordrechtLondon LibraryofCongressControlNumber:2012943956 ACMComputingClassification(1998):D.2,K.6 (cid:2)TheAuthor(s)2012 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe work. Duplication of this publication or parts thereof is permitted only under the provisions of theCopyrightLawofthePublisher’slocation,initscurrentversion,andpermissionforusemustalways beobtainedfromSpringer.PermissionsforusemaybeobtainedthroughRightsLinkattheCopyright ClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. While the advice and information in this book are believed to be true and accurate at the date of publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with respecttothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) To my parents —Scott Tilley To mom, dad, and grandpa —Tauhida Parveen ‘‘It’s testing, Jim, but not as we know it’’ Foreword This book fills a much-needed gap in research and guidance for advanced prac- titioners and scholarly investigators. It focuses on the neglected, but important topic of software testing in the cloud. Testing in the cloud is far more complex than standard software testing because resources constantly shift for different executions, many of the resources are not controlled by the consuming organi- zation, and the business case for success is significantly different for cloud pro- viders and consumer organizations.However,itis preciselyfor these reasons that testing of potential cloud applications can make or break the business case for migration to the cloud. Testing with the use of cloud resources can provide substantial potential ben- efits because of the potential for large-scale economies of scale and the ability to accessstateofthepracticecomputerresourceswithoutasubstantialITinvestment. However,itisnotasolutionforeveryorganization;norareitsbenefitsautomatic. Because of these factors, testing in the cloud requires a sober strategic and tech- nicalanalysisthatfullyidentifiesgoalsoftheeffort,totalcosts,potentialrisks,and expected outcomes. This type of analysis, while still not commonplace, can demonstrateitsworthbyenablingdecisionstobemadewithfactsandlogicrather thanemotion.Thisbookintroducesasetofmethodsandtoolstobeginsystematic testing in the cloud. During the 15 years that I have been associated with Prof. Scott Tilley, he has consistently tackled hard problems with a rare combination of tight discipline, pragmatic application, and strong technical insight. This book is no exception. Scottrecognizedthe potential ofcloudcomputingwhenitwasmerelyaspeck on thehorizon,andhebegantoorganizeacommunityofinterestaroundthetopicof testing in the cloud. Scott’s collaboration with Dr. Tauhida Parveen has been fruitfulinthewritingofthisbookandthedevelopmentofacommunityofinterest. Tauhida has co-developed, implemented, and refined the SMART-T decision frameworkandtheHadoopUnitexecutionenvironmentoverthepastseveralyears. Each of the three major sections of the book illustrate how the co-authors routinely improve on ideas, methods, and tools and then apply them to novel situations. The SMART-T framework provides the decision-making backbone for vii viii Foreword the rest of the book. This method extends the SMART framework that several colleagues and I developed at Carnegie Mellon’s Software Engineering Institute. The extensions tocloud testing provide a uniquefoundation for making informed business, technical, and operational decisions. While the SMART-T framework enables rational decision-making, Hadoop- Unit builds on Apache Hadoop, which is a framework for supporting large-scale distributed applications. HadoopUnit provides a distributed environment for con- current execution of testing on a cluster of machine using cloud resources. This enables empirical verification of the decisions that are made, detailed data on whether the anticipated economies of scale in the use of cloud testing are being realized, and early feedback on whether mid-course corrections are needed. The five case studies represent the third leg of this tightly woven book. They demonstrate promising early results and provide practical lessons learned in migrating testing to the cloud through a combination of SMART-T and HadoopUnit. These cases studies, which were developed to vary both in com- plexity and in the nature of the application, measure the relative performance improvement of using the cloud in testing as opposed to single machine testing. I found the following lessons learned to be particularly compelling for advanced practitioners: • the wisdom of factoring in sufficient time to master the complexities of the current testing environment, • theadvisabilityofplanningforpartialandphasedmigratingtestingtothecloud, • the importance of the selection of test cases to take advantage of the perfor- mance improvement from the use of HadoopUnit with large test suites, • theneedtoplanforthechallengesofselection,configuration,andautomationof softwarebecauseofthelackofvisibilityintokeyinfrastructureelementsthatis inherent in testing in the cloud, • the necessity of detailed analyses to determine the causes for failures in the execution of test cases because of the multitude offactors that may be causing the failures, such as bandwidth, network latency, and other configurations. Forresearchers,thisbookoffersanintellectualfoundationthatprovidesasetof building blocks for future research programs. For practitioners considering a migration to the cloud, this book, with its focus on strong intellectual grounding and practical applications, should be required reading. The empirical applications servetheinterestsofboththepractitionerandresearchcommunitiesandprovidea much-needed practical basis for the methods and tools that are introduced. Dennis Smith Software Engineering Institute Carnegie Mellon University Preface This book is focused on one of the most important and long-standing problems facing the software testing community: lengthy test execution times. For a large, complexsystem,theamountoftestcasesinaregressiontestsuitecanrangefroma few hundred to several thousands, which can take hours or even days to execute. Regression testing also requires considerable resources that are often not readily available, precluding their use in an interactive setting, further contributing to an inefficient testing process. Traditional approaches to reduce the execution time for regression testing typicallyfocusonexcludingselectedtestsfromthesuitethatneedtoberunaftera change is made to the system. This research provides an alternate solution that does not require test case exclusion: leveraging the resources provided by cloud computinginfrastructuretofacilitatetheconcurrentexecutionoftestcases.Cloud computing offers the use of virtualized hardware, effectively unlimited storage, and software services that can aid in reducing the execution time of large test suites in a cost-effective manner. This book presents a decision framework called SMART-T to support migra- tionofsoftwaretestingtothecloud,adistributedenvironmentcalledHadoopUnit for concurrent execution of test cases in the cloud, and a series of case studies illustrating the use of the framework and the environment. Experimental results indicate a significant reduction in test execution time is possible when compared with a typical sequential environment. What Is Unique About This Book? Our work in software testing in the cloud began in 2009 with a collaborative project with a large industrial partner. In April 2010, we organized the workshop on the topic at the IEEE ICST conference in Paris [117]. The interest generated from this event resulted in the edited volume, Software Testing in the Cloud: Perspectives on an Emerging Discipline [118]. ix x Preface Our own focus has been primarily on testexecution. To facilitate this, we also examined the precursor issue of migrating testing to a cloud-based environment. Accordingly,theuniqueworkdescribedinthisbookhasthreeprimaryobjectives: 1. Create a decision framework that facilitates the disciplined migration of soft- ware testing to the cloud. 2. Develop a prototype distributed execution framework for concurrent test exe- cution in the cloud. 3. Demonstrate the viability and evaluate the effectiveness of the approach using case studies carried out in real world settings. This work has the potential to significantly impact the three major areas of software testing, cloud computing, and system migration. The results reported in thebookaffirmthatanewareaofresearchcanbenefitfromajudiciousblendingof bestpracticesfromtheothers.Weillustratetheneedforadecisionframeworkfor migrating testing to the cloud and provide the testing community a sample deci- sion framework that they can use. We also demonstrate that Hadoop, typically used for distributed data processing at scale, can also be beneficial when used in software testing. The intellectual merit of this research is the advancement of knowledge and understandingofthenascentfieldofsoftwaretestinginthecloud.Acreativeblend oftheunderlyingareasofsoftwaretesting,cloudcomputing,andsystemmigration was used to develop this potentially transformative area of research. Our results show that when migrating software testing is desirable from a business point of view, and feasible from a technical point of view, the results can be acceptable from an operational point of view. For example, as described in Chap. 4, a 165x performance improvement was realized after migrating test cases for GCC to the cloud. Who Should Read This Book? Software testing in the cloud is a subject that should interest advanced practitio- ners, academic researchers, and those in education and training. For advanced practitioners, the issue of cloud computing and its impact on the field of software testing is becoming increasingly relevant. For academic researchers, this is a subjectthatisrepletewithinterestingchallenges;therearesomanyopenproblems that graduate students will be busy for years to come. Thetopicofsoftwaretestinginthecloudisanexcellentvehicleforeducational exploration.Forexample,itbuildsuponcoreknowledgeofsoftwaretestingwhile introducing new concepts such as cloud computing into the course material. This topic was used in a graduate course on software engineering with considerable success: the students seemed to enjoy the exposure to a leading-edge research program, and the instructors benefits from the contributions of the students in the Preface xi assignments and group project—which closely followed the main themes pre- sented in this book [116]. To disseminate the results of this work, a community of interest was created called ‘‘Software Testing in the Cloud’’ (www.STITC.org). This community has beenactiveinworkingtogetherandsharingideasinmigratingtestingtothecloud, performingtestinginthecloud,andtestingofthe clouditself.Weencourage you to contact us and get involved in this exciting new area. Outline of This Book The book is structured according to the three key elements of our research: migrating software testing to the cloud using SMART-T, executing test cases in the cloud using HadoopUnit, and performing case studies to illustrate the use of the approach in a variety of problem settings. SMART-T: Chapter 2 describes SMART-T, a decision framework for migrating testing in the cloud. The descriptive framework has three parts: (1) addressingbusinessdrivers(whymigratetothecloud);(2)technicalfactors(when tomigratetothecloud);and(3)operationalresults(howtomigratetothecloud). Each part of the framework ends in a decision point related to the desirability (business), feasibility (technical), and acceptability (operational) of the migration to the cloud. HadoopUnit: Chapter 3 explains HadoopUnit, a distributed environment for concurrentexecutionoftestcasesinthecloud.HadoopUnitisbuiltuponHadoop, the open source implementation of the MapReduce programming model. Several techniqueswereillustratedtomigratetestcasesfromalegacyenvironmenttothe cloud-based environment, such as wrapping test cases using JUnit interfaces. Case Studies: Chapter 4 presents several case studies where SMART-T and HadoopUnithasbeenapplied.Experimentalresultsindicateasignificantreduction in test execution time is possible when compared with a typical sequential envi- ronment.Theresultsalsoshowedthatmigratingtestingtothecloudisnotalways thebestsolutiontotheproblem;sometimesadifferentapproachmightbeneeded, suchasreengineeringthetestcasestoidentifyshortcomingsbeforeanymigration is attempted. The book concludes with Chap. 5, which provides a summary of the results from the case studies and summarizes our lessons learned. Acknowledgments We are extremely grateful for the wonderful personnel at Springer who helped makethisbookareality,Inparticular,wewouldliketothankRalfGerstnerforhis patience and encouragement.