Leveraging the Wisdom of the Crowd in Software Testing BOOKS ON SOFTWARE AND SYSTEMS DEVELOPMENT AND ENGINEERING FROM AUERBACH PUBLICATIONS AND CRC PRESS Adaptive, Dynamic, and Resilient Systems Enterprise Integration and Information Architecture: Niranjan Suri and Giacomo Cabri (Editors) A Systems Perspective on Industrial Information ISBN 978-1-4398-6848-5 Integration Li Da Xu Agile Strategy Management: Techniques for ISBN 978-1-4398-5024-4 Continuous Alignment and Improvement Soren Lyngso The Essential Guide to Serial ATA and SATA Express ISBN 978-1-4665-9607-8 David A. Demming ISBN 978-1-4822-4331-4 Big Data, Mining, and Analytics: Components of Strategic Decision Making Formal Languages and Computation: Models and Stephan Kudyba Their Applications ISBN 978-1-4665-6870-9 Alexander Meduna ISBN 978-1-4665-1345-7 Big Data Analytics Strategies for the Smart Grid The Influential Project Manager: Winning Over Carol L. Stimmel Team Members and Stakeholders ISBN 978-1-4822-1828-2 Alfonso Bucero Bursting the Big Data Bubble: The Case for ISBN 978-1-4665-9633-7 Intuition-Based Decision Making Introduction to Software Project Management Jay Liebowitz (Editor) Adolfo Villafiorita ISBN 978-1-4822-2885-4 ISBN 978-1-4665-5953-0 Buying, Supporting, Maintaining Software and Large Scale and Big Data: Processing and Equipment: An IT Manager’s Guide to Controlling Management the Product Lifecycle Sherif Sakr and Mohamed Gaber (Editors) Gay Gordon-Byrne ISBN 978-1-4665-8150-0 ISBN 978-1-4822-3278-3 Mobile Social Networking and Computing: BYOD for Healthcare A Multidisciplinary Integrated Perspective Jessica Keyes Yufeng Wang and Jianhua Ma ISBN 978-1-4822-1981-4 ISBN 978-1-4665-5275-3 Case Studies in System of Systems, Enterprise Programming Languages for MIS: Concepts Systems, and Complex Systems Engineering and Practice Alex Gorod, Brian E. White, Vernon Ireland, Hai Wang and Shouhong Wang S. Jimmy Gandhi, and Brian Sauser (Editors) ISBN 978-1-4822-2266-1 ISBN 978-1-4665-0239-0 The SAP Materials Management Handbook The CIO’s Guide to Oracle Products and Solutions Ashfaque Ahmed Jessica Keyes ISBN 978-1-4665-8162-3 ISBN 978-1-4822-4994-1 Secure Data Provenance and Inference Control Creating and Marketing New Products with Semantic Web and Services Bhavani Thuraisingham, Tyrone Cadenhead, Rosanna Garcia Murat Kantarcioglu, and Vaibhav Khadilka ISBN 978-1-4822-0360-8 ISBN 978-1-4665-6943-0 Empowering Project Teams: Using Project Zen of Cloud: Learning Cloud Computing by Followership to Improve Performance Examples on Microsoft Azure Marco Sampietro and Tiziano Villa Haishi Bai ISBN 978-1-4822-1755-1 ISBN 978-1-4822-1580-9 Leveraging the Wisdom of the Crowd in Software Testing Mukesh Sharma Rajini Padmanaban CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2015 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20140805 International Standard Book Number-13: 978-1-4822-5449-5 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit- ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright. com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com Contents Foreword ix Acknowledgments xi overview xiii chApter 1 introducing the crowd 1 What Is a Crowd? 2 History of Crowdsourcing 2 History of Crowdsourcing in Software Product Development 4 What Are the Traits of the Crowd That Make It Relevant to Be Engaged in Various Stages of Product Development? 6 Did You Know? 9 chApter 2 An overview into crowdsourcing 11 You Perform, We Provide the Platform 11 What Is Crowdsourcing? 11 Understanding the Varied Forms of Crowdsourcing 12 Crowd Creation 14 Crowd Voting 16 Crowd Wisdom 18 Crowd Funding 20 Is Crowdsourcing the Same as Outsourcing? 21 What Is Crowdsourced Testing and Where Does It Fit in This Picture? 23 Understanding Crowdsourced Testing through Examples 27 The Rising Popularity of Crowdsourcing 29 Did You Know? 30 v vi Contents chApter 3 why leverAge the crowd For soFtwAre testing? 33 Are You Ready to Stand Out in a Crowd? 33 A Peek at Software Quality 33 Software Quality in the Current Day and Age 35 The Need for Alternate Solutions to Assure and Control Software Quality 37 Tying the Gap in Quality to the Definition of Crowdsourced Testing 38 A Case Study on Facebook Relevant in the World of Agile Operations 39 Scenario 39 Challenges Posed by Scale 40 Solutioning 41 Summary 42 Leveraging Crowdsourced Testing in Various Testing Areas 42 Functional Testing 42 UI Testing 43 Usability Testing 43 Accessibility Testing 44 Performance Testing 44 Security Testing 45 Globalization Testing 45 Compatibility Testing 46 Did You Know? 47 chApter 4 is crowdsourced testing A no-BrAiner solution to All QuAlity proBlems? 49 What Not to Crowdsource in Your Test Effort 51 Challenges and Limitations of a Crowdsourced Test Effort 55 Did You Know? 62 chApter 5 how to successFully implement crowdsourced testing 63 Best Practices in Implementing a Successful Crowdsourced Test Effort 64 When Do We Crowdsource Our Testing Effort? 65 What Do We Crowdsource in Our Testing Effort? 68 How Do We Crowdsource Our Testing Effort? 71 Examples: Practical Implementations of Crowdsourced Testing 75 Engagement Models in Leveraging a Crowdsourced Test Effort 82 Did You Know? 84 chApter 6 deFect mAnAgement in crowdsourced testing 85 What Is Defect Management? 85 What Are Some Core Challenges in Defect Management in a Product Team? 86 Contents vii Is Defect Management Any Different in a Crowdsourced Test Effort? 90 What Can Go Wrong If Defect Management Is Not Effective Specific to Crowdsourced Testing? 92 Can I Leverage Best Practices in Defect Management from a Formal Testing Effort in My Crowdsourced Testing? 93 Are Defect Tracking and Measurement Important in Crowdsourced Testing Too? 94 Are There Any Case Studies to Help Understand Crowdsourced Testing Defect Management in Real Time? 95 Did You Know? 97 chApter 7 win your teAm’s support in implementing crowdsourced testing 99 Did You Know? 108 chApter 8 let’s understAnd crowdsourced testing holisticAlly 111 Chapter 1: Introducing the Crowd 112 Chapter 2: An Overview into Crowdsourcing 112 Chapter 3: Why Leverage the Crowd for Software Testing? 113 Chapter 4: Is Crowdsourced Testing a No-Brainer Solution to All Quality Problems? 114 Chapter 5: How to Successfully Implement Crowdsourced Testing 115 Chapter 6: Defect Management in Crowdsourced Testing 116 Chapter 7: Win Your Team’s Support in Implementing Crowdsourced Testing 117 Chapter 8: Let’s Understand Crowdsourced Testing Holistically 118 Additional Myths and Facts 119 Is Crowdsourced Testing a Viable Career to Build? 120 Who Can Become a Crowdsourced Tester? 120 What Should I Do to Prepare and Be Successful in Crowdsourced Testing Assignments? 122 Did You Know? 127 chApter 9 Future oF crowdsourced testing 131 Sizing the Crowdsourcing Industry 131 Future of Crowdsourced Testing 133 Market Trends 135 Business Trends 137 Technology Trends 139 End User Trends 140 Did You Know? 142 chApter 10 Building An ecosystem Around crowdsourced testing 143 Crowd Testers 145 Crowd Seekers 145 viii Contents The Platform 146 Common Knowledge Repository on Crowdsourced Testing 148 Tools to Help with the Crowdsourced Testing Effort 149 Who Is Responsible for Building This Ecosystem? 150 What to Watch for as We Let the Ecosystem Develop 151 Did You Know? 151 reFerences 153 index 157 Foreword Thousands of years of history, dating back to the Paleolithic Era, have taught humans the value of working together. Several strong warriors were required to work together to bring down a wooly mammoth for food and clothing. Specialists needed to learn how to grow food or pray for crops, while others depended on their talents. Human his- tory is full of examples of people working together to overcome seem- ingly insurmountable challenges. The building of the Great Pyramid of Giza or the Temples of Ancient Greece, curating the Hanging Gardens of Babylon, constructing the Freedom Tower in New York City through the accuracy of Wikipedia and the r ichness of Twitter and Facebook—all of these amazing accomplishments in human his- tory came to be only because of the coordinated work of many. In my own home many years ago, the phrase “many hands makes light work” was met with groans, as it was shorthand for “kids help out with chores.” However, as we have entered a 21st century with u biquitous connectivity and an “always on” mentality, the idea of everyone being available to contribute some of their discretionary time to a com- mon cause is not too far-fetched. In fact, it’s becoming the norm. We saw the rise of SMS displaced temporarily by Farmville updates, and then on to WhatsApp, SnapChat, Instagram and Pinterest. People working together to share information and improve each other’s lives through a collective effort. While it may be premature to dismiss ix x Foreword Garret Hardin’s Tragedy of the Commons – in which individuals act selfishly contrarian to the best interests of a group – it’s safe to say that technology has leveled the playing field and enabled a more col- laborative model of collective contribution where everyone can “win” in their own unique way. There are thousands of examples throughout human history to illustrate the value of a diverse group of people focused on a chal- lenge. If we were to pore through that hypothetical list, I would imag- ine that much time would elapse before software testing came in to view. Fortunately, Mukesh and Rajini have done the heavy lifting to create a framework for all of us to use to successfully apply these collaborative crowdsourcing techniques to improve the quality of our software. This book is unique in explaining how to combine the power of crowdsourcing with the craft of software testing to share practi- cal experience, lessons learned, and provide guidance for others to hit the ground running. The rich set of examples provided here helps set the context for all practitioners to help improve understanding of a fast moving field. The depth of experience comes through with guid- ance on how to build a platform, create an ecosystem, and avoid the many potential problems and pitfalls that present a threat to success- ful crowdsource testing initiatives. The Did-You-Know c onclusion offers pragmatic suggestions and offers a synopsis of advice to move forward on crowdsourced testing. Quality is an amorphous, ephemeral condition, especially in software—and very hard to achieve in an increasingly online and mobile world. High quality requires meeting the needs and require- ments of a broad and diverse set of people—and to do so requires a diverse set of analysts—a crowd—to help ensure everyone’s needs are met. This book offers a formula for helping everyone be success- ful in leveraging the crowd to improve the quality of the software experience. Ross Smith Director of Test, Skype Division, Microsoft