Measuring the effect of gamification on user adoption of a software application Kelly van den Dool [email protected] 3672719 MBI Master Thesis First supervisor: F. Dalpiaz (Utrecht University) Second supervisor: J. M. Houtkamp (Utrecht University) Facilitator: F. Zielemans (Centric) Measuring the effect of gamification on user adoption of a software application Abstract The focus of this research is on the intersection of user adoption and gamification. User adoption is important for both developers and buyers of new technologies, and it is a complex problem that involves many variables. A promising and relatively new way to increase user adoption is gamification, which is expected to increase user engagement and participation. Conventional wisdom tells that gamification affects user adoption via user motivation. To be able to effectively influence user adoption via gamification, it is important to know if and how gamification can influence the motivation of users. There are few scientific studies that focus on this exact problem, and its underlying principles. Therefore, the goal of this research is to investigate the underlying motivational principles that play a role when using gamified software applications. Another goal is to discover how to implement gamification in a software application to increase user adoption. The research approach includes assembling a conceptual model, on the basis of literature about user adoption, motivation and gamification. Next, we develop a strategy to implement gamification to increase identified regulation (a form of motivation) in users. Based on this strategy, we implement a gamified mockup version of an existing software application in Axure RP. We also develop a control mockup version of the application without game elements. The designs are evaluated in two ways: (i) interviews with experts from the fields user experience, psychology and communication, and game design; (ii) through a controlled experiment that measures the attitude and beliefs of participants after using one of the mockup applications. The findings in literature suggest that using game elements other than points, badges, and leaderboards (PBL) could lead to a more sustainable kind of user motivation. However, the results of the controlled experiment show no significant effect of the gamification implementation on user adoption or the underlying variables we hypothesized to be influenced. The experts find the gamified application to be more suitable (than the control version) for onboarding new users of the application. The main conclusions from this research are that the way gamification was implemented in this research does not lead to higher identified regulation, and higher hedonic motivation, and has no effect on intention to use, nor intention to recommend the application. However, the expert interviews lead to several recommendations for developers of gamified applications. This thesis recommends to first thoroughly investigate the goal of the application, and based on that to devise a strategy on how to implement gamification to support this goal. An additional recommendation is to implement one or more elements that support the user in the transition into a game mindset. Kelly van den Dool 3672719 2 Measuring the effect of gamification on user adoption of a software application Acknowledgements Conducting this research and writing my master thesis about it has truly been an educational experience. I enjoyed the variety of tasks I conducted; from researching literature from multiple disciplines, to creating a gamified prototype in Axure, to interviewing experts, to running a MANCOVA in SPSS. During the nine months it took to devise the conceptual model, design the prototype applications, collect participant data, conduct interviews, analyze, and report the whole process, many people contributed to this research project. I could not have completed this thesis without their help and support. I want to thank Fabiano Dalpiaz, for the many meetings that provided much insight and motivation, and for his helpful suggestions on the structure of my thesis. I want to thank Joske Houtkamp for her useful directions on creating and conducting a questionnaire and the comprehensibility of the report. Thanks also to François Zielemans and Frederick de Wit, who were willing to give me the opportunity to conduct this research at Centric, and gave me many tips and advices to bring several parts of my project to satisfying conclusions. Furthermore, I want to thank Jeroen van Venrooij and Harry Killaars, for their effort to help me recruit enough participants, and all participants for filling out the questionnaire. I want to thank Tom Hoffmeijer, Christine Swankhuisen, and Marries van de Hoef, for participating in an interview. Thanks also to the people that participated in the pilot study, for their comments and suggestions on how to improve the designs and questionnaire. Last but not least, I want to thank my friends, the WhatsApp group “scriptie fun”, and my colleagues at Centric, who supported me, provided different viewpoints, and motivated me with humor. I especially want to thank Joost, who supported me both morally and scientifically. Enjoy reading! Kelly van den Dool 3672719 3 Measuring the effect of gamification on user adoption of a software application Table of Contents 1. Introduction .......................................................................................................................................... 7 1.1 Problem statement ....................................................................................................................... 7 1.2 Research questions ....................................................................................................................... 8 1.3 Context and constraints ................................................................................................................ 9 1.4 Structure ....................................................................................................................................... 9 2. Theoretical background ...................................................................................................................... 10 2.1 User adoption ............................................................................................................................. 10 2.2 Engagement ................................................................................................................................ 21 2.3 Motivation ................................................................................................................................... 22 2.4 Gamification ................................................................................................................................ 24 3. Conceptual framework ....................................................................................................................... 28 3.1 Hypotheses ................................................................................................................................. 29 4. Research methods .............................................................................................................................. 30 4.1 The design science method......................................................................................................... 30 4.2 The empirical cycle ...................................................................................................................... 31 4.3 Expert interview setup ................................................................................................................ 32 4.4 Controlled experiment setup ...................................................................................................... 33 4.5 Validity and reliability ................................................................................................................. 35 5. Artifact design ..................................................................................................................................... 37 5.1 Requirements gamified ............................................................................................................... 37 5.2 Design decisions .......................................................................................................................... 38 5.3 Non-gamified design ................................................................................................................... 44 5.4 Pilot testing ................................................................................................................................. 47 5.5 Limitations................................................................................................................................... 48 6. Results ................................................................................................................................................. 49 6.1 Interview results ......................................................................................................................... 49 6.2 Controlled experiment results .................................................................................................... 52 7. Discussion ............................................................................................................................................ 62 7.1 Influencing user adoption ........................................................................................................... 62 Kelly van den Dool 3672719 4 Measuring the effect of gamification on user adoption of a software application 7.2 Implementing gamification ......................................................................................................... 63 7.3 Designing experiments ................................................................................................................ 63 7.4 Analyzing the results ................................................................................................................... 64 7.5 Generalizing the results .............................................................................................................. 65 7.6 Recommendations and future research ..................................................................................... 66 7.7 Limitations................................................................................................................................... 67 7.8 Conclusion ................................................................................................................................... 68 8. References .......................................................................................................................................... 69 9. Appendix ............................................................................................................................................. 76 9.1 Appendix A: The interview protocol ........................................................................................... 76 9.2 Appendix B: The informed consents of the interviewed experts ............................................... 80 9.3 Appendix C: Overview of the questionnaire ............................................................................... 83 9.4 Appendix D: SPSS output ............................................................................................................ 87 List of tables Table 2.1 Rogers' definitions of innovation characteristics. ....................................................................... 12 Table 2.2. Overview of the added external factors in TAM3. ..................................................................... 15 Table 2.3. Overview of UTAUT constructs and reasons for their exclusion. .............................................. 18 Table 2.4. Moore and Benbasat's user perceptions. .................................................................................. 20 Table 2.5. The processes supporting different forms of external regulation. ............................................ 23 Table 2.6. Overview of game mechanics. ................................................................................................... 27 Table 2.7. Overview of game components. ................................................................................................ 27 Table 4.1. Calculations per variable. ........................................................................................................... 35 Table 5.1. Overview of strategies to increase user adoption. .................................................................... 38 Table 5.2. Overview of changes to our designs after pilot testing. ............................................................ 47 Table 6.1. The assumptions for MANCOVA. ............................................................................................... 55 Table 6.2. The correlations between the dependent variables. ................................................................. 56 Table 6.3. Results of the Shapiro-Wilk test for each of the variables for each of the groups. ................... 56 Kelly van den Dool 3672719 5 Measuring the effect of gamification on user adoption of a software application List of figures Figure 2.1. Rogers’ innovation-decision process model. ............................................................................ 11 Figure 2.2. The antecedents and consequents of user perceptions. .......................................................... 13 Figure 2.3. Davis’ Technology Acceptance Model. ..................................................................................... 14 Figure 2.4. The UTAUT model. .................................................................................................................... 17 Figure 2.5. The end-user computing satisfaction model. ........................................................................... 19 Figure 2.6. The motivation-continuum. ...................................................................................................... 23 Figure 2.7. Game element framework. ....................................................................................................... 26 Figure 3.1. Conceptual model of our research. .......................................................................................... 28 Figure 4.1. The design science method....................................................................................................... 30 Figure 5.1. The homepage of the gamified mockup. .................................................................................. 38 Figure 5.2. Johan invites the user to help him. ........................................................................................... 40 Figure 5.3. The screen showing the game missions.................................................................................... 41 Figure 5.4. Flowchart of the gamified mockup. .......................................................................................... 42 Figure 5.5. The wrap-up page for the toxic cloud mission.......................................................................... 43 Figure 5.6. The rewards for completing the missions. ................................................................................ 44 Figure 5.7. Screenshot of the homepage of the non-gamified mockup. .................................................... 45 Figure 5.8. The homepage explanation arrow for the emergency kits. ..................................................... 46 Figure 5.9. The second step in directing the user to the actual emergency kits page. .............................. 46 Figure 6.1. The gender distribution of the participants within and between the groups. ......................... 53 Figure 6.2. The age distribution of the participants within and between the groups. ............................... 53 Figure 6.3. Boxplot of identified regulation for both groups. ..................................................................... 58 Figure 6.4. Boxplot of hedonic motivation for both groups. ...................................................................... 59 Figure 6.5. Boxplot of intention to use for both groups. ............................................................................ 59 Figure 6.6. Boxplot of intention to recommend for both groups. .............................................................. 60 Figure 6.7. Scatterplot of Relative autonomy index and intention to use. ................................................. 61 Figure 6.8. Boxplot of relative autonomy index for both groups. .............................................................. 61 Figure 7.1. Diagram with recommendations. ............................................................................................. 67 Kelly van den Dool 3672719 6 Measuring the effect of gamification on user adoption of a software application 1. Introduction The possibilities to create technologies that make people’s life easier are endless. However, not all innovations are actually adopted by their intended users. The relation between the characteristics of an innovation and its adoption has been a classical problem in innovation literature (Tornatzky & Klein, 1982). In the Eighties, the adoption of technologies also caught the interest of information technology researchers, and technology adoption became a new line of research within the information systems field (Huff & Munro, 1985). Although there are multiple models that attempt to explain why certain information systems get adopted and others do not, this remains a complex problem (Dwivedi, Rana, Chen, & Williams, 2011) with many variables involved. To increase the user adoption of a software application, there are different options. When looking at models explaining user adoption (Agarwal & Prasad, 1998; Venkatesh, Thong, & Xu, 2012), some examples of ways to increase user adoption are: increasing the usability of the application, employing various marketing channels, or pricing the application competitively. We propose to employ gamification as an additional method for increasing user adoption. Gamification is the concept of using game design elements in non-game contexts, and is hypothesized to have a positive effect on user engagement and user participation with software (Deterding, Dixon, Khaled, & Nacke, 2011). Gamification has been a trending topic since a few years, and is presented as the next generation method to increase customer engagement. Following the big hype of the concept in industry, gamification also attracted the attention of academic researchers (Hamari, Koivisto, & Sarsa, 2014). Although the body of research on gamification is growing, there is more research needed on different aspects of the concept. This research explores if, and in what way gamification can be applied to increase the user adoption of a software application. 1.1 Problem statement Both organizations that build and organizations that buy a new technology hope to realize a return on their investment, in terms of either financial return or other key performance indicators, such as higher efficiency and improved processes. Whether or not an investment in a new technology generates the desired return depends heavily on the user adoption of that technology (Agarwal & Prasad, 1998). If the intended end-users do not use a new technology, the expected return for the buyer of the system will be lacking. In turn this could lead to lower revenues for the organization that built the new technology. Therefore, it is desirable for organizations to catalyze the user adoption in any possible way. To be able to influence this process an understanding of why technologies are adopted is needed. As stated above, this is a complex problem and multiple models exist that try to explain it (Dwivedi et al., 2011). However, knowing which variables influence user adoption is not enough, for the most useful knowledge is obtained by understanding how to influence these variables. In popular literature and business practice, gamification is labeled as a method for increasing customer engagement (Hamari et al., 2014). Conventional wisdom tells that gamification increases user motivation, which leads to increased engagement with the system. In turn, engagement should lead to usage activity and quality (Hamari, 2013). To know how to influence user motivation, it is important to Kelly van den Dool 3672719 7 Measuring the effect of gamification on user adoption of a software application have an understanding of motivation theories. Research on motivation is grounded in psychology, and distinguishes between intrinsic and extrinsic motivation (Deci & Ryan, 1985). Gamification could be influencing either one of these or both, but several studies suggest that increasing extrinsic motivation undermines intrinsic motivation (Ryan & Deci, 2000a). This finding shows that it is important to know exactly how gamification influences motivation, so organizations can use gamification to cause the wanted motivational effect in users. In scientific research there are few studies that focus on this exact problem. Hamari and Koivisto (2013) focus on social motivations to use a gamified application. However, most existing research on gamification focuses on the relation between gamification and engagement without measuring the underlying motivations of the users (Cheong, Cheong, & Filippou, 2013; Downes-Le Guin, Baker, Mechling, & Ruyle, 2012; Halan, Rossen, Cendan, & Lok, 2010). Though this is also interesting, it is scientifically relevant to search for the underlying principles of phenomena. Other researchers also recognize the need to look beyond user engagement and into why it is that gamification is effective (Amir & Ralph, 2014; Kappen & Nacke, 2013). From this we deduce that it is important to research the underlying motivational aspects that play a role when using gamified software applications. Summarized, organizations want to increase motivation, user engagement and user adoption of software applications, by applying gamification. However, it is not entirely clear in what way gamification influences motivation, and thus it is not clear how gamification should be implemented in a software application to generate the desired effect. 1.2 Research questions The described problem statement leads to the main question of this research: How to determine the effect of gamification on user adoption of a software application? This main research question is split up in the following sub research questions: RQ 1 What variables play a role in user adoption of a software application? RQ 2 How could gamification be implemented to increase user adoption? RQ 3 What experiment could be used to measure the effect of the devised gamification implementation on user adoption? RQ 4 What effect has the devised gamification implementation on user adoption of a certain software application? RQ 5 How and to what extent can the outcomes of such an experiment be generalized to other software applications? The first research question focuses on getting an overview of the current state of user adoption research, and the variables that play a role in increasing user adoption. The second research question is about mapping the various kinds of game elements that could be implemented and selecting the right ones to influence users in the desired way. The intention of the third research questions is to set up an experiment and measure the effects of the implemented gamification. The fourth and fifth questions evaluate the findings of the experiment. Kelly van den Dool 3672719 8 Measuring the effect of gamification on user adoption of a software application 1.3 Context and constraints The problem this thesis tackles—how can gamification be implemented to increase user adoption?—is a problem that is experienced in a real world situation. This research is conducted in the context of a specific case company, Centric. Centric is a full service IT-provider that offers software solutions, IT outsourcing, business process outsourcing, and staffing services. The company was founded in 1992, as Sanderink group, and in the year 2000 the name was changed to Centric. While Centric is originally a Dutch organization, it now operates in eight European countries. In the Netherlands it operates in various markets, such as financial, construction, local government, supply chain, housing associations, and health care. Within Centric there are different divisions, and our research is carried out for the Solution Engineering division. Centric Solution Engineering is developing a new software application in the domain of public order and safety, for three Dutch municipalities. The name of the application is ‘Omgevingsdashboard’ (ODB), which translates to ‘Environment dashboard’. The purpose of the application is to inform inhabitants of the municipalities about safety risks in their living environment, as well as to inform them during actual emergency situations. This distinction between communication about risks and communication about actual incidents is indicated with the terms ‘cold phase’ and ‘hot phase’. The ODB is a responsive web application, and the front page shows a map with icons indicating risks and/or incidents in the surroundings of the user. These icons are clickable to view more detailed information, as well as the action perspective of the user. The application can be used anonymously, but creating a user- account will give the user the opportunity to sign up for SMS-alerts. Those alerts are sent out to subscribers in the area of an incident. Users can select one or more areas for which they want to receive alerts, for example the area around their home, as well as the school of their children. For the ODB Centric and the municipalities want to find ways to boost the user adoption when the application is launched. They are interested in using gamification for this purpose, and this research will therefore include creating a gamified version of the application that intends to solve this problem. Since the development of the application should go on as planned, this research project should interfere the least possible with the actual development. Thus we are creating a gamified mockup version of the ODB, which will be tested and evaluated before using the results to improve the actual ODB. 1.4 Structure The structure of this thesis is the following. Chapter 2 gives an overview of the theoretical background concerning user adoption, engagement, motivation and gamification. Chapter 3 shows the conceptual model that depicts the variables we expect to influence user adoption, and the hypotheses we want to test. Chapter 4 presents the research methods we used to answer our research questions. Chapter 5 explains the design decisions we made while creating the gamified mockup ODB, and the strategies we used to increase user adoption. Chapter 6 gives an analysis of our experiment results, and Chapter Error! Reference source not found. summarizes our findings, and answers our research questions. Kelly van den Dool 3672719 9 Measuring the effect of gamification on user adoption of a software application 2. Theoretical background The goal of this literature review was to gain an understanding of the variables and their relationships that play a role in user adoption and gamification. Therefore we conducted a so-called theoretical review of the literature. This form of literature study uses existing conceptual and empirical studies to uncover theoretical structure and relationships between concepts and constructs. The main goal is to construct a conceptual framework with corresponding hypotheses (Paré, Trudel, Jaana, & Kitsiou, 2015). The search engine we used is google scholar, because it gives the most comprehensive results from all journals, conference and workshop proceedings, book chapters and books, etc. We started our search by using general keywords: user adoption, technology adoption, technology acceptance, gamification, user engagement, motivation. By selecting prominent articles, that are cited many times, and meta-analyses we first gained an overview of the field. While reading these articles we navigated back- and forward using references and citations, to gain a deeper understanding of the fields. We also used keywords we find in literature to search for articles we possibly missed during earlier searches, due to the use of different keywords. Based on our findings in literature we started composing a conceptual model. While creating this model we kept looking for literature in several incremental steps, to fill occurring gaps in the model. The rest of this chapter gives an overview of our findings in the four topics we drew from while building our conceptual model. Section 2.1 gives an overview of models that are used to explain user adoption. Section 2.2 focuses on engagement. Section 2.3 investigates motivation, and finally Section 2.4 explains gamification and its related concepts. 2.1 User adoption The concept of user adoption has been researched for more than half a century now, and has been indicated with different names throughout the years. Researchers in innovation studies were the first ones to analyze this concept, with a focus on the relation between the characteristics of an innovation and its adoption and implementation (Tornatzky & Klein, 1982). A well-known researcher in this field is Rogers, who first published a book containing a meta-analysis of 405 studies on innovation diffusion in 1962. Nine years later, he published a second edition, and the number of studies had already grown to 1500 (Rogers & Shoemaker, 1971). Rogers’ research is not limited to ICT innovations, and based on his meta-analysis of innovation studies, he proposed five characteristics that influence the adoption rate of an innovation (Rogers, 1983). In subsequent years, many researchers have investigated these five characteristics. In the eighties the concept of user adoption gained attention in the MIS quarterly journal (Huff & Munro, 1985), and a new extensive model was proposed by another well-known researcher in the field: Fred Davis (1986). The model Davis proposes, the Technology Acceptance Model (TAM), focuses on investigating the mediating variables between system characteristics and actual system use (Davis, 1986). Just as with the innovation characteristics of Rogers, many researchers used Davis’ TAM in later studies. Unlike Rogers (1962), who uses the term adoption, Davis uses the term ‘acceptance’. The terms adoption and acceptance seem to be referencing to the same concept; the use of a system by its Kelly van den Dool 3672719 10