Towards the adoption of DevOps in software product organizations: A maturity model approach Rico de Feijter Rob van Vliet Erik Jagroep Sietse Overbeek Sjaak Brinkkemper Technical Report UU-CS-2017-009 May 2017 Department of Information and Computing Sciences Utrecht University, Utrecht, The Netherlands www.cs.uu.nl ISSN: 0924-3275 Department of Information and Computing Sciences Utrecht University P.O. Box 80.089 3508 TB Utrecht The Netherlands Towards the adoption of DevOps in software product organizations: A maturity model approach Rico de Feijter1, Rob van Vliet2, Erik Jagroep2, Sietse Overbeek1, Sjaak Brinkkemper1 1Utrecht University, Utrecht, The Netherlands {R.deFeijter,S.J.Overbeek,S.Brinkkemper}@uu.nl 2Centric, Gouda, The Netherlands {Rob.van.Vliet,Erik.Jagroep}@centric.eu Abstract. This technical report describes a study conducted at Centric and concerns the adoption of DevOps in software product organization (SPOs), which are organizations that produce software for multiple customers and thus need to take into account the wishes and needs from all these customers, while developing software. For these SPOs there is a need to constantly release faster to customers in order to preserve customer satisfaction in the form of being able to quickly release new features and provide bug fixes. However, releasing software at a faster pace comes with the need to better align the concerns of stakeholders that reside in the chain of releasing soft- ware. In particular, development and operations need to be aligned as these parties traditionally have arranged their processes differently from one another and work in silos. However, in order to create a smooth and fast end-to-end flow when it comes to releasing software, DevOps provides a way to deal with the aforementioned and takes into consideration not only development and op- erations, but also other stakeholders such as quality assurance (Q/A), product management and information security. When further observing DevOps, the phenomenon touches upon both cultural and technical matters to attain fast release of software, has a wide scope and could be seen as a movement, but is still young and not yet formally defined. Also, noadoptionmodelsorfine-grainedmaturitymodelsshowingwhattocon- sider to adopt DevOps and how to grow more DevOps mature were identified. As a consequence, this research attempted to fill these gaps and consequently brought forward six DevOps drivers and sixty three capabilities aiming to adopt DevOps to a mature extent. These sixty three capabilities form part of six- teen focus areas, which in turn belong to three perspectives. This combination of drivers, perspectives, focus areas and capabilities was used to construct a DevOps competence model showing the areas to be taken into account in the adoption of DevOps. Next to that, the perspectives, focus areas and capabil- ities were used to create a maturity model showing a fine grained path to be followed in order to reach a mature DevOps state. In order to come to the artifacts described above, several data collection meth- ods were leveraged, among which are semi-structured interviews at three dif- ferent organizations and a literature review. Other than that, two validation rounds were conducted of which the first round encompassed expert opinion sessions in which the DevOps competence model and the perspectives, focus areas and capabilities were validated. The second validation round entailed expert opinion sessions in which the focus areas, capabilities and the maturity model were validated. Finally, a case study was carried out with the final capa- bilities, which were processed in a self assessment that was sent out to nineteen assessees. Of these nineteen assessees, eight assessees filled in the self assess- ment, which ultimately ended up in seven useful filled in assessments for which maturity profiles could be made that showed the assessees the state of DevOps maturity and the next steps to be taken in order to grow more DevOps mature. Even though initial results showed that the artifacts were found applicable, many opportunities for future research are still left including the gathering of a richer dataset, a more profound validation of the perspectives, focus areas, capabilities, DevOps competence model and maturity model and a wider case study that evaluates all capabilities to their fullest extent in different settings. Other future research could aim at situational factors and deeper scrutiniza- tion of the intertwinement of product management with DevOps. Keywords: DevOps, Competence model, Maturity model 1 Introduction This technical report covers a study regarding the adoption DevOps in software product organizations (SPOs), which are organizations that produce software for multiple customers and also install and maintain the software for these customers in their own datacenter or at the customer’s site, which differs from a situation in which an organization develops software for its own sake. Yet, the tendency is that increasingly more SPOs opt for cloud-based software that is installed and maintained at a central location being it in an own datacenter or at another central location. More concretely, SPOs move away from on-premise, and move to the cloud (Pahl, Xiong, & Walshe, 2013) that allows for faster releasing (Lawton, 2008). However, when striving for faster releasing, stakeholders residing in a SPOs are also required to collaborate more closely (Wettinger, Andrikopoulos, & Leymann, 2015). At this point, DevOps comes into play. Moreover, the term concerns improving the collab- oration between stakeholders with a special focus on development and operations to achieve fast high quality releases (Waller, Ehmke, & Hasselbring, 2015;Wettinger, Breitenbu¨cher, & Leymann, 2014b). Moreover, the stakeholders involved in this process stretch from product manage- ment, which is concerned with requirements management and release planning re- lated activities, up to and including the customer. DevOps thus has a wide scope and, in fact, could touch upon the entire organization (Zwieback, 2014). However, DevOps in particular focuses on the stakeholders concerned with the development of the software, the maintenance and monitoring of the infrastructure on which the software runs and support. Despite the increasing popularity of DevOps and organizations having an under- standing of the motivations to adopt DevOps and the advantages the notions brings (Smeds, Nybom, & Porres, 2015), DevOps requires further investigation, as there is still no clear overview of DevOps practices that enables organizations to adopt DevOps (Lwakatare, Kuvaja, & Oivo, 2016). In the same vein, there are hardly processes and methods available that prescribe how to implement DevOps (Erich, Amrit,&Daneva,2014)makingitdifficultfororganizationstodecidewhatpractices to adopt, how to adopt these and how to improve incrementally. Rong, Zhang, and Shao (2016) also underpin this fact by stating that no scientifically validated models for DevOps that aid organizations in implementing DevOps and thus maturing in DevOps in a step wise way exist. Also, while many sources address the existence of a relationship between DevOps and the remainder of the organization, it often remains unclear what this relationship exactly entails. Kim, Behr, and Spafford 1 (2014), for instance, mention the relationship with product management, but only detail this at the level of adopting small batch sizes, among others. To make an attempt to fill the gap described above, this research aims to create a DevOps competence model that incorporates so-called focus areas, which show SPOs the areas to be considered in the adoption of DevOps and is inspired by Bekkers, Van de Weerd, Spruit, and Brinkkemper (2010). Another artifact aimed to be established in this research is a maturity model, which is inspired by Van Steen- bergen, Bos, Brinkkemper, Van de Weerd, and Bekkers (2013) and builds further on the contents of the DevOps competence model and incorporates a growth path aiding SPOs in moving towards a mature DevOps situation. The chapters that follow in this technical report are the theoretical framework, which positions the DevOps topic and the instruments to be used in order to realize the aforementioned artifacts. Next, the research approach chapter describes the path to be followed in order to come to research results and also reflects on this path by describing the research approach results. Thereafter, the journey towards the DevOps maturity model chapter explains the research results, whereafter the case study chapter reports on the case study results. Subsequently, the discussion chapter describes the main contributions and limitations of the conducted study. At last, the technical report concludes with outlining the conclusions and future work. 2 2 Theoretical framework In this chapter, an overview is provided of literature to position the research topic. To this end, the literature review starts with an explanation of DevOps. Then, after outlining DevOps, the emphasis is put on DevOps models and competence models, after which the theoretical framework concludes with describing DevOps maturity models and maturity models in general. 2.1 DevOps For decades, organizations have been looking for new ways to improve their software development processes to keep up with business and market demands (Boehm, 1988; Haley, 1996; Herden, Farias, & Albuquerque, 2016). To this end, organizations have been embracing new ways of working to develop software at a higher pace. It is, for instance, a well-known fact that many organizations have shown interest in adopting an agile way of developing to face ever changing requirements (Capodieci, Mainetti, & Manco, 2014). Aside from optimizing software development, organizations have also been looking into improving other facets of Information Technology (IT). For instance, many companies have adopted frameworks such as the Information In- frastructure Library (ITIL) encompassing best practices that aim to enhance ICT service management and structure operational processes (Potgieter, Botha, & Lew, 2005). Nevertheless, traditional organizations have often had their development and opera- tions activities carried out by siloed development and operations units, each having their own goals. In such a situation, development is tasked with creating new func- tionality and fixing bugs, whilst operations is tasked with maintaining a reliable infrastructure that enables software to run stably and giving customer support. In other words, both development and operations strive for the opposite, as develop- ment strives for change, whereas operations strives for stability. This is problematic since they both are dependent on one another, when it comes to releasing soft- ware in a timely manner. Therefore, it comes as no surprise that in the process of both stakeholders achieving their different goals, frictions come across because of divergent motivations, processes and tooling, which demotivates stakeholders to cooperate (Hu¨ttermann, 2012). The aforementioned, thus, makes clear that cooperation between development and operations is of crucial importance to create a smooth end-to-end flow. More specif- ically, if development and operations are not aligned, the whole process of releasing 3 functionality and processing feedback will slow down, which results in a slower re- sponse to customer demand(Ward & Zhou, 2006). Hu¨ttermann (2012) makes the alignment between the aforementioned parties clear by presenting four DevOps ar- eas in a so called DevOpsAreaMatrix, which is adopted in Figure 2.1. Here, the first area addresses that development and operations should collaborate on deliv- ering functionality to production. The second area incorporates the feedback loop from operations back into development in the form of streaming information such as runtime behavior from operations back to development. The third area aims at embedding development into operations by having development focus on non- functional requirements that are concerned with operations. The fourth and last area stipulates that DevOps strives to embed operations into development by hav- ing operations share knowledge with developers on the feasibility of solutions under development. Area 3: Embed development into operations Area 1: Extend development to operations Dev Ops Area 2: Extend operations to development Area 4: Embed operations into development Figure 2.1: The DevOps area matrix adapted from Hu¨ttermann (2012). However, not only development and operations stakeholders fall under the DevOps umbrella. Rather, DevOpsaimsatachievinganoverallsmoothend-to-endflow. The notion, therefore, also takes into consideration other stakeholders involved in releas- ing software such as quality assurance (QA), product management and information security (Kim, Behr, & Spafford, 2014; Iden, Tessem, & Pa¨iv¨arinta, 2011; Swartout, 2014). As a result, all stakeholders in the DevOps movement should be aligned to create an overall smooth end-to-end flow (Kim, Behr, & Spafford, 2014). SPOs should thus consider DevOps to streamline the organization in order to cre- ate a better end-to-end flow. However, when observing the DevOps notion more profoundly, literature shows that the term still lacks a formal definition (Smeds, Nybom, & Porres, 2015; Bass, Weber, & Zhu, 2015), but is perceived as a move- mentencompassingpracticesthataimtoestablishacultureofcollaborationbetween stakeholders involved in the software development process wherein development and IT operations, as engaging parties, receive most of the attention (Kim, Humble, De- bois, Allspaw, & Willis, 2016; Plwakatare, Kuvaja, & Oivo, 2015). Apart from the enhancement of collaboration between stakeholders, another aim of DevOps concerns releasing software faster and reacting on feedback faster in order to bring added value to the customer at a fast pace and therewith preserve customer satisfac- 4 tion (Riungu-Kalliosaari, Ma¨kinen, Lwakatare, Tiihonen, & M¨annisto¨, 2016; Kim, 2013; Dyck, Penners, & Lichter, 2015). Furthermore, the literature above shows that DevOps is perceived as a movement, which indicates that DevOps is not a fixed process, tool or technology, but rather a mindset in which trust and respect between stakeholders is a fact and collaboration between stakeholders takes place in order to work towards a release in a cooperative fashion (Davis & Daniels, 2016). Because DevOps is described as a movement, the notion is continuous and not prone to aging out. Other observations in literature make clear that DevOps supports intentional pro- cesses that accelerate the rate by which businesses realize value (Davis & Daniels, 2016). To achieve this, DevOps advocates practices pertained to culture (e.g. sus- taining a culture of trust and respect (Davis & Daniels, 2016)), collaboration (e.g. adopting cross functional teams (Kim et al., 2016)), lean thinking (e.g. using the value stream mapping technique to optimize the DevOps movement (Kim, Behr, & Spafford, 2014) and automation (e.g. automating builds, tests and deployments to release software quicker (Hu¨ttermann, 2012)). In addition, DevOps aims to measure the effect of technical and social change (Davis & Daniels, 2016). This implies that DevOps alsoencompasses practices that link tomonitoring and measurement, which engender continuous improvement (Fitzgerald & Stol, 2017). Moreconcretely, practicesdescribedinliteraturerelatetomonitoringandmeasuring technical processes by using techniques such as application performance monitoring from which resulting data can be leveraged to provide fast feedback to, for in- stance, product management and development (Plwakatare, Kuvaja, & Oivo, 2015; Hu¨ttermann, 2012). As for social change, Davis and Daniels (2016) show that col- laboration between people can be monitored and measured by, for instance, using peer reviews in order to improve how people socially interact. As said before, DevOps aims to react faster on customer demand, which includes re- leasingsoftwarefasterandrespondingtofeedbackfaster. However, doingsorequires quality to be in check. Moreover, since release cycles are shorter and a more agile way of working is recommended by DevOps, software quality becomes a concern. Khomh, Dhaliwal, Zou, and Adams (2012) even claim that fewer bugs are fixed when using a rapid release model in lieu of a traditional release model. To handle this, lean thinking can again be employed to quickly find and eliminate bottlenecks that create waste in the DevOps movement at an early stage. Lean thinking in this form is then reflected in broken build detection mechanisms, which make it possible to detect broken software builds at an early stage and thus allows them to be fixed early in the process (Fitzgerald & Stol, 2014). Also, value stream mapping could again be useful to identify and eliminate bottlenecks standing in the way of releasing high quality software (Kim et al., 2016). However, practices associated with automation also contribute to better quality, since the automation of activities playing a role in the DevOps movement, such as 5 build creation, testing, and deployment, take away less consistent manual interven- tions, which are known to be more error prone (Virmani, 2015; Chen, 2015). Yet, not only practices purely aimed at lean thinking and automation contribute to better quality. In fact, continuous improvement through monitoring and mea- surement contribute to better quality as well (Orzen & Paider, 2016;Meissner & Junghanns, 2016). After all, monitoring and measurement might reveal that a piece of software needs to be improved as it causes an increased CPU load, to name an example. Perceiving the “soft” side, on the other hand, monitoring and measure- ment might reveal that better collaboration between development and operations is needed, which benefits the quality as well (Rossberg, 2014). As it turns out from the above, DevOps is a notion that covers many aspects. Therefore, to bring structure to the DevOps notion, the aspects mentioned in this section are summarized in a semantic net, which is used to visualize key concepts (Bock, Kattenstroth, & Overbeek, 2014) and can be consulted in Figure 2.2. is involved in DevOps involves Stakeholder Culture movement contributes to Fast high involves Collaboration quality release Is preserved engenders by involves Automation Customer involves Lean satisfaction involves Continuous improvement Figure 2.2: Semantic net of key concepts proposed by Bock et al. (2014) applied to the DevOps domain. The semantic net shows the interrelation of the concepts that fall into the domain of DevOps. Besides that, the contents (i.e. the concepts and the relationships) of the semantic net are shaped by the literature used in this section to outline DevOps. The observation was, for instance, made that automation forms part of DevOps and is needed to achieve higher quality. This fact led to the decision to mark automation as a concept that is involved in the concept ”DevOps movement” and plays a role in contributing to ”fast high quality release”. This way of thinking is also applied in the realization of the remainder of the semantic net. 6