Technical Debt Management in the Context of Agile Methods in Software Development Graziela Simone Tonin THESIS PRESENTED AT INSTITUTE OF MATHEMATICS AND STATISTIC OF THE UNIVERSITY OF SÃO PAULO TO OBTAIN THE TITLE OF DOCTOR IN SCIENCE Program: Computer Science Advisor: Prof. Dr. Alfredo Goldman Co-advisor: Prof. Ph.D. Carolyn Seaman This research was supported by Fapesp - Brazil proc.2011/23470-7, CNPQ - Brazil proc. 555595/2010-2, and IBM proc.207731/2014-4 São Paulo, March, 2018 Technical Debt Management in the Context of Agile Methods in Software Development The original version of the Ph.D. thesis in Computer Science presented by the candidate Graziela Simone Tonin to a judging committee. Technical Debt Management in the Context of Agile Methods in Software Development The original version of the Ph.D. thesis in Computer Science presented by the candidate Graziela Simone Tonin to a judging committee. Doctoral Thesis Committee: • Prof. Ph.D. Carolyn Seaman (Co-Advisor) - (UMBC - Department of Information Systems) - USP - University of São Paulo • Prof. Ph.D. Damian Andrew Tamburri - POLIMI - Department of Electronics, Information, and Bioengineering • Prof. Dr. Eduardo Guerra - INPE - Associated Laboratory for Computing and Applied Mathematics • Prof.Dr.GuilhermeHortaTravassos-UFRJ-SystemsEngineeringandComputer Science Program • Prof. Ph.D. Xiaofeng Wang - UNIBZ - Faculty of Computer Science Abstract TONIN,G.S. Technical Debt Management in the Context of Agile Methods in Software Development. Thesis (PhD) - Institute of Mathematics and Statistics, University São Paulo, São Paulo, 2018. The technical debt field covers an critical problem of software engineering, and this is one of the reasons why this field has received significant attention in recent years. The technical debt metaphor helps developers to think about, and to monitor software quality. The metaphor refers to flaws in software (usually caused by shortcuts to save time) that may affect future maintenance and evolution. It was created by Cunningham to improve the quality of software delivery. Many times the technical debt items are unknown, unmonitored and therefore not managed, thus resulting in high maintenance costs throughout the software life-cycle. We conducted an empirical study in an academic environment, during two offerings of a labo- ratory course on Extreme Programming (XP Lab) at University of São Paulo and in two Brazilian Software Companies (Company A and B). We analyzed thirteen teams, nine in the Academy and four in the Companies environment. The teams had a comprehensive lecture about technical debt and several ways to identify and manage technical debt were presented. We monitored the teams, performed interviews, did close observations and collected feedback. The obtained results show that the awareness of technical debt influences team behavior. Team members report thinking and discussing more on software quality after becoming aware of techni- cal debt in their projects. We identified some impacts on the teams and the projects after having considered technical debt. A conceptual model for technical debt management was created includ- ing ways of how identifying, monitoring, categorizing, measuring, prioritizing, and paying off the technical debt. A few approaches and techniques for the technical debt management, identification, monitoring, measure, and payment are also suggested. Keywords: Technical Debt, Technical Debt Management, Technical Debt Awareness, Identifying Technical Debt, Monitoring Technical Debt, Technical Debt Impact, Software Quality. i Resumo TONIN, G. S. Gerenciamento de Dívida Técnica no Contexto de Desenvolvimento de Software Ágil. Thesis (Doctorate Degree) - Instituto de Matemática e Estatística, Universidade de São Paulo, São Paulo, 2018. A metáfora de dívida técnica engloba um importante problema da engenharia de software e essa é uma das razões pelas quais este campo tem recebido uma grande atenção nos últimos anos. Essa metáfora auxilia os desenvolvedores de software a refletirem sobre e a monitorarem a qualidade de software.Ametáforaserefereafalhasnosoftware(geralmentecausadasporatalhosparaeconomizar tempo) que podem afetar a futura manutenção e evolução do mesmo. A metáfora foi criada por Cunningham com o objetivo de melhorar a qualidade das entregas de software. Muitas vezes as dívidas técnicas não são conhecidas, monitoradas e nem geridas, resultando em um alto custo de manutenção ao longo do ciclo de vida do software. Logo, conduziu-se um estudo empírico na academia, durante duas ofertas da disciplina de Pro- gramação Extrema (XP Lab) na Universidade de São Paulo e em duas empresas Brasileiras de de- senvolvimento de software (Empresa A e B). Foram analisados treze times, sendo nove na academia e quatro nas empresas. Os times tiveram uma apresentação sobre dívida técnica e foram apresen- tadas algumas sugestões de abordagens para gerir dívida técnica. Monitorou-se os times, foram realizadas entrevistas, observações fechadas e informações foram coletadas. Os resultados mostraram que considerar dívida técnica influenciou o comportamento dos times. Eles reportaram que após considerar dívida técnica passaram a refletir e discutir mais a qualidade do software. Identificou-se alguns impactos nos times e nos projetos depois de considerarem dívida técnica. Um modelo conceitual para gestão de dívida técnica foi criado, incluindo formas, técnicas e abordagens de como identificar, monitorar, categorizar, medir, priorizar e pagar os itens de dívida técnica. Palavras-chave: Dívida Técnica, Gestão de Dívida Técnica, Consciência de Dívida Técnica, Iden- tificação de Dívida Técnica, Monitoramento de Dívida Técnica, Impacto de Dívida Técnica, Qual- idade de Software. ii Acknowledgments My sincere gratitude to everyone who, in any way, has been part of my life at this important moment. People who have supported and helped me along the way. I would like to thank my advisor, Professor Alfredo Goldman, for his relentless support in all these years. For encouraging me and opening doors so that I could achieve my objective. Professor Alfredo, who is a brilliant scientist, has taught me so much more than science and technical debt, he has taught me what true advisor is like. He did everything in his power to help me win awards, like the IBM Fellowship, attend renowned scientific summits and never stop learning. Beyond it all, he was always concerned about my study-life balance. Professor, thank you so much for your support, tips, suggestions, advice, and proofreading. I am forever thankful to you. I would also like to express my gratitude to my co-advisor, Professor Carolyn Seaman, for the unconditional support troughtout my PhD, especially when we worked together at UMBC. She made me feel warm and at home even in negative temperatures, and despite being so far from home. I am grateful to you and your amazing family. Working with you every day is having the opportunitytobeinspired,evolveandlearn,whilestandingnexttoabrilliantmindandabigheart. Meeting you was one of the best aspects of my PhD and what an honor it was to work with you. Thank you so much for your support, tips, advice, suggestions, and proofreading. I will be forever grateful to you for teaching me and inspiring with so much dedication and love. Dear advisor and co-advisor, you are not just excellent advisers, you inspire me. It is a privilege to have had you on my journey. A big thanks to the Professors who accepted to contribute to this research as a member of the Thesis committee. Thank you for sharing your knowledge and experience at this very important moment in my life. Thank you so much, Professors PhD. Damian Andrew Tamburri, Dr. Eduardo Guerra, Dr. Guilherme Horta Travassos and PhD. Xiaofeng Wang. I truly admire your work. A big thanks to the Federal University of Fronteira Sul and to the magnificent Dean Professor JaimeGioloforauthorizingmyabsencewhileIwasworkingonmythesis.Theheadsofdepartments Rosane Binotto and Lísia Regina Ferreira for approving and encouraging the training and learning of our professors. A big thanks to the current and former coordinators of the Computer Science course for always collaborating to make it possible and providing a compatible schedule, and to all the colleagues who supported me, encouraged and worked harder and longer so that I could focus on my thesis. A special thank you to Professor Dr. Raquel Pegoraro, who gave me all support since I started to UFFS. A big thanks to the colleagues and friends I made at USP, who gave me support, working hard, sharing their knowledge and being crucial partners on this journey. A special thanks to the English Teacher and Proofreader, who gave me such essential support, working hard, so that I could delivery my best. My affection and gratitude. iii iv A special thanks to my parents, my true heroes, who, by example, taught me to never give up on my dreams, even at difficult times. Despite not having the opportunity to study, they always encouraged and supported me unconditionally in the pursuit of my dreams and never let me give up the knowledge. Today, if I have the opportunity to achieve the highest level of knowledge it is because of the many years you have worked hard on the farm so that I could have the privilege of dedicating to my studies. This achievement is a tribute to you. I love you! To my sisters who have always supported me and have been worried about my well-being to endure the hardships of life. Thank you so much! I love you both! To my brother, sister-in-law and the little nephew and niece who even though live so far away but are always rooting for my success. My little niece, this brilliant mind that inspires and motivates me to go beyond. Thank you all. Family, I love you all and without your support, none of this would have been possible. My aunt Nair, uncle Jacir, godfathers Glaucia and Ivan, cousins Enriete, Everton and Tuilo, and all their lovely family who take care of me in important things in life that I often can not care. Thank you for being my daily support and making me strong to move forward. You were instrumental in making this achievement possible. My gratitude! I love you! To my grandparents, those who support me and protect me from the skies, and especially to my grandmother Alfreda, and the grandfather Alcides, who in his 93 years gives us the privilege of learningsomuchbysharinghiswisdom.GrandFawhatagreatexampleyouare.Iwantyoutoknow I always follow your advice and teachings – and as you say, “You know”. I love you grandfather. A big thank you to my cousins, uncles, aunts, for supporting me and making my days lighter. My gratitude. I love you! To my friends, who sent me many messages, who support me, forgive my absence, and give me the most invigorating hugs: thank you, thank you and thank you! To those who are close and those who live far, thank you for your unconditional love and support, you are my example. You are the lightness, joy and strength that allow me to rise and move on. I love you! A very special thank you to the love of my life, Michel Miotto Barbosa, this brilliant mind that inspires me. You are so genial and your heart is so human. You are incredible, is my mentor and guide, a true idol. There are no words to describe how fantastic you have been on this long journey. Michel always absolutely and unconditionally supports and encourages my dreams, either by sponsoring them or by turning the night by my side just to ensure I am well.Your support makes a world of difference. Michel is always worried – often more than myself - if I am well fed, if I am warm in the American winter, if I am fine, whether I need a ’mate’, a coffee or a hug. Thank you for understanding and enduring my absence, the stressful days and for always being there. No matter where I am following my dreams, I know that somewhere in the world I have there is an incredible man that I can count on unconditionally. How could I not fall in love with you every day?! This achievement not only is dedicated to you, it has your fundamental participation, this is your achievement too, my love. You are the best! Thank you! I love you! In particular, I dedicate this thesis to the memory of two dear friends, Tania Mara Ferla and Daniel Dalabona, I miss you guys. Two incredible humans now shining from the sky. A special thank you to those who spend some time sharing their knowledge and discussing the models and results of this research, especially to Joseph Yoder, a bright mind. To you all!!! My affection and gratitude! Contents Acknowledgments iii List of Abbreviations ix List of Figures x List of Tables xiii 1 Introduction 1 1.1 Research Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.5 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Technical Debt 8 2.1 The Technical Debt Metaphor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Technical Debt Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3 Technical Debt Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4 Incurring Technical Debt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5 Impact of Technical Debt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6 Technical Debt Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.6.1 What Is Not Technical Debt . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.6.2 Technical Debt Prevention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.6.3 Technical Debt Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.6.4 Technical Debt Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.6.5 Technical Debt Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.6.6 Technical Debt Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.6.7 Tools to Support Technical Debt Management . . . . . . . . . . . . . . . . . . 36 2.6.8 Technical Debt Prioritization and Decision Making . . . . . . . . . . . . . . . 37 2.6.8.1 Technical Debt Payment . . . . . . . . . . . . . . . . . . . . . . . . 38 2.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3 Research Method and Design 41 3.1 Research Method and Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.1.1 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 v CONTENTS vi 3.1.2 Action Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2 Methods and Techniques of Data Collection and Analysis . . . . . . . . . . . . . . . 43 3.2.1 Method of Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3 Research Steps Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4 Classroom Studies 49 4.1 The Extreme Programming Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.1.1 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.1.2 The Informative Workspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.2 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.2.1 Technical Debt Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.2.1.1 Extreme Programming Laboratory 2013 Offering . . . . . . . . . . . 53 4.2.1.2 Extreme Programming Laboratory 2014 Offering . . . . . . . . . . . 54 4.2.2 Interviews and questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.2.1 Extreme Programming Laboratory 2013 Offering . . . . . . . . . . . 54 4.2.2.2 Extreme Programming Laboratory 2014 Offering . . . . . . . . . . . 55 4.2.2.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.3.1 Extreme Programming Laboratory 2013 Offering . . . . . . . . . . . . . . . . 56 4.3.2 Extreme Programming Laboratory 2014 Offering . . . . . . . . . . . . . . . . 58 4.3.3 Open Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.3.4 Axial Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.3.5 Selective Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.3.6 Tool-Based Static Analysis Results . . . . . . . . . . . . . . . . . . . . . . . . 64 4.3.7 Threats to the Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5 Company A Studies 68 5.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.1.1 Study 01. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.1.1.1 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.1.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.1.2 Study 02. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.1.2.1 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.1.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 5.1.2.2.1 The Results of Data Analysis with Grounded Theory . . . 78 5.1.2.2.1.1 Open Coding . . . . . . . . . . . . . . . . . . . . . . 78 5.1.2.2.1.2 Axial Coding . . . . . . . . . . . . . . . . . . . . . . 78 5.1.2.2.1.3 Selective Coding . . . . . . . . . . . . . . . . . . . . 80 5.1.2.2.2 The Proposed Template . . . . . . . . . . . . . . . . . . . . 80 5.1.2.2.3 The Conceptual Model for Technical Debt Management . . 82 5.1.3 Threats to the Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.2 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 CONTENTS vii 6 Company B Studies 85 6.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6.1.1 Team A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.1.2 Team B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.1.3 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.1.3.1 Team A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.1.3.1.1 First Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.1.3.1.2 Second Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.1.3.1.3 Third Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.1.3.2 Team B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.1.3.2.1 First Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.1.3.2.2 Second Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.1.3.2.3 Third Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.1.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.2 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 7 Conceptual Model for Technical Debt Management 97 7.1 The Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.1.1 Technical Debt Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 7.1.2 Technical Debt Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.1.2.1 Comparison with the Models . . . . . . . . . . . . . . . . . . . . . . 107 7.1.3 Framework for Technical Debt Management . . . . . . . . . . . . . . . . . . . 107 7.1.4 Framework to Manage and Reduce Technical Debt . . . . . . . . . . . . . . . 108 7.1.5 Conceptual Model of Technical Debt . . . . . . . . . . . . . . . . . . . . . . . 109 7.1.6 Technical Debt Management Framework . . . . . . . . . . . . . . . . . . . . . 110 7.1.7 Managing Architectural Technical Debt: A Unified Model . . . . . . . . . . . 112 7.1.8 Technical Debt Management Cycle . . . . . . . . . . . . . . . . . . . . . . . . 113 7.1.9 Summary of the Comparison of the Models . . . . . . . . . . . . . . . . . . . 114 7.1.9.1 Threats to the Validity . . . . . . . . . . . . . . . . . . . . . . . . . 116 7.2 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 8 Conclusion 118 8.1 Summary of the Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 8.2 Discussion of findings by each research question. . . . . . . . . . . . . . . . . . . . . 120 8.2.1 RQ1 (Company A): Which decisions may motivate a team to incur technical debt in agile software development projects? . . . . . . . . . . . . . . . . . . . 121 8.2.2 RQ2 & RQ6 (Company A and B): What types of technical debt items did the agile software development team incur? . . . . . . . . . . . . . . . . . . . 121 8.2.3 RQ3 (Company A): What is the impact on an agile software development team of the technical debt in their project? . . . . . . . . . . . . . . . . . . . 121 8.2.4 RQ4 & RQ7 (Company A and B): Which techniques and tools can agile software development teams use to identify, monitor and manage technical debt? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122