v Contents About the Author.........................................................................................vii Acknowledgments.........................................................................................ix Introduction.....................................................................................................xi Chapter 1: Getting Started with APM.........................................................1 PART I: Planning..........................................................................................37 Chapter 2: Business Justification................................................................39 Chapter 3: Assessments...............................................................................57 Chapter 4: Staffing and Responsibilities..................................................107 Chapter 5: APM Patterns...........................................................................139 Chapter 6: The Pilot Evaluation...............................................................161 PART II: Implementation......................................................................177 Chapter 7: Deployment Strategies..........................................................179 Chapter 8: Essential Processes.................................................................215 Chapter 9: Essential Service Capabilities................................................229 PART III: Practitioners...........................................................................251 Chapter 10: Solution Sizing.......................................................................253 Chapter 11: Load Generation..................................................................285 Chapter 12: Baselines.................................................................................309 Chapter 13: The Application Audit.........................................................329 Chapter 14: Triage with Single Metrics..................................................357 Chapter 15: Triage with Baselines...........................................................395 Chapter 16: Triage with Trends..............................................................405 Chapter 17: Firefighting and Critical Situations....................................439 Index................................................................................................................459 xi Introduction Application Performance Management (APM) is the name given to the use of technology to initiate, deploy, monitor, fix, update and/or optimize systems within an organization. Application management software employs measurements of response times, and other component and resource interactions, to help manage the overall stability and usability of the software within its purview. This book, presented in three parts, is intended to take you from the first discussions about planning and assessing your monitoring needs, through the details and considerations of an APM technology implementation, and onto the types of skills, processes and competencies that you will need to be completely successful in the utilization of APM. And it is not simply a discussion of ideas for you to consider. The approach used is largely metrics-based. I have included the appropriate artifacts so that you may conduct these activities exactly the same as I would if I were working with you. My role as an APM practitioner is to guide stakeholders through every phase of the journey to APM excellence and the discussions in this book are taken directly from my presentations, engagement artifacts and training materials. In particular, I lead client teams through a comprehensive program of best practices that allow them to establish and staff an APM discipline. What is unique about this collection of best practices is that they are vendor-neutral, which is an enormous consideration when you reflect on the reality that your IT tools are selected from many different vendors. For my work, and especially my credibility in defining these practices as essential, they have to work for every vendor technology. I have included specific exercises which you can use to track your progress in establishing an APM discipline. For a wide variety of reasons, folks have difficulty getting the full measure of usefulness from their investments in APM. There is often a disconnect between the APM technology capabilities and the organizatioal capabilities in actually understanding and employing the technology. I developed many of these APM best practices by helping clients recover xii from any number of missteps in selecting and employing APM technology. APM can be complex as much as it is easy to use. While a good part of what I do might be considered brainstorming (discussing new techniques and approaches), an equal part is gut-wrenching and frustration-laden. This is the hard part—figuring out how to get your organization to reliably adopt APM. My motivation in writing this book is to show you what many users of APM miss and also how powerful the technology and processes can be. When I began in this industry there were no meaningful references nor texts from which to get some useful ideas. Even today I really don’t find anything substantial to address APM education across planning, implementation and best practices – especially with a vendor-neutral tone. So I’m going to put a stake in the ground with a bold and ambitious statement: APM starts here. Vendor Neutrality Yes, I am employed by a monitoring solution vendor. And you can figure out who by looking on the book cover. It is not the goal of this book to highlight or recommend any particular vendor or technology. Instead, we will focus on the processes that are applicable to any vendor solution, especially in a multi-vendor or heterogeneous monitoring environment. We will also pay particular attention to when it makes sense to employ a particular technology. When it is necessary to illustrate how a tool presents data, or otherwise how a tool might be employed to support one of these processes, you may reliably assume that any vendor of the underlying technology may provide that capability. And if I simply cannot avoid mentioning a specific type of tool then I will use a freely available or open source technology as the example. Any monitoring tool is appropriate—provided you understand its application and limitations. We will address that as well. In general, any visibility is better than “no visibility”. However, you cannot simply drop in a tool without considering some framework of processes to support and utilize the tool appropriately. This process gap is what many organizations fail to address and so their monitoring initiatives are less effective. My goal is simply to take whatever system you have today and reliably increase its capabilities and value to your organization – as far as you may want to advance it. xiii Conveniently, the guideline of vendor neutrality also fits in with our first and most important objective for a monitoring initiative: it is not the time to talk about specifics of a specific vendor technology solution. It is time to talk about the process of realizing an APM initiative. APM Starts Here This book is about how to do APM ‘right’. APM means different things to different people and this quickly becomes a fantastic gap between reality and expectation. My professional role is to jump into this gap and get things back on track. Maybe you want my job? Maybe you want to avoid seeing me turn up? Maybe you want to get this right the first time. Nothing available today is going to help you do it right the first time – save for this book. The single biggest gap in understanding APM is this: APM is not simply an end-user monitoring application. It is part of the IT infrastructure and the use of the tool will be divided over many stakeholders. Fail to accommodate and empower these stakeholders and your APM initiative will follow. APM is much more than a monitoring technology. It is a collaboration technology. Through the techniques in this book you are going to educate your stakeholders in the language of APM. You are going to harness their capabilities and tear down the barriers to cooperation. It is how your organization is going to finally get control of the application life cycle and improve software quality, without excuse or exception. Organization of this Book This book is presented in three parts: Planning, Implementation and Practitioner’s Guide. Introduction to Part 1: Planning Planning for an APM initiative is a course of action that will balance the capabilities of the IT team, the demands of the business and the alignment with corporate objectives. It is advantageous to move quickly from establishing the overall goals of the initiative, to formulate and xiv present (sell internally) a tactical plan that will have a high degree of success. Successful management and execution of this planning activity sets the stage for a successful deployment of the technology and implementation of the supporting processes. Failure to plan will result in an uneven or failed initiative. We are concerned with five major themes: • Establishing the business justification (Chapter 2) • Assessing your current monitoring capabilities (Chapter 3) • Understanding staffing requirements (Chapter 4) • Defining the catalog of services that will track the progress of the initiative (Chapter 5) • Executing a successful pilot evaluation leading to selection of a suitable technology (Chapter 6) Each of these themes will be handled in its own chapter and each needs to be addressed prior to the deployment phase, which is covered in the second part of this book: Implementation. Of these themes, staffing is probably the most controversial as this exposes the organization’s understanding of impact of APM and how it should be leveraged. Frankly put, while the greatest potential for APM initiative failure is inadequate scope, the second greatest has to do with staffing: who does the work of using APM technology and processes? You should not really be concerned with total manpower but more about who will fulfill the various APM roles, where the monitoring initiative will sit in the organization, and how it will interact with other stakeholders. An important aspect of the APM best practices presented here is the focus on an early and detailed assessment of existing skills, processes and competencies. This is specifically targeted to address the looming staffing/ownership question but also may be extended to survey the strengths/weaknesses of existing tools and groups within the IT organization and how they may be leveraged to help support and continue the pace of the APM initiative while avoiding any immediate increase in headcount. Doing more with less is always a theme in modern IT management and I find that any APM initiative will be under significant pressure to do exactly this. xv Some discussion of the service catalog, and how we use these defined capabilities as milestones to manage the evolutions of an APM discipline, is important because this topic is a frequent point of confusion. A catalog of services is a popular IT feature, and one that is directly descended from the ITIL descriptions of service management. I find while many organizations believe that this work is already done; few can quickly point to precisely what this catalog currently includes. This is another gap which can significantly impair the progress of the APM initiative. We focus on this service catalog concept because this is the area where we will need the greatest degree of tracking, that in fact, these services are actually being achieved. Because many organizations have already been initiated to this service catalog concept, much of my planning response is to make up for the earlier gaps, so I needed an efficient way to measure exactly what was missing from the existing service catalog. Very often a client will have a comprehensive set of services defined but no corresponding implementation roadmap is specified. And after some months, attention wanes and other priorities come to the fore, and the service catalog fades from view. I will show you how to successfully manage the realization of the APM service catalog by aligning with an evolution of staff skills and demonstrable competencies. This results in a more manageable job of tracking and reporting progress towards realizing the service catalog. The final discussion topic in this planning part of the book: how to define a pilot of a new technology, is also a point where the IT team can easily fail. The IT organization has probably been evaluating a variety of technologies for many years. However, the current crop of economic pressures, down-sizing and transition of responsibilities off-shore have eliminated much of the hard-earned experience in defining and executing a pilot evaluation. I see the pilot evaluation as an important validation of the overall objectives of the monitoring initiative. Failing to do a thorough and timely job here will of course cripple any chance of a successful deployment and ongoing use of the technology. I will take the opportunity here to simply address that gap and present a successful best practice for doing a solid evaluation of a prospective technology. xvi Introduction to Part 2: Implementation In this Implementation part, chapters 7-9, we want to look at the implementation of APM from the perspective of a project manager tasked with bringing the initiative to realization. With a small scope, it is rare that you will have a project manager role assigned, but whomever has to oversee the progress of the initiative will benefit from the project management perspective that will be discussed here. For large initiatives, especially those with multiple and recurring monitoring deployments, you should considering moving to an APM-specific project manager, along the lines of what was outlined in Chapter 4 – Planning – Staffing and Responsibilities. First, a word of caution. Implementation is often used interchangeably with deployment. Deployment of software does not include getting the software to do what you want. This is the largest point of confusion and dissatisfaction with performance monitoring software—IT thought their responsibility was to deploy the monitoring solution while the business expected an implementation that would help them identify performance problems. Part of this confusion has to do with who “owns” the software. IT deploys lots of applications on behalf of the application owners, why is APM different? Often, an initial APM deployment is limited to a single stakeholder and the reality of APM being more of an infrastructure choice is lost. Here we mean implementation to cover the initial physical deployment, developing essential processes for employing the tool and how these processes are assembled into service capabilities. These processes are the difference in establishing a meaningful APM discipline. Implementation of an APM solution is surprisingly quick, compared to other IT initiatives: usually on the order of weeks. Sometimes in just a day or two for a small environment. The overall coordination however can be large, depending on the scope. Such coordination is required for any re-testing of applications and also for integration of alerts with existing Trouble Management systems. The scope (level of effort) will increase proportionally to the number of applications. For 1-2 applications, all of these implementation considerations can be discussed in a 1-2 hours planning session. For 20-30 applications, this might take 2-3 months before consensus and approval of the plan is achieved. xvii This section will review each of the topics that the implementation plan needs to consider. Traditionally, this task is managed by a project manager who will build the project schedule and then work to coordinate the resources to achieve the deployment. Smaller initiatives may want to forgo the project manager role, which incurs some risk (which we expect that an understanding of this section will help you to avoid). Even the smaller initiatives will benefit from the implementation considerations that are presented here. I also know that there will be a number of other goals and activities that the monitoring initiative needs to accomplish that are outside the traditional definition of deployment: getting the technology up and running. For the project manager, these are simply additional objectives which they will schedule and resource accordingly. You will need to provide the project manager with a model or template concerning how these non-traditional requirements will be achieved. I call this template the Phased Deployment Model. In the event that your APM initiative planning will involve multiple phased deployments leading to a dedicated APM team, then you will also want to consider using the service catalog approach to set goals and track the progress of your APM Service Bureau or Center of Excellence for the enterprise. At minimum you will want to document capabilities from the service catalog that you are not pursuing as part of you implementation. Often, a smaller initiative does not have the latitude to address these enterprise considerations but you can do a much better job in managing expectations if you fully understand and fully communicate to your stakeholders what you are setting aside. The bottom line for the APM implementation is to show you how to scale the available resources in advance of dedicated staff for APM, spread out the workload and focus on what is important for your organization. Introduction to Part 3: Practioner’s Guide This section introduces a variety of techniques for how to employ the tools that are important in realizing the best possible value from your APM investment. This has always been a controversial topic because the motivation for investing in monitoring software is that it “does stuff.”