The RAND/UCLA Appropriateness Method User’s Manual Kathryn Fitch Steven J. Bernstein María Dolores Aguilar Bernard Burnand Juan Ramón LaCalle Pablo Lázaro Mirjam van het Loo Joseph McDonnell John Paul Vader James P. Kahan Prepared for Directorate General XII, European Commission R R Europe Health The research described in this report was prepared for Directorate General XII, European Commission. Library of Congress Cataloging-in-Publication Data The Rand/UCLA appropriateness method user’s manual / Kathryn Fitch ... [et al.]. p. cm. “MR-1269-DG-XII/RE.” Includes bibliographical references. ISBN 0-8330-2918-5 1. Medical care—Decision making. 2. Medical care—Utilization review. 3. Clinical indications. 4. Delphi method. I. Fitch, Kathryn. R723.5 .R36 2000 362.1—dc21 00-045830 RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND® is a registered trademark. RAND’s pub- lications do not necessarily reflect the opinions or policies of its research sponsors. © Copyright 2001 RAND All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from RAND. Published 2001 by RAND 1700 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 RAND URL: http://www.rand.org To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Internet: [email protected] page iii PREFACE The concepts of appropriate and necessary care are fundamental to the creation of an efficient and equitable health-care delivery system. Evidence of inappropriate overuse and underuse of procedures has been documented even in health systems characterised by the absence of global budgets, capitation, utilisation review or the pressure of requiring a second opinion. Health systems should function in such a way that inappropriate care is progressively reduced, while appropriate and especially necessary care are maintained or increased. The ability to determine and identify which care is overused and which is underused is essential to this functioning. To this end, the "RAND/UCLA Appropriateness Method" (here given the acronym RAM) was developed by RAND and UCLA in the 1980s. It has been further developed and refined in North America and, increasingly, in Europe. This manual presents step-by-step guidelines for conceptualising, designing, and carrying out a study of the appropriateness of medical or surgical procedures (for either diagnosis or treatment) using the RAM. The manual distills the experience of many researchers in North America and Europe, and presents current (as of the year 2000) thinking on the subject. Although the manual is self-contained and as complete as we could make it, the authors do not recommend that someone unfamiliar with the RAM independently conduct an appropriateness study; instead, we strongly advocate a strategy of "seeing one" before "doing one." To this end, the authors of this manual, as well as their collaborators, stand ready to assist potential users of the method. Interested parties should contact one of the authors of this manual (see Annex I for addresses) or, more generally, RAND Health in Santa Monica, California, USA or RAND Europe in Leiden, the Netherlands. The writing and editing of this manual was carried out as part of the Concerted Action "A method to integrate scientific and clinical knowledge to achieve the appropriate utilisation of major medical and surgical procedures," financed by Directorate General XII of the European Commission under the BIOMED II programme (contract no. BMH4-CT96-0212), during the period 1996-1999. page iv RAND Health is the health policy unit of RAND Europe is the European RAND. For further information about operating unit of RAND. For further RAND Health, contact its director, information about RAND Europe, Robert H. Brook. contact its President, David C. Gompert. Robert H. Brook, MD, ScD David C. Gompert RAND Health RAND Europe 1700 Main Street Newtonweg 1 P.O. Box 2138 2333 CP Leiden Santa Monica, CA 90407-2138 THE NETHERLANDS USA telephone: +31-71-524-5151 telephone: +1-310-393-0411 telefax: +31-71-524-5191 telefax: +1-310-393-4818 e-mail: [email protected] e-mail: [email protected] page v CONTENTS PREFACE............................................................................................................ iii ACKNOWLEDGMENTS..................................................................................... xiii CHAPTER 1 INTRODUCTION......................................................................................... 1 Background..................................................................................... 1 Purpose of this Manual................................................................... 3 An Overview of the Method............................................................ 3 Comparison with Other Group Judgement Methods..................... 6 CHAPTER 2 SELECTING A TOPIC................................................................................. 8 CHAPTER 3 REVIEW AND SYNTHESIS OF THE LITERATURE................................ 10 Purpose............................................................................................ 10 Search Methodology........................................................................ 11 Selection and Classification of Articles.......................................... 11 Synthesising the Evidence.............................................................. 12 Resources Required......................................................................... 13 CHAPTER 4 DEVELOPING THE LIST OF INDICATIONS AND DEFINITIONS........ 14 Characteristics of the List of Indications....................................... 14 Structure of the List of Indications................................................ 15 Modifying the List of Indications (Before the Panel Meeting)...... 19 Definitions of Terms....................................................................... 20 CHAPTER 5 THE EXPERT PANEL................................................................................. 23 Panel Composition.......................................................................... 23 Panel Nominations.......................................................................... 23 page vi How Many Members?..................................................................... 24 Motivation to Participate................................................................ 25 Contacts with Panellists................................................................. 25 CHAPTER 6 THE RATING PROCESS: APPROPRIATENESS AND NECESSITY....... 30 Introduction..................................................................................... 30 Rating Appropriateness: First Round........................................... 31 Rating Appropriateness: Second Round........................................ 33 Documents Required....................................................................... 35 Automating the Rating Process...................................................... 39 Other Materials and Resources for the Panel Meeting ................ 39 Panel Observers.............................................................................. 42 Rating Necessity: Third Round ..................................................... 43 CHAPTER 7 CHAIRING AN EXPERT PANEL................................................................ 46 The Moderator................................................................................. 46 Preparing for the Meeting.............................................................. 46 At the Meeting................................................................................ 49 Modifying the List of Indications (During the Panel Meeting)..... 51 Guiding the Discussion................................................................... 53 CHAPTER 8 CLASSIFYING APPROPRIATENESS........................................................ 56 Introduction..................................................................................... 56 Operational Definitions of Levels of Appropriateness................... 56 Operational Definitions of Levels of Disagreement....................... 56 A New Approach to Measuring Disagreement............................... 59 Development of the "IPRAS".......................................................... 60 CHAPTER 9 SOFTWARE TOOLS FOR DATA PROCESSING AND ANALYSIS .......... 64 Introduction..................................................................................... 64 RAND, Santa Monica, California................................................... 65 page vii Erasmus University, Rotterdam.................................................... 65 Carlos III Health Institute Madrid................................................ 66 Institute of Social and Preventive Medicine, University of Lausanne......................................................................................... 66 CHAPTER 10 APPLYING APPROPRIATENESS CRITERIA RETROSPECTIVELY TO MEASURE OVERUSE........................................................................... 68 Introduction..................................................................................... 68 Study Objectives ............................................................................. 70 Construction of the Sampling Frame............................................. 71 Preliminary Organisation of Field Work........................................ 73 Draft Data Collection Form and Instruction Manual.................... 76 Data Analysis Plan......................................................................... 77 Data Processing Program............................................................... 77 Pilot Study....................................................................................... 78 Final Sample Design and Selection................................................ 80 Final Field Work Organisation....................................................... 80 Final Data Collection Form and Instruction Manual.................... 81 Field Work....................................................................................... 81 Data Processing............................................................................... 81 Data Analysis.................................................................................. 82 Conclusions/Publications................................................................ 82 CHAPTER 11 APPLYING NECESSITY CRITERIA TO MEASUREUNDERUSE........... 84 Introduction..................................................................................... 84 Necessity Criteria........................................................................... 84 Defining the Target Population...................................................... 85 Funding........................................................................................... 86 Conclusions..................................................................................... 87 page viii CHAPTER 12 APPLYING APPROPRIATENESS CRITERIA PROSPECTIVELY TO ASSIST IN DECISION-MAKING.......................................................... 88 Problems with Retrospective Studies............................................. 88 Obstacles to Prospective Use.......................................................... 89 An Algorithmic Approach............................................................... 90 A WWW-Based Approach............................................................... 91 A CD-ROM Approach...................................................................... 94 CHAPTER 13 METHODOLOGICAL ISSUES.................................................................... 97 Reliability and Validity................................................................... 97 Resolving Inconsistencies............................................................... 98 Obsolescence.................................................................................... 100 Rigidity............................................................................................ 100 Dissemination................................................................................. 101 REFERENCES.................................................................................................... 103 ANNEXES Annex I Members of the European Union BIOMED Concerted Action on "A Method to Integrate Scientific and Clinical Knowledge to Achieve the Appropriate Utilisation of Major Medical and Surgical Procedures".................................................. 107 Annex II List of Acronyms............................................................................. 109 page ix FIGURES Figure 1 The RAND/UCLA Appropriateness Method.............................. 4 Figure 2 Variables in List of Indications for Rating the Appropriateness of Coronary Revascularization. Spanish Panel, December 1996.................................................. 16 Figure 3 Sample Indications List from Spanish Coronary Revascularization Panel, December 1996.................................. 17 Figure 4 Sample Indications List from European Gastrointestinal Endoscopy Panel, November 1998............................................. 18 Figure 5 Sample Definitions from European Gastrointestinal Endoscopy Panel, November 1998............................................. 21 Figure 6 Sample Definitions from Swiss Back Surgery Panel, 1995....... 22 Figure 7 Sample Letter to Panellists........................................................ 27 Figure 8 Checklist for Panellist Contacts................................................. 29 Figure 9 Sample First Round Rating Sheet, European Panel on Coronary Revascularization, December 1998............ 34 Figure 10 Sample Second Round Rating Sheet, European Panel on Coronary Revascularization, December 1998............................ 37 Figure 11 Sample Post-Panel Questionnaire............................................. 41 Figure 12 Checklist of Materials for Second Round Ratings..................... 42 Figure 13 Sample Form for Rating Necessity, European Gastrointestinal Endoscopy Panel, November 1998................. 45 Figure 14 Sample Colour-Coded Moderator Sheet from a 15-Member European Coronary Revascularization Panel............................ 48 Figure 15 Eliminating a Category from the List of Indications................ 51 Figure 16 Adding a Category to the List of Indications............................. 52 Figure 17 Overuse of Procedures................................................................ 69 Figure 18 Design of a Retrospective Study to Measure Appropriateness.......................................................................... 70 Figure 19 Transformation of Variables...................................................... 78 Figure 20 Underuse of Procedures............................................................. 85 page x Figure 21 Examples of Proportion of Procedures Studied that Were Inappropriate or Uncertain........................................................ 89 Figure 22 Hysterectomy for Premenopausal Abnormal Uterine Bleeding of Unknown Aetiology................................................. 92 Figure 23 Sample Page from WWW Instrument for Prospective Evaluation of the Appropriateness of Low-Back Surgery......... 93 Figure 24 Partial View of Results Section of WWW Instrument Designed to Assist in the Evaluation of the Appropriateness of Low-Back Surgery.................................................................. 93 Figure 25 User Interface for Software Designed to Compare Treatment Choice for Benign Prostatic Hyperplasia with Recommendations of a Dutch Expert Panel.............................. 95