Systematic Reviews in the Social Sciences A PRACTICAL GUIDE Mark Petticrew and Helen Roberts Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 3 4.8.2005 1:07am Systematic Reviews in the Social Sciences Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 1 4.8.2005 1:07am Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 2 4.8.2005 1:07am Systematic Reviews in the Social Sciences A PRACTICAL GUIDE Mark Petticrew and Helen Roberts Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 3 4.8.2005 1:07am � 2006 by Mark Petticrew and Helen Roberts BLACKWELL PUBLISHING 350 Main Street, Malden, MA 02148-5020, USA 9600 Garsington Road, Oxford OX4 2DQ, UK 550 Swanston Street, Carlton, Victoria 3053, Australia The right of Mark Petticrew and Helen Roberts to be identified as the Authors of this Work has been asserted in accordance with the UK Copyright, Designs, and Patents Act 1988. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photo- copying, recording or otherwise, except as permitted by the UK Copyright, Designs, and Patents Act 1988, without the prior permission of the publisher. First published 2006 by Blackwell Publishing Ltd 1 2006 Library of Congress Cataloging-in-Publication Data Petticrew, Mark. Systematic reviews in the social sciences : a practical guide/Mark Petticrew and Helen Roberts. p. cm. Includes bibliographical references and index. ISBN-13: 978-1-4051-2110-1 (hard cover: alk. paper) ISBN-10: 1-4051-2110-6 (hard cover: alk. paper) ISBN-13: 978-1-4051-2111-8 (pbk. : alk. paper) ISBN-10: 1-4051-2111-4 (pbk. : alk. paper) 1. Social sciences—Research—Methodology. 2. Social sciences—Statistical methods. I. Roberts, Helen, 1949–II. Title. H62.P457 2005 300’.72’3—dc22 2005011632 A catalogue record for this title is available from the British Library. Set in 10.5pt/12.5pt Bembo by SPI Publisher Services, Pondicherry, India Printed and bound in the United Kingdom by TJ International, Padstow, Cornwall The publisher’s policy is to use permanent paper from mills that operate a sustainable forestry policy, and which has been manufactured from pulp processed using acid-free and elementary chlorine-free practices. Furthermore, the publisher ensures that the text paper and cover board used have met acceptable environmental accreditation standards. For further information on Blackwell Publishing, visit our website: www.blackwellpublishing.com Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 4 4.8.2005 1:07am Contents Foreword – William R. Shadish vi Acknowledgments x Preface xiii Chapter 1 Why do we need systematic reviews? 1 Chapter 2 Starting the review: Refining the question and defining the boundaries 27 Chapter 3 What sorts of studies do I include in the review? Deciding on the review’s inclusion/exclusion criteria 57 Chapter 4 How to find the studies: The literature search 79 Chapter 5 How to appraise the studies: An introduction to assessing study quality 125 Chapter 6 Synthesizing the evidence 164 Chapter 7 Exploring heterogeneity and publication bias 215 Chapter 8 Disseminating the review 247 Chapter 9 Systematic reviews: Urban myths and fairy tales 265 Glossary 277 Appendix 1 The review process (and some questions to ask before starting a review) 284 Appendix 2 MOOSE Guidelines 288 Appendix 3 Example of a flow diagram from a systematic review 291 Appendix 4 Example data extraction form 293 Appendix 5 Variations in the quality of systematic reviews 296 Bibliography 298 Index 324 Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_02_toc Final Proof page 5 4.8.2005 1:07am Foreword The idea of evidence-based practice is not new. For example, nearly 60 years ago, the scientist-practitioner model of clinical psychology in the US was based in part on the idea that clinical psychologists would scientifically investigate the effectiveness of their own treatments, and use what proved to be effective. Around 100 years ago, the British Army asked Sir Karl Pearson to review the evidence about the effects of a new typhoid vaccine, to help them decide whether to adopt that vaccine more widely.1 Such relatively isolated examples are common for hundreds of years. What is new is the widespread acceptance that evidence-based practice is practice whose time has come. Whether it be through the Cochrane Collaboration, the Camp- bell Collaboration, the What Works Clearinghouse in the US Department of Education, or any of a host of similar efforts, the idea that practices and policies that are supported by scientific evidence can be identified and disseminated has now been institutionalized. That in itself is a remarkable accomplishment. In this context, this book by Petticrew and Roberts is a timely gem, full of detailed history, pithy quotes, marvelous examples, compelling argu- ments, practical advice, and transparent demonstrations of how to do one of the most common ways of putting the evidence into evidence-based practice – the systematic review. Practical advice leaps out at the reader from every chapter, with novel information I have not seen in related works. Let me start with reference to the key justification for systematic reviews, that ‘‘single studies taken in isolation are often seriously misleading’’ (chapter 1, this volume). Or as I tend to put it: Never trust the results of single studies. One need only graph a frequency distribution of effect sizes to see that some studies will show unusually positive effects, and some un- usually negative effects, substantially by virtue of chance. Yet the public, the media, and even other scientists routinely place great faith in single studies. Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 6 4.8.2005 1:06am As a result, ineffective or inaccurate policies, practices, or ideas are adopted, accurate or effective ones are dismissed, and all parties get confused. The analogy that Petticrew and Roberts make with a survey is a good one. We would never take seriously the results of a sample survey where the sample was of a single person, and we all understand why – we will often get a different answer by asking a different person. The same is true in systematic reviews. We will often get a different answer when we look at a different study. In sample surveys, only a large sample is capable of giving us a reliable answer. In reviewing literature on a question, only the systematic review process is capable of helping to clarify where the answer really lies. Yet as Petticrew and Roberts point out in chapter 2 and elaborate in chapter 3, we have had an unnecessarily narrow view of what a systematic review can do – answer questions about cause and effect relationships. Systematic reviews need not only address what works, as important as that question is. Such reviews can address the evidence about nearly any kind of question, for example, about the prevalence of a problem, the correlation of two variables to each other, the implementation of a program, the meaning program stakeholders ascribe to a program or its context, the causal mediating processes that help us understand why it works, and program costs, to name just a few. Similarly, our understanding of the question of what works has often been too narrow. That question involves high quality evidence about cause and effect relationships, to be sure. But we are also informed about what works by, for example, knowledge that some stake- holder groups refuse to participate, that costs are too high to be practical, or that the need for the treatment is questionable. What is less clear is whether we have well-developed technologies for addressing some of these questions. The technological problem is twofold, how to integrate studies that are all of one kind (e.g., about need) with each other, and how to integrates studies of one kind (e.g., need) with studies of another kind (e.g., treatment effectiveness). The former problem is relat- ively unproblematic for most kinds of quantitative studies, but is far less clear for qualitative studies. The latter problem is almost completely unaddressed. It is this context that makes me so pleased to see the effort Petticrew and Roberts have put into both these problems. Many people talk about the value of integrating quantitative and qualitative studies, for example, but actual examples of such integration are few and far between. Petticrew and Roberts lay out the logic of such integrations, some of the methods that might be used for doing them, along with examples. And they don’t shy away from the difficulties such as a lack of standardized search terms to use in locating qualitative studies on a given topic. For many years, when critics have decried the narrow focus on experiments in systematic reviews of what FOREWORD vii Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 7 4.8.2005 1:06am works, I have responded by saying we are working on that problem. Now I can actually point to the work. I was also pleased to see the huge number of references in chapter 5 about assessing the quality of different kinds of research designs. Just the compila- tion itself is a major contribution. Yet we should remember that quality can mean two different things: (a) quality of the implementation of the method (e.g., did the primary study author implement the qualitative method properly); and (b) quality of the inferences from the method (do the methods used support a causal inference). My sense is that the latter kinds of quality scales are pretty common for experiments, though serious ques- tion remains about whether those scales are well enough developed to warrant widespread use. But inference quality scales are far less well devel- oped for qualitative methods, where the available scales seem more con- cerned with whether the methods were implemented well. The problem is that good implementation of a method is orthogonal to whether that method supports a given inference. The best-implemented randomized experiment is rarely going to tell us much about the need for an interven- tion, just as the best implemented ethnography is rarely going to yield a strong inference about cause-and-effect relationships. Finally, chapter 8 on dissemination and use of systematic reviews raises a crucial issue. Too often we assume that policymakers and practitioners are likely to adopt interventions that are shown to work. That assumption is at best only partly right. We learned that the hard way in the field of program evaluation in the 1960s and 1970s. The early emphasis in that field was on experimentation to determine effective interventions that could be adopted as policy. We quickly found out that they were frequently not adopted even when they were effective, and began to study the process of use itself.2 The fate of Fairweather’s Lodge for the chronically mentally ill is illustrative.3 That intervention was far more cost effective than the alternatives, was well- liked by patients and successful on most outcome criteria; but it was relatively little used despite extensive dissemination funding because it had no funding mechanism, conflicted with the extant treatment institutions, and required citizens to accept a house of patients next door. Adopting effective practices involves far more than knowing what works, and Petti- crew and Roberts do a commendable job of laying out the issues. The lessons learned are remarkably similar to those we learned in program evaluation years ago, and we would do well to revisit the modern versions of that literature (e.g., Patton)4 to help us understand more about how to make our work useful. As you can see, I am a fan of what Petticrew and Roberts have done in this book. They lay out a broad and ambitious agenda for the art and science viii FOREWORD Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 8 4.8.2005 1:06am of systematic reviews, an agenda that cogently presages issues that I think will be at the cutting edge of this field of inquiry in the coming decades. I hope you enjoy it as much as I did. William R. Shadish University of California, Merced REFERENCES 1. Pearson, K. Report on certain enteric fever inoculation statistics. British Medical Journal 1904, 3: 1243–6. 2. Shadish, W. R., Cook, T. D., and Leviton, L. C. Foundations of program evalu- ation: Theories of practice. Newbury Park, CA: Sage Publications, 1991. 3. Shadish, W. R. Policy research: Lessons from the implementation of deinstitu- tionalization. American Psychologist 1984, 39: 725–38. 4. Patton, M. Q. Utilization-focused evaluation (3rd edn.). Newbury Park, CA: Sage Publications, 1996. FOREWORD ix Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 9 4.8.2005 1:06am Acknowledgments Writing books incurs a large number of debts. This one is no different, and we would particularly like to thank Susan Kennedy and Rodney Barker, who have gone well beyond the usual demands of being our partners by reading, commenting, and constructively criticizing. Professor Will Shadish, one of the foremost meta-analysts in the USA readily and generously agreed to write a foreword. The anonymous peer reviewers of our proposal and, in particular, the reviewer of our final manuscript contributed a great deal to our thinking and the final product. At our request, Blackwell approached them to ask if they would forgo anonymity so that we could acknowledge them by name, and we are happy to say that they agreed to this. Many thanks then, to Trevor Sheldon, at the University of York, who nobly reviewed both the proposal and the book; and to the proposal reviewers Jim Connelly, University of Leeds, Emma Davies, Auckland University of Technology, David DuBois, Uni- versity of Illinois at Chicago, Loraine Gelsthorpe, University of Cambridge, David Gough, Institute of Education, University of London, Geraldine Macdonald, Commission for Social Care Inspection, Susan Michie, Uni- versity College London, Deborah Phillips, Georgetown University, and Maggie Savin-Baden, Coventry University. We are also grateful to Kate Pool from the Society of Authors. Our colleagues in Glasgow and City Universities, our co-researchers and colleagues on the ESRC-funded project on narrative synthesis: Jennie Popay, Amanda Sowden, Nicky Britten, and the late Sally Baldwin, Lisa Arai, Mark Rodgers, and Katrina Roen, and our colleagues in the ESRC Evidence Network, and the Cochrane and Campbell Collaborations (par- ticularly those whose work we have quoted) have all been parties to discussions and conversations that have helped inform our work. Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 10 4.8.2005 1:06am Mark Petticrew would particularly like to thank colleagues in the ESRC Centre for Evidence-Based Public Health Policy, and in his research pro- gram (funded by the Chief Scientist Office of the Scottish Executive Department of Health), whose work and thinking the book draws on: among them Pam Attree, Beth Milton, Clare Bambra, Frances Drever, Matt Egan, Hilary Graham, Val Hamilton, Sally Macintyre, Liz McDer- mott, David Morrison, David Ogilvie, Sian Thomas, Hilary Thomson, and Margaret Whitehead. Colleagues elsewhere who read and commented on parts of the book, or who kindly gave permission to use their work, include Iain Chalmers, Iain Crombie, the EPOC (Effective Practice and Organiza- tion of Care) group of the Cochrane Collaboration, Simon Gilbody, David Gough and colleagues at the EPPI Centre, Rebecca Ivers and colleagues, Betsy Kristjansson, Geoffrey Nelson and colleagues, Anthony Petrosino, Steve Platt, Scott Reeves, Roberta Scherer, Fujian Song, Peter Tugwell, and Paul Wilson. Thanks for permission to reproduce various figures or tables go to John Wiley & Sons Ltd (the Forest plots in chapters 6 and 7, which are copyright of the Cochrane Library), The Sainsbury Archive (Figure 6.2 in chapter 6), Hawaii Medical Library, the National Portrait Gallery (for the portrait of Francis Bacon in chapter 4), the British Medical Journal (for permission to reproduce material in chapter 9), David Ogilvie (the photograph in chapter 6), and the Centre for Reviews and Dissemin- ation (CRD) at the University of York for permission to reproduce material from their systematic reviews guidance (CRD Report 4). Thanks also to Mary Robins for library support. Helen Roberts is particularly grateful to colleagues at City University, especially current and past colleagues in the Child Health Research and Policy Unit, and to fellow members of the What Works for Children ‘‘node’’ funded as part of the ESRC evidence network – Trevor Sheldon at the University of York, Kristin Liabo, Madeleine Stevens, Greg Khine, and Alison Moore at City University and Di McNeish, Tony Newman, Sarah Frost, Angela Hutton, and Philippa Roberts at Barnardo’s. Shruti Uppal provided valuable assistance with reference and Web checking. Helen has learned a great deal from colleagues at the EPPI Centre at the University of London Institute of Education and others who contributed to the seminar series on research evidence organized by the Evidence Net- work, funded by the UK Health Development Agency. Other colleagues with whom Helen has worked on systematic reviews including Carol Joughin, Gabrielle Laing, Stuart Logan, Sandra Dowling, Catherine Law, Jos Kleijnen, Patricia Lucas, Janis Baird, and David Fisher have helped her thinking while working on joint systematic reviews. Ruth Gilbert, Ian Roberts, and Dennis Cheek all responded quickly and helpfully to requests ACKNOWLEDGMENTS xi Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 11 4.8.2005 1:06am for advice. Helen would also like to thank her former colleagues in the R&D team in Barnardo’s with whom she worked on some of the material on which we draw in chapter 8, and to ESRC and David Walker for allowing us to draw on their communications toolkit and related work. Thanks to Getpetsonline for permission to use the photograph in figure 2.2 and to the Thomas A. Edison papers archive at Rutgers University for permission to use the image in Box 4.5. Sarah Bird, Will Maddox, Joanna Pyke, Rhonda Pearce, Mervyn Tho- mas, Helen Kemp, and Zeb Korycinska from Blackwell looked after this book as it moved through publication. We thank them for their careful work. Mark Petticrew Helen Roberts xii ACKNOWLEDGMENTS Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 12 4.8.2005 1:06am Preface Research studies use a variety of methods, are of variable quality and may appear to have contradictory findings . . . Research information can seem to users like small jigsaw pieces in a box where there may be several pictures, several duplicates and several missing pieces. Under these circumstances it can be difficult for the researcher, research funder or user to make sense of the research or take stock of the knowledge base. (Sheldon, 1998)1 Literature reviews have many purposes. They can examine old theories and propose new ones, consider where the balance of evidence lies in relation to a particular topic, and provide a basis for recommendations for interven- tions, (such as teaching in smaller or larger classes, running an anti-bullying program, or offering parenting education). They can provide guidance to researchers planning future studies, and provide convenient summaries of the literature on a particular issue. They can also be used to examine methodically the reasons why different studies addressing the same question sometimes reach different conclusions – a common and frustrating problem for both researchers and research users. This book is a guide to planning and conducting a particular type of literature review, one that is increasingly used as a scientific tool: the systematic literature review. The book is aimed at social science researchers, but it provides a more general discussion of systematic reviews for those who want to use and understand them, but don’t necessarily want to do one themselves. Most of the recent interest in systematic reviews focuses on reviews of the effectiveness of interventions, reflected in the growth of initiatives like the Campbell and Cochrane Collaborations. This book therefore focuses on reviews of effectiveness, but not exclusively. Systematic reviews are also widely used to synthesize other sorts of evidence, for example in order to answer questions about etiology (causes of problems), Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 13 4.8.2005 1:06am