Multiple Comparison Procedures Multiple Comparison Procedures YOSEF HOCHBERG Tel-Aviv University Ramat-Aviv, Israel AJIT C. TAMHANE Northwestern University Evanston, Illinois JOHN WILEY & SONS New York Chichester Brisbane Toronto Singapore 0 0 0 0 This book has electrofiically reproduced been h informatioa stared at Wiley & Sons, digital John h. We are pleased that the use of thrs new technology wili enable us to works of endurmg scholarly aGeep value in print BS long as there is a reasonable demand for them. The content of this book is identical to previous pnnhgs. Copyright 0 1987 by John Wiley & Sons, Inc. qll rights reserved. Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system or transmitted in my form 01- by any means, electronic, nlechanical, photucopying, reconling, scanning or othenvisc, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without cirher the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clwancc Center, 222 Rosewood Drivc, Danvcrs, MA01923, (978) 750-8400, fax (978j 750-4470. Requests to the Publisher for permission should be addressed io the Permisbions Department. John Wiley & Sons. Inc., 1 I1 River Street, Hoboken. NJ 07030. (201) 738-601 I, fax (201) 748.608. E-Mail: [email protected]. To order or customer service please. call l(800bCALL-WILEY (225-5945). books for Library of Congm~C atslogiag PuMkation Data: in Hochberg, Yosef. Multiple comparison procedures. (Wiley series in probability and mathematical statistics. Applied probability and statistics, ISSN 0271.6356) Bibliography: p. Includes index. 1. Muliple comparisons {Statistics) 1. Tamhane, Ajit C. 11. Title. 111. Series. OA278.4.H63 1987 519.5'35 87-6226 0-471-82222-1 ISBN To our parents Preface The foundations of the subject of multiple comparisons were laid in the late 1940s and early 1950s, principaIly by David Duncan, Roy, S.N. Henry Scheffi, and John Tukey, although some of the ideas appeared much earlier in the works of Fisher, Student, and others; see Harter 1980 for a complete historical account. Tukey’s (1953) mimeographed notes on the subject form a rich source of results (some of which are being rediscovered even today), but unfortunately they had only a limited circulation. Miller’s (1966) book helped to popularize the use of multiple comparison procedures (MCPs) and provided an impetus to new research in the field. There have been a large number of review articles surveying the field, most published during the 1970s (e.g., Aitkin 1969, Chew 1976a, Dunnett 1970, Dunnett and Goldsmith 1981, Games 1971, Gill 1973, Miller 1977, 1985, O’Neill and Wetherill 1971, Ryan 1959, Shaffer 1986b, Spj~tvoll1 974, and Thomas 1973). Commonly used MCPs are discussed in many elementary and advanced texts. However, no unified comprehensive treatment of the subject that incorporates the develop- ments in the last two decades and that plays a role similar to Miller’s monograph is available. The aim of the present is to fulfill this need. book Because MCPs have been around for quite some time and because their use is well accepted (and sometimes even required) in many disciplines, one might expect that there would be few, if any, unresolved questions and controversies. Such is not the case, however. Considerable confusion still exists in regard to what the different MCPs provide, which ones should be used, and when. There are many statisticians who even question the very need and appropriateness of any multiple comparison approach (see, e.g., the discussion following O’Neill and Wetherill 1971, and the papers by Carmer and Walker 1982, Dawkins 1983, Little 1978, O’Brien 1983, Perry 1986, and Petersen 1977). vii vili PREFACE Some of the controversial issues cut across the general subject of statistical inference and are not germane specifically to multiple com- parisons. These include testing versus confidence estimation, and the choice of approach to inference (e.g., a classical Fisherian or a Neyman- Pearsonian approach, a decision-theoretic approach, a Bayesian ap- proach, or an informal graphical approach). We prefer not to take rigid stands on these issues. Certainly applications can be found for each basic approach to inference, although some types of applications are more common than others. We have tried to present a variety of techniques based on different approaches, spelling out the pros and cons in each case. The emphasis, however, is on classical approaches and on confi- dence estimation. Another line of criticism stems from the misuse of MCPs in practice. A common example of this is the application of an MCP (a popular choice being Duncan’s 1955 stepwise procedure) for making pairwise com- parisons among all treatments when the treatments have a certain struc- ture, as is the case when they correspond to multifactorial combinations or to increasing levels of some quantitative factor (Chew 1976b). In such situations, comparisons other than pairwise comparisons may be of interest; for example, orthogonal contrasts, and a specially tailored MCP for such a family of comparisons may be required. Sometimes procedures other than MCPs are required, for example, when the researcher’s goal is to select the “best” treatment or to explore the possible clustering patterns among the treatments. The key point is that statistical proce- dures that should be used in a given problem depend on the questions of interest and the nature of research. Problems of inference encountered in empirical research are of diverse nature and indiscriminate use of MCPs or, for that matter, any other statistical technique in all problems is clearly inappropriate. In Chapter 1 we have addressed the questions of when and why to use MCPs. This discussion should also help to answer other basic criticisms voiced against multiple comparisons. The subject of multiple comparisons forms a part of the broader subject of simultaneous statistical inference. In this book we focus on problems involving multiplicity and selection (“data-snooping”) of infer- ences when comparing treatments based on univariate responses. (Refer to Krishnaiah, Mudholkar, and Subbaiah 1980 and Krishnaiah and Reis- ing 1985 for the corresponding multivariate techniques.) Roy’s union- intersection method forms the unifying theme used to derive the various classical MCPs for these problems. We do not discuss the problems of simultaneous point estimation and simultaneous confidence bands in regression. We also do not discuss the related topic of ranking and PREFACE ix selection procedures, on which there are full-length books by Gibbons, Olkin, and Sobel (1977) and Gupta and Panchapakesan (1979). The following is a brief outline of the book. In Chapter 1 we e!aborate on our philosophy and approach to multiple comparison problems, and discuss the basic notions of families, error rates, and control of error rates. The remainder of the book is divided into two parts. Part I, consisting of Chapters 2-6, deals with MCPs based on a classical error rate control approach for fixed-effects linear models under the usual normal theory assumptions. Part 11, consisting of Chapters 7-11, deals with MCPs for other models and problems, and MCPs based on alterna- tive approaches (e.g., decision-theoretic and Bayesian). Pere are three appendixes. Appendix 1 gives some general theory of MCPs that is not restricted to the setting of Part I (for which case the corresponding theory is given in Chapter 2). Appendix 2 reviews the probability inequalities that are used in deriving and computing conservative critical points for various MCPs. Finally, Appendix 3 discusses the probability distributions that arise in the context of some classical MCPs. Tables of the percentage points of these distributions are also included in this appendix. Sections are numbered fresh starting in each chapter. Subsections, equations, examples, figures, and tables are numbered fresh starting in each section but with the corresponding section identification. In both cases there is no chapter identification provided. Thus, for example, within a given chapter, equation (3.10) is the tenth equation in Section 3 of that chapter. When the same equation is referenced in another chapter, it is referred to as equation (3.10) of such and such chapter. This book is intended to serve the needs of researchers as well as practitioners. The comprehensive review of the published literature until 1986, as well as the many open problems noted throughout the book, should prove valuable to researchers. (It should be remarked, however, that the literature on multiple comparisons is vast and it is impossible to reference each and every publication. We have limited our references to those publications that are relevant to the discussions of the topics covered in the book.) Practitioners should find it helpful to have the steps involved in implementing the various MCPs clearly spelled out and illustrated by small numerical examples. Many of the examples are taken from the original papers where the corresponding MCPs first appeared. As mentioned before, the examples and discussion of Chapter 1 provide useful guidelines to the perennial problem of when to use MCPs; a need for such case studies was noted by Anscombe (1985). We have assumed that the reader has had a course in mathematical statistics covering the basic concepts of inference and is familiar with X PREFACE matrix algebra, linear models, and experimental designs. For the benefit of those who may not have had training in the last two topics, we have included a brief review of the necessary ideas in Section 1 of Chapter 2. This can be used as a text in a special topics course on multiple book comparisons or for supplementary reading in a course on linear models and analysis of variance. In conclusion, we hope that this book will meet its stated objectives. In addition, we hope that it will help to dispel some of the confusion and controversy that have surrounded the subject of MCPs, and encourage their correct use when appropriate in practice. YOSEFH WHBERG AJITC . TAMHANE Tel Aviv, Israel Evanston, Illinois Acknowledements We would like to gratefully acknowledge the facilities provided principal- ly by Northwestern University and also by Tel-Aviv and New York Universities, which made the work on this book possible. Early drafts of some of the chapters were written when the second author visited Cornell University in 1982-1983 on sabbatical leave with partial support from the U.S. Army Research Durham Contract DAAG-29-81-K-0168. Office, We are also indebted to many individuals who contributed in different ways to this book. Bob Berk, Morton Brown, Tony Hayter, Jason Hsu, Tom Santner, Juliet Shaffer, Gary Simon, and Yung Liang Tong read sections of the manuscript and provided comments and suggestions. G.V.S. Gopal read the entire manuscript and noted several errors. Charlie Dunnett computed Tables in Appendix 3 as well as some special 4-7 percentage points of the multivariate ?-distribution required in Examples 2.1 and 2.3 of Chapter 5. Bob Bechhofer and Ruben Gabriel gave timely advice and encouragement. Beatrice Shube, the senior editor of the statistics series at Wiley, was always patient and cooperative with us. The other staff at Wiley were also very helpful and efficient in carrying out a speedy and quality production of the book. Particular mention should be made of Margaret Comaskey, the editorial supervisor and Diana Cisek, the senior production supervisor. Finally, it is our pleasure to acknow- ledge the assistance of June Wayne, Carol Surma and Marianne Delaney of Northwestern University. June cheerfully typed and retyped numerous drafts of the manuscript (which required a considerable amount of patience and perseverance in spite of all the modern word processor features). Carol graciously pitched in whenever June was away. Marianne typed the author and subject indexes. We wish to express our most sincere thanks to them. xi