Springer Texts in Statistics Series Editors: G. Casella S. Fienberg I. Olkin Springer Texts in Statistics Athreya/Lahiri: Measure Theory and Probability Theory Bilodeau/Brenner: Theory of Multivariate Statistics Brockwell/Davis: An Introduction to Time Series and Forecasting Carmona: Statistical Analysis of Financial Data in S-PLUS Casella: Statistical Design Chow/Teicher: Probability Theory: Independence, Interchangeability, Martingales, Third Edition Christensen: Advanced Linear Modeling: Multivariate, Time Series, and Spatial Data; Nonparametric Regression and Response Surface Maximization, Second Edition Christensen: Log-Linear Models and Logistic Regression, Second Edition Christensen: Plane Answers to Complex Questions: The Theory of Linear Models, Second Edition Cryer/Chan: Time Series Analysis, Second Edition DasGupta: Asymptotic Theory of Statistics and Probability Davis: Statistical Methods for the Analysis of Repeated Measurements Dean/Voss: Design and Analysis of Experiments Dekking/Kraaikamp/Lopuhaä/Meester: A Modern Introduction to Probability and Statistics Durrett: Essentials of Stochastic Processes Edwards: Introduction to Graphical Modeling, Second Edition Everitt: An R and S-PLUS Companion to Multivariate Analysis Gentle: Matrix Algebra: Theory, Computations, and Applications in Statistics Ghosh/Delampady/Samanta: An Introduction to Bayesian Analysis Gut: Probability: A Graduate Course Heiberger/Holland: Statistical Analysis and Data Display; An Intermediate Course with Examples in S-PLUS, R, and SAS Jobson: Applied Multivariate Data Analysis, Volume I: Regression and Experimental Design Jobson: Applied Multivariate Data Analysis, Volume II: Categorical and Multivariate Methods Karr: Probability Kulkarni: Modeling, Analysis, Design, and Control of Stochastic Systems Lange: Applied Probability Lange: Optimization Lehmann: Elements of Large Sample Theory Lehmann/Romano: Testing Statistical Hypotheses, Third Edition Lehmann/Casella: Theory of Point Estimation, Second Edition Longford: Studying Human Populations: An Advanced Course in Statistics Marin/Robert: Bayesian Core: A Practical Approach to Computational Bayesian Statistics Nolan/Speed: Stat Labs: Mathematical Statistics Through Applications Pitman: Probability Rawlings/Pantula/Dickey: Applied Regression Analysis (continued after index) Anirban DasGupta Asymptotic Theory of Statistics and Probability Anirban DasGupta Purdue University Department of Statistics 150 North University Street West Lafayette, IN 47907 [email protected] Editorial Board George Casella Stephen Fienberg Ingram Olkin Department of Statistics Department of Statistics Department of Statistics University of Florida Carnegie Mellon University Stanford University Gainesville, FL 32611-8545 Pittsburgh, PA 15213-3890 Stanford, CA 94305 USA USA USA ISBN 978-0-387-75970-8 e-ISBN 978-0-387-75971-5 DOI: 10.1007/978-0-387-75971-5 Library of Congress Control Number: 2008921241 © 2008 Springer Science+Business Media, LLC All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY, 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identifi ed as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper. 9 8 7 6 5 4 3 2 1 springer.com To mymother,andtothelovingmemories ofmyfather Preface This book developed out of my year-long course on asymptotic theory at PurdueUniversity.Tosomeextent,the topicscoincidewith whatIcoverin that course. There are already a number of well-known books on asymp- totics.This bookis quite different.It coversmoretopicsin onesourcethan areavailableinanyothersinglebookonasymptotictheory.Numeroustopics covered in this book are available in the literature in a scattered manner, andthey are broughttogetherunderone umbrella in this book.Asymptotic theory is a central unifying theme in probability and statistics. My main goalinwriting thisbookistogiveitsreadersafeelfortheincrediblescope and reach of asymptotics. I have tried to write this book in a way that is accessible and to make the reader appreciate the beauty of theory and the insightsthatonlytheorycanprovide. Essentiallyeverytheoremin the bookcomeswith atleastonereference, precedingorfollowingthestatementofthetheorem.Inaddition,Ihavepro- videdaseparatetheorem-by-theoremreferenceasanentryonitsowninthe front of the book to make it extremely convenient for the reader to find a proof thatwas not providedin the text. Also particularly worth mentioning is a collection of nearly 300 practically useful inequalities that I have col- lected together from numerous sources. This is appended at the very end of the book. Almost every inequality in this collection comes with at least one reference. I have often preferred to cite a book rather than an original publication for these inequalities, particularly if the book contained many of the inequalities that I present. I also emphasize in this book conceptual discussionofissues,workingoutmanyexamplesandprovidingagoodcol- lectionofunusualexercises.Anotherfeatureofthisbookistheguidanceto theliteratureforsomeonewhowishestodigdeeperintothetopicofapartic- ularchapter.Ihavetriedtomakethechapter-by-chapterbibliographiesboth modernandrepresentative.Thebookhas574exercisesand293worked-out examples.Ihavemarkedthemorenonroutineexerciseswithanasterisk. vii viii Preface Ihopethatthisbookisusefulasagraduatetext,forindependentreading, and as a general and nearly encyclopedicresearch reference on asymptotic theory.Itshouldbepossibletodesigngraduate-levelcoursesusingthisbook withemphasisonparametricmethodsornonparametricmethods,onclassic topics or more current topics, on frequentist topics or Bayesian topics, or evenonprobabilitytheory.Forthebenefitofinstructors,Ihaveprovidedrec- ommended chapters for ten different one-semester courses, with emphasis on different themes. I hope that this provides some useful guidancetoward designingcoursesbasedonthisbook. Because the book covers a very broad range of topics, I do not have a uniform answer for what background I assume for a reader of this book. For most chapters, a knowledge of advancedcalculus and linear algebra is enoughtoenablethereadertofollowthematerial.However,somechapters require someuseof measuretheory andadvancedanalysisandsomeexpo- sure to stochastic processes.One course on statistical theory at the level of BickelandDoksum(citedinChapter3ofthisvolume)orCasellaandBerger (1990) and one on probability at the level of Hoel, Port, and Stone (1971) orDurrett(1994)arecertainlyneededtofollowthediscussioninthisbook. Chapter 1 is essentially a review of somewhat more advanced probability should one need it. The more advanced chapters in this book can be much betterappreciatedifonehashadcoursesonthetwobooksofErichLehmann (Lehmann and Casella (cited in Chapter 16), Lehmann and Romano (cited in Chapter 24)) and a course based on Breiman (1992), Durrett (2004) or Billingsley(seeChapter1). MygreatestthanksareduetoPeterHallforbeinganinspirationalandcar- ingadvisor,reader,andintellectualfilteroverthelastseveralyearsasIwas writing drafts of this book. Peter has deeply influenced my understanding, appreciation,andtaste forprobability andstatistics, andIhavefelt grateful thatIhavehadaccesstohimatalltimesandwithunlimitedpatience.Ihave received much more from Peter than I could wish or expect. I could not have written this book without Peter’s exemplary warmth and mentorship. However, all mistakes and ambiguities in the book are exclusively my re- sponsibility.Iwouldlovetoknowofallseriousmistakesthatareaderfinds inthisbook,andtheremustbemistakesinabookofthislength. I alsowant to expressmy very specialthanksto JohnMarden and Larry Wassermanforrepeatedlyofferingtheirfriendlyandthoughtfulsuggestions onvariousdecisionsIhadtomakeonthisbook.Iwanttomentionthegener- oushelpandsupportfromErichLehmann,PeterBickel,RabiBhattacharya, andJonWellneronspecificchaptersinthebook.Numerouscolleagues,and inparticularC.R.Rao,ArupBose,PersiDiaconis,JoeEaton,JianqingFan, Iain Johnstone, T. Krishnan, Bruce Lindsay, Wei-Liem Loh, Peter McCul- lagh,Dimitris Politis, B.V.Rao,BobSerfling,J.Sethuraman,KesarSingh, Preface ix and Michael Woodroofe, made helpful comments on parts of earlier drafts of the book. Chun Han, Martina Muehlbach, and Surya Tokdar helped me graciouslywithputtingtogetherTeXfilesofthechapers.JohnKimmeland JeffreyTaubatSpringerwereextremelyhelpfulandprofessional,andIen- joyed working with them very much. I will work with Johnand Jeff at any timewithpleasure.Severalanonymousrefereesdidunbelievablyhelpfuland constructivereadingsofmanypartsofthebook.TheSpringerserieseditors gavemegraciousinputwheneverneeded.ThecopyeditorHalHengleinand the typesetters – Integra India of Springer did a magnificent job. I am im- menselythankfultoallofthem.IamalsothankfultoPurdueUniversityfor providing me with computing and secretarial assistance. Doug and Cheryl Crabill,inparticular,assistedmenumeroustimeswithasmile. I was an impressionable fifteen-year-old when I entered the Indian Sta- tisticalInstitute (ISI) asafirst-yearstudent.Ihadheardthatstatisticiansdo boringcalculationswith large numbersusingclumsycalculatingmachines. Dev Basu entered the lecture room on my first day at the ISI and instantly changed my perception of statistics. No one I met could explain so effort- lessly the study of randomness and how to use what we learn about it to make useful conclusions. There was not one person at the ISI who didn’t regardhimasanincomparablerolemodel,apersonificationofscholarship, and an angelic personality. I am fortunate that I had him as my foremost teacher. I am grateful to C. R. Rao for the golden days of the ISI and for makingallofusfeelthatevenasstudentswewereequalsinhiseyes. At a personal level, I am profoundly grateful to Jenifer Brown for the uniquenessandconstancyofhertreasuredsupport,counsel,wellwishes,and endearingcamaraderieformanyyears,allofwhichhaveenrichedmeatmy mostdifficulttimesandhavehelpedmebecomeabetterhumanbeing.Iwill always remain much indebted to Jenifer for the positive, encouraging, and crystallizinginfluenceshehasbeenatalltimes.IhaveconsideredJeniferto beanimpeccablerolemodel. I am also thankful to Supriyo and Anuradha Datta, Julie Marshburn, Teena Seele, Gail Hytner, Norma Lucas, Deb Green, Tanya Winder, Hira Koul,RajeevaKarandikar,Wei-LiemLoh,DimitrisPolitis,andLarryShepp for their loyalty, friendship and warmth. Jim and Ann Berger, Herman Ru- bin, B.V. Rao, T. Krishnan, Larry Brown, Len Haff, Jianqing Fan, and Bill Strawderman have mentored, supported and cared for me for more than a decade. I appreciate all of them. But most of all, I appreciate the love and warmthofmyfamily.Idedicatethisbooktothecherishedmemoriesofmy father,andtomymotheronhereightiethbirthday. AnirbanDasGupta PurdueUniversity,WestLafayette,IN Recommended Chapter Selections CourseType Chapters SemesterI,Classical 1,2,3,4,7,8,11,13,15,17,21,26,27 Asymptotics SemesterII,Classical 9,14,16,22,24,25,28,29,30,31,32 Asymptotics SemesterI,Inference 1,2,3,4,7,14,16,17,19,20,21,26,27 SemesterII,Inference 8,11,12,13,22,24,25,29,30,32,33,34 Semester I, Emphasis on 1,2,3,4,5,6,8,9,10,11,12,23 Probability Semester I, Contemporary 1,2,3,8,10,12,14,29,30,32,33,34 Topics SemesterI, 1,3,5,7,11,13,15,18,24,26,29,30,32 Nonparametrics Semester I, Modeling and 1,3,4,8,9,10,16,19,26,27,29,32,33 DataAnalysis MyUsualCourse, 1,2,3,4,6,7,8,11,13,14,15,16,20 SemesterI MyUsualCourse, 5,9,12,17,21,22,24,26,28,29,30,32,34 SemesterII xi Key Theorems and References TheoremNumber ASourceforProof 1.2 Breiman(1968,1992) 1.4 Billingsley(1995) 1.5 Ferguson(1996) 1.6 Breiman(1968,1992) 1.7 Ash(1973),Billingsley(1995) 1.11 Breiman(1968,1992) 1.13 Billingsley(1995) 1.15 Breiman(1968,1992) 2.1 Reiss(1989) 2.2 Reiss(1989),Billingsley(1995) 2.7 Johnson(2004) 2.12 LeCam(1960) 3.1 Feller(1966),Ferguson(1996) 3.4 Breiman(1968,1992) 3.5 Breiman(1968,1992) 3.6 Provedinthetext 3.9 BickelandDoksum(2001) 4.1,4.3 Brown,Cai,andDasGupta(2006) 5.1 Billingsley(1995) 5.9 Anscombe(1952) 5.10,5.11 Breiman(1968,1992),Ash(1973) 5.13,5.16,5.18,5.19 Breiman(1968,1992) 6.1,6.2 Billingsley(1995) 6.6 Shiryaev(1980) 7.1,7.2,7.3 Serfling(1980) 7.5 KoenkerandBassett(1978) 8.4,8.5,8.6 Galambos(1977) 8.12,8.13,8.14 Galambos(1977) 8.16,8.17 Leadbetteretal.(1983) xiii