“Howandwhyiscomputationalstatisticstakingovertheworld?Inthisseriouswork ofsynthesisthatisalsofuntoread,EfronandHastie,twopioneersintheintegrationof parametricandnonparametricstatisticalideas,givetheirtakeontheunreasonableeffec- tivenessofstatisticsandmachinelearninginthecontextofaseriesofclear,historically informedexamples.” —AndrewGelman,ColumbiaUniversity “Thisunusualbookdescribesthenatureofstatisticsbydisplayingmultipleexamples ofthewaythefieldhasevolvedoverthepast60years,asithasadaptedtotherapid increaseinavailablecomputingpower.Theauthors’perspectiveissummarizednicely when they say, ‘Very roughly speaking, algorithms are what statisticians do, while inference says why they do them.’ The book explains this ‘why’; that is, it explains the purpose and progress of statistical research, through a close look at many major methods,methodstheauthorsthemselveshaveadvancedandstudiedatgreatlength. Bothenjoyableandenlightening,ComputerAgeStatisticalInferenceiswrittenespe- ciallyforthosewhowanttohearthebigideas,andseetheminstantiatedthroughthe essentialmathematicsthatdefinesstatisticalanalysis.Itmakesagreatsupplementto thetraditionalcurriculaforbeginninggraduatestudents.” —RobKass,CarnegieMellonUniversity “This is a terrific book. It gives a clear, accessible, and entertaining account of the interplaybetweentheoryandmethodologicaldevelopmentthathasdrivenstatisticsin thecomputerage.Theauthorssucceedbrilliantlyinlocatingcontemporaryalgorithmic methodologiesforanalysisof‘bigdata’withintheframeworkofestablishedstatistical theory.” —AlastairYoung,ImperialCollegeLondon “Thisisaguidedtourofmodernstatisticsthatemphasizestheconceptualandcompu- tationaladvancesofthelastcentury.Authoredbytwomastersofthefield,itoffersjust therightmixofmathematicalanalysisandinsightfulcommentary.” —HalVarian,Google “EfronandHastieguideusthroughthemazeofbreakthroughstatisticalmethodologies followingthecomputingevolution:whytheyweredeveloped,theirproperties,andhow theyareused.Highlightingtheirorigins,thebookhelpsusunderstandeachmethod’s roles in inference and/or prediction. The inference–prediction distinction maintained throughoutthebookisawelcomeandimportantnoveltyinthelandscapeofstatistics books.” —GalitShmueli,NationalTsingHuaUniversity “A masterful guide to how the inferential bases of classical statistics can provide a principleddisciplinaryframeforthedatascienceofthetwenty-firstcentury.” —StephenStigler,UniversityofChicago,authorof SevenPillarsofStatisticalWisdom “ComputerAgeStatisticalInferenceoffersarefreshingviewofmodernstatistics.Algo- rithmicsareputonequalfootingwithintuition,properties,andtheabstractarguments behindthem.Themethodscoveredareindispensabletopracticingstatisticalanalysts intoday’sbigdataandbigcomputinglandscape.” —RobertGramacy,TheUniversityofChicagoBoothSchoolofBusiness “Everyaspiringdatascientistshouldcarefullystudythisbook,useitasareference,and carry it with them everywhere. The presentation through the two-and-a-half-century historyofstatisticalinferenceprovidesinsightintothedevelopmentofthediscipline, puttingdatascienceinitshistoricalplace.” —MarkGirolami,ImperialCollegeLondon “Efron and Hastie are two immensely talented and accomplished scholars who have managedtobrilliantlyweavethefiberof250yearsofstatisticalinferenceintothemore recent historical mechanization of computing. This book provides the reader with a mid-leveloverviewofthelast60-someyearsbydetailingthenuancesofastatistical communitythat,historically,hasbeenself-segregatedintocampsofBayes,frequentist, andFisheryetinmorerecentyearshasbeenunifiedbyadvancesincomputing.Whatis lefttobeexploredistheemergenceof,androlethat,bigdatatheorywillhaveinbridg- ingthegapbetweendatascienceandstatisticalmethodology.Whatevertheoutcome, theauthorsprovideavisionofhigh-speedcomputinghavingtremendouspotentialto enablethecontributionsofstatisticalinferencetowardmethodologiesthataddressboth globalandsocietalissues.” —RebeccaDoerge,CarnegieMellonUniversity “Inthisbook,twomastersofmodernstatisticsgiveaninsightfultouroftheintertwined worldsofstatisticsandcomputation.Throughaseriesofimportanttopics,Efronand Hastieilluminatehowmodernmethodsforpredictingandunderstandingdataarerooted inbothstatisticalandcomputationalthinking.Theyshowhowtheriseofcomputational powerhastransformedtraditionalmethodsandquestions,andhowithaspointedusto newwaysofthinkingaboutstatistics.” —DavidBlei,ColumbiaUniversity Absolutelybrilliant.Thisbeautifullywrittencompendiumreviewsmanybigstatistical ideas,includingtheauthors’own.Amustforanyoneengagedcreativelyinstatistics andthedatasciences,forrepeateduse.EfronandHastiedemonstratetheever-growing powerofstatisticalreasoning,past,present,andfuture. —CarlMorris,HarvardUniversity ComputerAgeStatisticalInference Thetwenty-firstcenturyhasseenabreathtakingexpansionofstatisticalmethodology, bothinscopeandininfluence.“Bigdata,”“datascience,”and“machinelearning”have becomefamiliartermsinthenews,asstatisticalmethodsarebroughttobearuponthe enormousdatasetsofmodernscienceandcommerce.Howdidwegethere?Andwhere arewegoing? Thisbooktakesusonanexhilaratingjourneythroughtherevolutionindataanaly- sisfollowingtheintroductionofelectroniccomputationinthe1950s.Beginningwith classical inferential theories – Bayesian, frequentist, Fisherian – individual chapters take up a series of influential topics: survival analysis, logistic regression, empirical Bayes, the jackknife and bootstrap, random forests, neural networks, Markov chain MonteCarlo,inferenceaftermodelselection,anddozensmore.Thedistinctlymodern approach integrates methodology and algorithms with statistical inference. The book endswithspeculationonthefuturedirectionofstatisticsanddatascience. BRADLEY EFRONisMaxH.SteinProfessor,ProfessorofStatistics,andProfessorof BiomedicalDataScienceatStanfordUniversity.Hehasheldvisitingfacultyappoint- ments at Harvard, UC Berkeley, and Imperial College London. Efron has worked extensivelyontheoriesofstatisticalinference,andistheinventorofthebootstrapsam- plingtechnique.HereceivedtheNationalMedalofSciencein2005andtheGuyMedal inGoldoftheRoyalStatisticalSocietyin2014. TREVOR HASTIEisJohnA.OverdeckProfessor,ProfessorofStatistics,andProfes- sorofBiomedicalDataScienceatStanfordUniversity.HeiscoauthorofElementsof StatisticalLearning,akeytextinthefieldofmoderndataanalysis.Heisalsoknownfor hisworkongeneralizedadditivemodelsandprincipalcurves,andforhiscontributions totheRcomputingenvironment.HastiewasawardedtheEmmanuelandCarolParzen prizeforStatisticalInnovationin2014. INSTITUTE OF MATHEMATICAL STATISTICS MONOGRAPHS EditorialBoard D.R.Cox(UniversityofOxford) B.Hambly(UniversityofOxford) S.Holmes(StanfordUniversity) J.Wellner(UniversityofWashington) IMSMonographsareconciseresearchmonographsofhighqualityonanybranchof statistics or probability of sufficient interest to warrant publication as books. Some concern relatively traditional topics in need of up-to-date assessment. Others are on emergingthemes.Inallcasestheobjectiveistoprovideabalancedviewofthefield. OtherBooksintheSeries 1. Large-ScaleInference,byBradleyEfron 2. NonparametricInferenceonManifolds,byAbhishekBhattacharyaandRabi Battacharya 3. TheSkew-NormalandRelatedFamilies,byAdelchiAzzalini 4. Case-ControlStudies,byRuthH.KeoghandD.R.Cox 5. ComputerAgeStatisticalInference,byBradleyEfronandTrevorHastie Computer Age Statistical Inference Algorithms, Evidence, and Data Science BRADLEY EFRON StanfordUniversity,California TREVOR HASTIE StanfordUniversity,California 32AvenueoftheAmericas,NewYorkNY10013-2473,USA CambridgeUniversityPressispartoftheUniversityofCambridge. ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof education,learningandresearchatthehighestinternationallevelsofexcellence. www.cambridge.org Informationonthistitle:www.cambridge.org/9781107149892 (cid:2)c BradleyEfronandTrevorHastie2016 Thispublicationisincopyright.Subjecttostatutoryexception andtotheprovisionsofrelevantcollectivelicensingagreements, noreproductionofanypartmaytakeplacewithoutthewritten permissionofCambridgeUniversityPress. Firstpublished2016 PrintedintheUnitedKingdombyClays,StIvesplc AcataloguerecordforthispublicationisavailablefromtheBritishLibrary ISBN978-1-107-14989-2Hardback CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyof URLsforexternalorthird-partyinternetwebsitesreferredtointhispublication, anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain, accurateorappropriate. ToDonnaandLynda Contents Preface xv Acknowledgments xviii Notation xix PartI ClassicStatisticalInference 1 1 AlgorithmsandInference 3 1.1 ARegressionExample 4 1.2 HypothesisTesting 8 1.3 Notes 11 2 FrequentistInference 12 2.1 FrequentisminPractice 14 2.2 FrequentistOptimality 18 2.3 NotesandDetails 20 3 BayesianInference 22 3.1 TwoExamples 24 3.2 UninformativePriorDistributions 28 3.3 FlawsinFrequentistInference 30 3.4 ABayesian/FrequentistComparisonList 33 3.5 NotesandDetails 36 4 FisherianInferenceandMaximumLikelihoodEstimation 38 4.1 LikelihoodandMaximumLikelihood 38 4.2 FisherInformationandtheMLE 41 4.3 ConditionalInference 45 4.4 PermutationandRandomization 49 4.5 NotesandDetails 51 5 ParametricModelsandExponentialFamilies 53 ix