Probability and Statistics for Programmers Think Stats Probability and Statistics for Programmers Version1.5.9 Allen B. Downey Green Tea Press Needham,Massachusetts GreenTeaPress 9WashburnAve NeedhamMA02492 Permission is granted to copy, distribute, and/or modify this document under thetermsoftheCreativeCommonsAttribution-NonCommercial3.0UnportedLi- cense,whichisavailableathttp://creativecommons.org/licenses/by-nc/3.0/. Preface Why I wrote this book Think Stats: Probability and Statistics for Programmers is a textbook for a new kind of introductory prob-stat class. It emphasizes the use of statistics to explorelargedatasets. Ittakesacomputationalapproach,whichhasseveral advantages: • Studentswriteprogramsasawayofdevelopingandtestingtheirun- derstanding. For example, they write functions to compute a least squares fit, residuals, and the coefficient of determination. Writing and testing this code requires them to understand the concepts and implicitlycorrectsmisunderstandings. • Students run experiments to test statistical behavior. For example, they explore the Central Limit Theorem (CLT) by generating samples fromseveraldistributions. Whentheyseethatthesumofvaluesfrom a Pareto distribution doesn’t converge to normal, they remember the assumptionstheCLTisbasedon. • Some ideas that are hard to grasp mathematically are easy to under- stand by simulation. For example, we approximate p-values by run- ning Monte Carlo simulations, which reinforces the meaning of the p-value. • Using discrete distributions and computation makes it possible to present topics like Bayesian estimation that are not usually covered in an introductory class. For example, one exercise asks students to compute the posterior distribution for the “German tank problem,” whichisdifficultanalyticallybutsurprisinglyeasycomputationally. • Because students work in a general-purpose programming language (Python), they are able to import data from almost any source. They are not limited to data that has been cleaned and formatted for a par- ticularstatisticstool. vi Chapter0. Preface The book lends itself to a project-based approach. In my class, students workonasemester-longprojectthatrequiresthemtoposeastatisticalques- tion,findadatasetthatcanaddressit,andapplyeachofthetechniquesthey learntotheirowndata. TodemonstratethekindofanalysisIwantstudentstodo,thebookpresents a case study that runs through all of the chapters. It uses data from two sources: • The National Survey of Family Growth (NSFG), conducted by the U.S. Centers for Disease Control and Prevention (CDC) to gather “information on family life, marriage and divorce, pregnancy, infer- tility, use of contraception, and men’s and women’s health.” (See http://cdc.gov/nchs/nsfg.htm.) • The Behavioral Risk Factor Surveillance System (BRFSS), conducted by the National Center for Chronic Disease Prevention and Health Promotionto“trackhealthconditionsandriskbehaviorsintheUnited States.” (Seehttp://cdc.gov/BRFSS/.) Other examples use data from the IRS, the U.S. Census, and the Boston Marathon. How I wrote this book When people write a new textbook, they usually start by reading a stack of old textbooks. As a result, most books contain the same material in pretty much the same order. Often there are phrases, and errors, that propagate fromonebooktothenext;StephenJayGouldpointedoutanexampleinhis essay,“TheCaseoftheCreepingFoxTerrier1.” I did not do that. In fact, I used almost no printed material while I was writingthisbook,forseveralreasons: • My goal was to explore a new approach to this material, so I didn’t wantmuchexposuretoexistingapproaches. • SinceIammakingthisbookavailableunderafreelicense,Iwantedto makesurethatnopartofitwasencumberedbycopyrightrestrictions. 1AbreedofdogthatisabouthalfthesizeofaHyracotherium(seehttp://wikipedia. org/wiki/Hyracotherium). vii • Manyreadersofmybooksdon’thaveaccesstolibrariesofprintedma- terial,soItriedtomakereferencestoresourcesthatarefreelyavailable ontheInternet. • Proponents of old media think that the exclusive use of electronic re- sourcesislazyandunreliable. Theymightberightaboutthefirstpart, but I think they are wrong about the second, so I wanted to test my theory. The resource I used more than any other is Wikipedia, the bugbear of li- brarianseverywhere. Ingeneral,thearticlesIreadonstatisticaltopicswere very good (although I made a few small changes along the way). I include references to Wikipedia pages throughout the book and I encourage you to follow those links; in many cases, the Wikipedia page picks up where my description leaves off. The vocabulary and notation in this book are gener- allyconsistentwithWikipedia,unlessIhadagoodreasontodeviate. Other resources I found useful were Wolfram MathWorld and (of course) Google. I also used two books, David MacKay’s Information Theory, In- ference, and Learning Algorithms, which is the book that got me hooked on Bayesian statistics, and Press et al.’s Numerical Recipes in C. But both books areavailableonline,soIdon’tfeeltoobad. AllenB.Downey NeedhamMA AllenB.DowneyisaProfessorofComputerScienceattheFranklinW.Olin CollegeofEngineering. Contributor List If you have a suggestion or correction, please send email to [email protected]. If I make a change based on your feed- back, I will add you to the contributor list (unless you ask to be omitted). Ifyouincludeatleastpartofthesentencetheerrorappearsin,thatmakesit easy for me to search. Page and section numbers are fine, too, but not quite aseasytoworkwith. Thanks! • Lisa Downey and June Downey read an early draft and made many correc- tionsandsuggestions. viii Chapter0. Preface • StevenZhangfoundseveralerrors. • Andy Pethan and Molly Farison helped debug some of the solutions, and Mollyspottedseveraltypos. • AndrewHeinefoundanerrorinmyerrorfunction. • Dr. NikolasAkerblomknowshowbigaHyracotheriumis. • AlexMorrowclarifiedoneofthecodeexamples. • JonathanStreetcaughtanerrorinthenickoftime. • GáborLiptákfoundatypointhebookandtherelayracesolution. • Many thanks to Kevin Smith and Tim Arnold for their work on plasTeX, whichIusedtoconvertthisbooktoDocBook. • GeorgeCaplansentseveralsuggestionsforimprovingclarity. • JulianCeipekfoundanerrorandanumberoftypos. • Stijn Debrouwere, Leo Marihart III, Jonathan Hammler, and Kent Johnson founderrorsinthefirstprintedition. • DanKearneyfoundatypo. • JeffPickhardtfoundabrokenlinkandatypo. • Jörg Beyer found typos in the book and made many corrections in the doc- stringsoftheaccompanyingcode. • TommieGannertsentapatchfilewithanumberofcorrections. Contents Preface v 1 Statisticalthinkingforprogrammers 1 1.1 Dofirstbabiesarrivelate? . . . . . . . . . . . . . . . . . . . . 2 1.2 Astatisticalapproach . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 TheNationalSurveyofFamilyGrowth . . . . . . . . . . . . 3 1.4 Tablesandrecords . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.6 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 Descriptivestatistics 11 2.1 Meansandaverages . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4 Representinghistograms . . . . . . . . . . . . . . . . . . . . . 14 2.5 Plottinghistograms . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6 RepresentingPMFs . . . . . . . . . . . . . . . . . . . . . . . . 16 2.7 PlottingPMFs . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.8 Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.9 Othervisualizations . . . . . . . . . . . . . . . . . . . . . . . . 20 x Contents 2.10 Relativerisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.11 Conditionalprobability . . . . . . . . . . . . . . . . . . . . . . 21 2.12 Reportingresults . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.13 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3 Cumulativedistributionfunctions 25 3.1 Theclasssizeparadox . . . . . . . . . . . . . . . . . . . . . . 25 3.2 ThelimitsofPMFs . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3 Percentiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.4 Cumulativedistributionfunctions . . . . . . . . . . . . . . . 29 3.5 RepresentingCDFs . . . . . . . . . . . . . . . . . . . . . . . . 30 3.6 Backtothesurveydata . . . . . . . . . . . . . . . . . . . . . . 32 3.7 Conditionaldistributions . . . . . . . . . . . . . . . . . . . . . 32 3.8 Randomnumbers . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.9 Summarystatisticsrevisited . . . . . . . . . . . . . . . . . . . 34 3.10 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4 Continuousdistributions 37 4.1 Theexponentialdistribution . . . . . . . . . . . . . . . . . . . 37 4.2 TheParetodistribution . . . . . . . . . . . . . . . . . . . . . . 40 4.3 Thenormaldistribution . . . . . . . . . . . . . . . . . . . . . 42 4.4 Normalprobabilityplot . . . . . . . . . . . . . . . . . . . . . 45 4.5 Thelognormaldistribution . . . . . . . . . . . . . . . . . . . 46 4.6 Whymodel? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.7 Generatingrandomnumbers . . . . . . . . . . . . . . . . . . 49 4.8 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50