Springer Texts in Statistics SeriesEditors: G.Casella S.E.Fienberg I.Olkin Forfurthervolumes: http://www.springer.com/series/417 Mary Kathryn Cowles Applied Bayesian Statistics With R and OpenBUGS Examples 123 MaryKathrynCowles DepartmentofStatistics andActuarialScience UniversityofIowa IowaCity,Iowa,USA ISSN1431-875X ISBN978-1-4614-5695-7 ISBN978-1-4614-5696-4(eBook) DOI10.1007/978-1-4614-5696-4 SpringerNewYorkHeidelbergDordrechtLondon LibraryofCongressControlNumber:2012951150 ©SpringerScience+BusinessMediaNewYork2013 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer. PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations areliabletoprosecutionundertherespectiveCopyrightLaw. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. While the advice and information in this book are believed to be true and accurate at the date of publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with respecttothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) To Brendan,Lucy, andDonald. Preface Ihavetaughta coursecalled“BayesianStatistics” atthe UniversityofIowaevery academicyearsince1998–1999.Thisbookisintendedtofitthegoalsandaudience addressedbymycourse.The“CourseObjectives”sectionofmysyllabusreads: Through hands-on experience withreal datafrom avarietyofapplications, studentswill learn the basics of designing and carrying out Bayesian analyses, and interpreting and communicatingtheresults.Studentswilllearntousesoftwarepackages includingRand OpenBUGStofitBayesianmodels. The course is intended to be intensely practical, focussing on building under- standingof the conceptsand proceduresrequiredto performBayesian analysisof realdatatoanswerrealquestions.Emphasisisgiventosuchissuesasdetermining what data is needed to address a particular question; choosing an appropriate probabilitydistributionforsampledata;quantifyingalready-existingknowledgein the form of a prior distribution on model parameters; verifying that the posterior distribution will be proper if improper prior distributions are used; and when and howtospecifyhierarchicalmodels.Interpretationandcommunicationofresultsare stressed,includingdifferencesfrom,andsimilaritiesto,classicalapproachestothe sameproblems. WinBUGS and OpenBUGS currently are the dominant software in applied use of Bayesian methods. I have chosen to introduce OpenBUGS as the primary data analysis software in this textbook because, unlike WinBUGS, OpenBUGS is undergoingcontinuingdevelopmentandhasversionsthatrunnativelyunderLinux andMacintoshoperatingsystemsaswellasWindows.Althoughsomebackground is provided on the Markov chain Monte Carlo sampling procedures employed by WinBUGS andOpenBUGS, the emphasisis on those tasks thatauser mustcarry outcorrectlyforreasonablytrustworthyinference.Theseincludeusingappropriate toolstoassesswhetherandwhenasamplerhasconvergedtothetargetdistribution, decidinghowmanyiterationsareneededforacceptableaccuracyinestimation,and how to reportresults of a Bayesian analysis conductedwith OpenBUGS. Caveats aboutthefallibilityofconvergencediagnosticsareemphasized. vii viii Preface Studentsofdifferentlevelsanddisciplinestakethecourse,including:undergrad- uatemathematicsandstatisticsmajors;master’sstudentsinstatistics, biostatistics, statistical genetics, educational testing and measurement, and engineering; and PhD students in economics, marketing,psychology,and geographyas well as the previouslylistedfields.Inaddition,severalpracticingstatisticiansemployedbythe UniversityofIowaandAmericanCollegeTesting(ACT)havetakenthecourse. Thegoalofthecourse,andofthisbook,istoprovideanintroductiontoBayesian principlesandpracticethatisclear,useful,andunintimidatingtomotivatedstudents evenif theydo nothavean advancedbackgroundin mathematicsand probability. I emphasizeintuitiveinsightwithoutsacrificingmathematicalcorrectness.Prereq- uisites are one or two semesters of calculus-based probability and mathematical statistics(atleastattheHoggandTannislevel)andoneortwosemestersofclassical statistical methods, including linear regression (David Moore’s Basic Practice of Statistics level). Elementaryintegraland differentialcalculusis occasionallyused inlecturesandhomework.Linearalgebraisnotrequired. Coralville,Iowa MaryKathrynCowles Contents 1 WhatIsBayesianStatistics? .............................................. 1 1.1 TheScientificMethod(ButItIsNotJustforScience...).......... 1 1.2 ABitofHistory ...................................................... 2 1.3 ExampleoftheBayesianMethod:DoesMyFriend HaveBreastCancer?................................................. 3 1.3.1 QuantifyingUncertaintyUsingProbabilities.............. 3 1.3.2 ModelsandPriorProbabilities............................. 5 1.3.3 Data ......................................................... 6 1.3.4 LikelihoodsandPosteriorProbabilities ................... 7 1.3.5 BayesianSequentialAnalysis.............................. 8 1.4 CalibrationExperimentsforAssessingSubjectiveProbabilities... 8 1.5 WhatIstoCome? .................................................... 10 Problems ..................................................................... 11 2 ReviewofProbability ...................................................... 13 2.1 ReviewofProbability................................................ 13 2.1.1 EventsandSampleSpaces................................. 13 2.1.2 Unions,Intersections,Complements ...................... 14 2.1.3 TheAdditionRule.......................................... 15 2.1.4 MarginalandConditionalProbabilities ................... 15 2.1.5 TheMultiplicationRule.................................... 17 2.2 PuttingItAllTogether:DidBrendanMailtheBillPayment?..... 17 2.2.1 TheLawofTotalProbability .............................. 17 2.2.2 Bayes’RuleintheDiscreteCase.......................... 19 2.3 RandomVariablesandProbabilityDistributions................... 20 Problems ..................................................................... 21 3 IntroductiontoOne-ParameterModels:Estimating aPopulationProportion ................................................... 25 3.1 WhatProportionofStudentsWouldQuitSchoolIfTuition WereRaised19%:EstimatingaPopulationProportion............ 25 ix x Contents 3.2 TheFirstStageofaBayesianModel................................ 25 3.2.1 TheBinomialDistributionforOurSurvey................ 26 3.2.2 KernelsandNormalizingConstants....................... 27 3.2.3 TheLikelihoodFunction................................... 27 3.3 TheSecondStageoftheBayesianModel:ThePrior............... 28 3.3.1 OtherPossiblePriorDistributions......................... 29 3.3.2 PriorProbabilityIntervals.................................. 31 3.4 UsingtheDatatoUpdatethePrior:ThePosteriorDistribution ... 32 3.5 ConjugatePriors...................................................... 34 3.5.1 ComputingthePosteriorDistribution withaConjugatePrior...................................... 34 3.5.2 ChoosingtheParametersofaBetaDistribution toMatchPriorBeliefs...................................... 35 3.5.3 ComputingandGraphingthePosteriorDistribution...... 38 3.5.4 PlottingthePriorDensity,theLikelihood, andthePosteriorDensity................................... 38 3.6 IntroductiontoRforBayesianAnalysis............................ 38 3.6.1 FunctionsandObjectsinR ................................ 39 3.6.2 Summarizingand GraphingProbability DistributionsinR........................................... 42 3.6.3 PrintingandSavingRGraphics ........................... 44 3.6.4 RPackagesUsefulinBayesianAnalysis.................. 44 3.6.5 EndingaSession............................................ 46 Problems ..................................................................... 46 4 InferenceforaPopulationProportion ................................... 49 4.1 EstimationandTesting:FrequentistApproach ..................... 49 4.1.1 MaximumLikelihoodEstimation.......................... 49 4.1.2 FrequentistConfidenceIntervals........................... 51 4.1.3 FrequentistHypothesisTesting ............................ 52 4.2 BayesianInference:SummarizingthePosteriorDistribution...... 54 4.2.1 ThePosteriorMean......................................... 54 4.2.2 OtherBayesianPointEstimates............................ 55 4.2.3 BayesianPosteriorIntervals ............................... 57 4.3 UsingthePosteriorDistributiontoTestHypotheses............... 59 4.4 PosteriorPredictiveDistributions ................................... 61 Problems ..................................................................... 63 5 SpecialConsiderationsinBayesianInference........................... 67 5.1 RobustnesstoPriorSpecifications .................................. 67 5.2 InferenceUsingNonconjugatePriors............................... 69 5.2.1 DiscretePriors.............................................. 69 5.2.2 AHistogramPrior.......................................... 71 5.3 NoninformativePriors ............................................... 72 5.3.1 ReviewofProperandImproperDistributions ............ 72 5.3.2 ANoninformativePriorfortheBinomialLikelihood .... 73