Werner Linde Probability Theory A First Course in Probability Theory and Statistics MathematicsSubjectClassification2010 Primary:60-01,62-01;Secondary:60A05 Author Prof.Dr.WernerLinde Friedrich-Schiller-UniversitätJena FakultätfürMathematik&Informatik InstitutfürStochastik Prof.fürStochastischeAnalysis D-07737Jena [email protected] and UniversityofDelaware DepartmentofMathematicalSciences 501EwingHall NewarkDE,19716 [email protected] ISBN978-3-11-046617-1 e-ISBN(PDF)978-3-11-046619-5 e-ISBN(EPUB)978-3-11-046625-6 ©2016WalterdeGruyterGmbH,Berlin/Boston Typesetting:IntegraSoftwareServicesPvt.Ltd. Printingandbinding:CPIbooksGmbH,Leck Coverimage:WernerLinde (cid:2)8Printedonacid-freepaper PrintedinGermany www.degruyter.com Preface Thisbookisintendedasanintroductorycourseforstudentsinmathematics,phys- ical sciences, engineering, or in other related fields. It is based on the experience ofprobabilitylecturestaughtduringthepast25years,wherethespectrumreached fromtwo-hourintroductorycourses,overMeasureTheoryandadvancedprobability classes,tosuchtopicsasStochasticProcessesandMathematicalStatistics.Until2012 theselecturesweredeliveredtostudentsattheUniversityofJena(Germany),andsince 2013tothoseattheUniversityofDelawareinNewark(USA). ThebookisthecompletelyrevisedversionoftheGermanedition“Stochastikfür dasLehramt,”whichappearedin2014atDeGruyter.AtmostuniversitiesinGermany, there exist special classes in Probability Theory for students who want to become teachersofmathematicsinhighschools.BesidesbasicfactsaboutProbabilityTheory, thesecoursesarealsosupposedtogiveanintroductionintoMathematicalStatistics. Thus,theoriginalmainintentionfortheGermanversionwastowriteabookthathelps thosestudentsunderstandProbabilityTheorybetter.Butsoonthebookturnedoutto alsobeusefulasintroductionforstudentsinotherfields,e.g.inmathematics,phys- ics,andsoon.Thuswedecided,inordertomakethebookapplicableforabroader audience,toprovideatranslationintheEnglishlanguage. DuringnumerousyearsofteachingIlearnedthefollowing: – Probabilisticquestionsareusuallyeasytoformulate,generallyhaveatightrela- tiontoeverydayproblems,andthereforeattracttheinterestoftheaudience.Every studentknowsthephenomenathatoccurwhenonerollsadie,playscards,tosses acoin,orplaysalottery.Thus,aninitialinterestinProbabilityTheoryexists. – In contrast, after a short time many students have very serious difficulties with understanding the presented topics. Consequently, a common opinion among students is that Probability Theory is a very complicated topic, causing a lot of problemsandtroubles. SurelythereexistseveralreasonsforthebadimageofProbabilityTheoryamongstu- dents.But,aswebelieve,themostimportantoneisasfollows.InProbabilityTheory, thetypeofproblemsandquestionsconsidered,aswellasthewayofthinking,differs considerablyfromtheproblems,questions,andthinkinginotherfieldsofmathem- atics,i.e.,fromfieldswithwhichthestudentsbecameacquaintedbeforeattendinga probabilitycourse.Forexample,inCalculusafunctionhasawell-describeddomain ofdefinition;mostlyitisdefinedbyaconcreteformula,hascertainpropertiesascon- tinuity,differentiability,andsoon.Afunctionissomethingveryconcretewhichcan bemadevividbydrawingitsgraph.Incontrast,inProbabilityTheoryfunctionsare mostlyinvestigatedasrandomvariables.Theyaredefinedonacompletelyunimport- ant,nonspecifiedsamplespace,andtheygenerallydonotpossessaconcreteformula fortheirdefinition.Itmayevenhappenthatonlytheexistenceofafunction(random variable) is known. The only property of a random variable which really matters is VIII Preface thedistributionofitsvalues.Thisandmanyothersimilartechniquesmakethewhole theorysomethingmysteriousandnotcompletelycomprehensible. Consideringthisobservation,weorganizedthebookinawaythattriestomake probabilistic problems more understandable and that puts the focus more onto ex- planationsofthedefinitions,notations,andresults.Thetoolsweusetodothisare examples;wepresentatleastonebeforeanewdefinition,inordertomotivateit,fol- lowedbymoreexamplesafterthedefinitiontomakeitcomprehensible.Hereweact uponthemaximexpressedbyEinstein’squote1: Exampleisn’tanotherwaytoteach,itistheonlywaytoteach. PresentingthebasicresultsandmethodsinProbabilityTheorywithoutusingresults, facts,andnotationsfromMeasureTheoryis,inouropinion,asdifficultastosquare the circle. Either one restricts oneself to discrete probability measures and random variables or one has to be unprecise. There is no other choice! In some places, it is possibletoavoidtheuseofmeasuretheoreticfacts,suchastheLebesgueintegral,or theexistenceofinfiniteproductmeasures,andsoon,butthepriceishigh.2Ofcourse, IalsostruggledwiththeproblemofmissingfactsfromMeasureTheorywhilewriting this book. Therefore, I tried to include some ideas and some results about 3-fields, measures, and integrals, hoping that a few readers become interested and want to learnmoreaboutMeasureTheory.Forthose,werefertothebooks[Coh13],[Dud02], or[Bil12]asgoodsources. Inthiscontext,letusmakesomeremarkabouttheverificationofthepresented results. Whenever it was possible, we tried to prove the stated results. Times have changed;whenIwasastudent,everytheorempresentedinamathematicallecture was proved – really every one. Facts and results without proof were doubtful and soon forgotten. And a tricky and elegant proof is sometimes more impressive than theprovenresult(atleasttous).Hopefully,somereaderswilllikesomeoftheproofs inthisbookasmuchaswedid. One of most used applications of Probability Theory is Mathematical Statistics. WhenImetformerstudentsofmine,Ioftenaskedthemwhichkindofmathematics they are mainly using now in their daily work. The overwhelming majority of them answeredthatoneoftheirmainfieldsofmathematicalworkisstatisticalproblems. Therefore,wedecidedtoincludeanintroductorychapteraboutMathematicalStatist- ics. Nowadays, due to the existence of good and fast statistical programs, it is very easy to analyze data, to evaluate confidence regions, or to test a given hypotheses. Butdothosewhousetheseprogramsalsoalwaysknowwhattheyaredoing?Since 1 Seehttp://www.alberteinsteinsite.com/quotes/einsteinquotes.html 2 Forexample,severalyearsago,toavoidtheuseoftheLebesgueintegral,Iintroducedtheexpected valueofarandomvariableasaRiemannintegralviaitsdistributionfunction.Thisismathematically correct,butattheendalmostnostudentsunderstoodwhattheexpectedvaluereallyis.Trytoprove thattheexpectedvalueislinearusingthisapproach! Preface IX wedoubtthatthisisso,westressedthefocusinthischaptertothequestionofwhy themainstatisticalmethodsworkandonwhatmathematicalbackgroundtheyrest. Wealsoinvestigatehowprecisestatisticaldecisionsareandwhatkindsoferrorsmay occur. The organization of this book differs a little bit from those in many other first- course books about Probability Theory. Having Measure Theory in the back of our mindscausesustothinkthatprobabilitymeasuresarethemostimportantingredi- entofProbabilityTheory;randomvariablescomeinsecond.Onthecontrary,many otherauthorsgoexactlytheotherway.Theystartwithrandomvariables,andprob- ability measures then occur as their distribution on their range spaces (mostly R). Inthiscase,astandardnormalprobabilitymeasuredoesnotexist,onlyastandard normaldistributedrandomvariable.Bothapproacheshavetheiradvantagesanddis- advantages,butaswesaid,forustheprobabilitymeasuresareinterestingintheirown right,andthereforewestartwiththeminChapter1,followedbyrandomvariablesin Section3. Thebookalsocontainssomefactsandresultsthataremoreadvancedandusually notpartofanintroductorycourseinProbabilityTheory.Suchtopicsare,forexample, theinvestigationofproductmeasures,orderstatistics,andsoon.Wehaveassigned those more involved sections with a star. They may be skipped at a first reading withoutlossinthefollowingchapters. Attheendofeachchapter,onefindsacollectionofsomeproblemsrelatedtothe contentsofthesection.Herewerestrictedourselvestoafewproblemsintheactual task;thesolutionsoftheseproblemsarehelpfultotheunderstandingofthepresented topics.Theproblemsaremainlytakenfromourcollectionofhomeworksandexams during the past years. For those who want to work with more problems we refer to manybooks,ase.g.[GS01a],[Gha05],[Pao06],or[Ros14],whichcontainahugecollec- tionofprobabilisticproblems,rangingfromeasytodifficult,fromnaturaltoartificial, frominterestingtoboring. FinallyIwanttoexpressmythankstothosewhosupportedmyworkatthetrans- lationandrevisionofthepresentbook.ManystudentsattheUniversityofDelaware helped me to improve my English and to correct wrong phrases and wrong expres- sions. To mention all of them is impossible. But among them were a few students who read whole chapters and, without them, the book would have never been fin- ished(orreadable).InparticularIwanttomentionEmilyWagnerandSpencerWalker. They both did really a great job. Many thanks! Let me also express my gratitude to ColleenMcInerney,RachelAustin,DanielAtadan,andQuentinDubroff,allstudents inDelawareandattendingmyclassesforsometime.Theyalsoreadwholesectionsof thebookandcorrectedmybrokenEnglish.Finally,mythanksgotoProfessorAnne LeuchtfromtheTechnicalUniversityinBraunschweig(Germany);herfieldofworkis MathematicalStatistics,andherhintsandremarksaboutChapter8inthisbookwere importanttome. X Preface AndlastbutnotleastIwanttothanktheDepartmentofMathematicalSciences attheUniversityofDelawarefortheexcellentworkingconditionsaftermyretirement inGermany. Newark,Delaware,June6,2016 WernerLinde Contents 1 Probabilities 1 1.1 ProbabilitySpaces 1 1.1.1 SampleSpaces 1 1.1.2 3-FieldsofEvents 2 1.1.3 ProbabilityMeasures 5 1.2 BasicPropertiesofProbabilityMeasures 9 1.3 DiscreteProbabilityMeasures 13 1.4 SpecialDiscreteProbabilityMeasures 18 1.4.1 DiracMeasure 18 1.4.2 UniformDistributiononaFiniteSet 18 1.4.3 BinomialDistribution 22 1.4.4 MultinomialDistribution 24 1.4.5 PoissonDistribution 27 1.4.6 HypergeometricDistribution 29 1.4.7 GeometricDistribution 33 1.4.8 NegativeBinomialDistribution 35 1.5 ContinuousProbabilityMeasures 39 1.6 SpecialContinuousDistributions 43 1.6.1 UniformDistributiononanInterval 43 1.6.2 NormalDistribution 46 1.6.3 GammaDistribution 48 1.6.4 ExponentialDistribution 51 1.6.5 ErlangDistribution 52 1.6.6 Chi-SquaredDistribution 53 1.6.7 BetaDistribution 54 1.6.8 CauchyDistribution 56 1.7 DistributionFunction 57 1.8 MultivariateContinuousDistributions 64 1.8.1 MultivariateDensityFunctions 64 1.8.2 MultivariateUniformDistribution 66 1.9 ⋆ProductsofProbabilitySpaces 71 1.9.1 Product3-FieldsandMeasures 71 1.9.2 ProductMeasures:DiscreteCase 74 1.9.3 ProductMeasures:ContinuousCase 76 1.10 Problems 79 2 ConditionalProbabilitiesandIndependence 86 2.1 ConditionalProbabilities 86 2.2 IndependenceofEvents 94 2.3 Problems 101 XII Contents 3 RandomVariablesandTheirDistribution 105 3.1 TransformationofRandomValues 105 3.2 ProbabilityDistributionofaRandomVariable 107 3.3 SpecialDistributedRandomVariables 117 3.4 RandomVectors 119 3.5 JointandMarginalDistributions 120 3.5.1 MarginalDistributions:DiscreteCase 123 3.5.2 MarginalDistributions:ContinuousCase 128 3.6 IndependenceofRandomVariables 131 3.6.1 IndependenceofDiscreteRandomVariables 134 3.6.2 IndependenceofContinuousRandomVariables 138 3.7 ⋆OrderStatistics 141 3.8 Problems 146 4 OperationsonRandomVariables 149 4.1 MappingsofRandomVariables 149 4.2 LinearTransformations 154 4.3 CoinTossingversusUniformDistribution 157 4.3.1 BinaryFractions 157 4.3.2 BinaryFractionsofRandomNumbers 160 4.3.3 RandomNumbersGeneratedbyCoinTossing 162 4.4 SimulationofRandomVariables 164 4.5 AdditionofRandomVariables 169 4.5.1 SumsofDiscreteRandomVariables 171 4.5.2 SumsofContinuousRandomVariables 175 4.6 SumsofCertainRandomVariables 177 4.7 ProductsandQuotientsofRandomVariables 189 4.7.1 Student’st-Distribution 192 4.7.2 F-Distribution 194 4.8 Problems 196 5 ExpectedValue,Variance,andCovariance 200 5.1 ExpectedValue 200 5.1.1 ExpectedValueofDiscreteRandomVariables 200 5.1.2 Expected Value of Certain Discrete Random Variables 203 5.1.3 ExpectedValueofContinuousRandomVariables 208 5.1.4 Expected Value of Certain Continuous Random Variables 211 5.1.5 PropertiesoftheExpectedValue 215 5.2 Variance 222 5.2.1 HigherMomentsofRandomVariables 222 5.2.2 VarianceofRandomVariables 226 Contents XIII 5.2.3 VarianceofCertainRandomVariables 229 5.3 CovarianceandCorrelation 233 5.3.1 Covariance 233 5.3.2 CorrelationCoefficient 240 5.4 Problems 243 6 NormallyDistributedRandomVectors 248 6.1 RepresentationandDensity 248 6.2 ExpectedValueandCovarianceMatrix 256 6.3 Problems 262 7 LimitTheorems 264 7.1 LawsofLargeNumbers 264 7.1.1 Chebyshev’sInequality 264 7.1.2 ⋆Infinite Sequences of Independent Random Variables 267 7.1.3 ⋆Borel–CantelliLemma 270 7.1.4 WeakLawofLargeNumbers 276 7.1.5 StrongLawofLargeNumbers 278 7.2 CentralLimitTheorem 283 7.3 Problems 298 8 MathematicalStatistics 301 8.1 StatisticalModels 301 8.1.1 NonparametricStatisticalModels 301 8.1.2 ParametricStatisticalModels 305 8.2 StatisticalHypothesisTesting 307 8.2.1 HypothesesandTests 307 8.2.2 PowerFunctionandSignificanceTests 310 8.3 TestsforBinomialDistributedPopulations 315 8.4 TestsforNormallyDistributedPopulations 319 8.4.1 Fisher’sTheorem 320 8.4.2 Quantiles 323 8.4.3 Z-TestsorGaussTests 326 8.4.4 t-Tests 328 8.4.5 72-TestsfortheVariance 330 8.4.6 Two-SampleZ-Tests 332 8.4.7 Two-Samplet-Tests 334 8.4.8 F-Tests 336 8.5 PointEstimators 337 8.5.1 MaximumLikelihoodEstimation 338 8.5.2 UnbiasedEstimators 346 8.5.3 RiskFunction 351 XIV Contents 8.6 ConfidenceRegionsandIntervals 355 8.7 Problems 362 AAppendix 365 A.1 Notations 365 A.2 ElementsofSetTheory 365 A.3 Combinatorics 367 A.3.1 BinomialCoefficients 367 A.3.2 DrawingBallsoutofanUrn 373 A.3.3 MultinomialCoefficients 376 A.4 VectorsandMatrices 379 A.5 SomeAnalyticTools 382 Bibliography 389 Index 391