An Introduction to Statistical Signal Processing f 1(F) (cid:0) f F Pr(f F) = P( ! :! F ) = P(f 1(F)) (cid:0) 2 f 2 g - November 22, 2003 i ii An Introduction to Statistical Signal Processing Robert M. Gray and Lee D. Davisson Information Systems Laboratory Department of Electrical Engineering Stanford University and Department of Electrical Engineering and Computer Science University of Maryland iii iv c 2003 by Cambridge University Press (cid:13) v to our Families vi Contents Preface page xii Glossary xvii 1 Introduction 1 2 Probability 12 2.1 Introduction 12 2.2 Spinning Pointers and Flipping Coins 16 2.3 Probability Spaces 27 2.3.1 Sample Spaces 32 2.3.2 Event Spaces 36 2.3.3 Probability Measures 49 2.4 Discrete Probability Spaces 53 2.5 Continuous Probability Spaces 64 2.6 Independence 81 2.7 Elementary Conditional Probability 82 2.8 Problems 86 3 Random Objects 96 3.1 Introduction 96 3.1.1 Random Variables 96 3.1.2 Random Vectors 101 3.1.3 Random Processes 105 3.2 Random Variables 109 3.3 Distributions of Random Variables 119 3.3.1 Distributions 119 3.3.2 Mixture Distributions 124 vii viii Contents 3.3.3 Derived Distributions 127 3.4 Random Vectors and Random Processes 132 3.5 Distributions of Random Vectors 135 3.5.1 ?Multidimensional Events 136 3.5.2 Multidimensional Probability Functions 137 3.5.3 Consistency of Joint and Marginal Distri- butions 139 3.6 Independent Random Variables 146 3.7 Conditional Distributions 149 3.7.1 Discrete Conditional Distributions 150 3.7.2 Continuous Conditional Distributions 152 3.8 Statistical Detection and Classi(cid:12)cation 155 3.9 Additive Noise 158 3.10 Binary Detection in Gaussian Noise 167 3.11 Statistical Estimation 168 3.12 Characteristic Functions 170 3.13 Gaussian Random Vectors 176 3.14 Simple Random Processes 178 3.15 Directly Given Random Processes 182 3.15.1 The Kolmogorov Extension Theorem 182 3.15.2 IID Random Processes 182 3.15.3 Gaussian Random Processes 183 3.16 Discrete Time Markov Processes 184 3.16.1 A Binary Markov Process 184 3.16.2 The Binomial Counting Process 187 3.16.3 ?Discrete Random Walk 191 3.16.4 The Discrete Time Wiener Process 192 3.16.5 Hidden Markov Models 194 3.17 ?Nonelementary Conditional Probability 194 3.18 Problems 196 4 Expectation and Averages 213 4.1 Averages 213 4.2 Expectation 217 4.3 Functions of Random Variables 221 4.4 Functions of Several Random Variables 228 4.5 Properties of Expectation 229 Contents ix 4.6 Examples 232 4.6.1 Correlation 232 4.6.2 Covariance 235 4.6.3 Covariance Matrices 236 4.6.4 Multivariable Characteristic Functions 237 4.6.5 Di(cid:11)erential Entropy of a Gaussian Vector 240 4.7 Conditional Expectation 241 4.8 ?Jointly Gaussian Vectors 245 4.9 Expectation as Estimation 248 4.10 ? Implications for Linear Estimation 256 4.11 Correlation and Linear Estimation 258 4.12 Correlation and Covariance Functions 267 4.13 ?The Central Limit Theorem 270 4.14 Sample Averages 274 4.15 Convergence of Random Variables 276 4.16 Weak Law of Large Numbers 284 4.17 ?Strong Law of Large Numbers 287 4.18 Stationarity 292 4.19 Asymptotically Uncorrelated Processes 298 4.20 Problems 302 5 Second-Order Theory 322 5.1 Linear Filtering of Random Processes 324 5.2 Linear Systems I/O Relations 326 5.3 Power Spectral Densities 333 5.4 Linearly Filtered Uncorrelated Processes 335 5.5 Linear Modulation 343 5.6 White Noise 346 5.6.1 Low Pass and Narrow Band Noise 351 5.7 ?Time-Averages 351 5.8 ?Mean Square Calculus 355 5.8.1 Mean Square Convergence Revisited 356 5.8.2 Integrating Random Processes 363 5.8.3 Linear Filtering 367 5.8.4 Di(cid:11)erentiating Random Processes 368 5.8.5 Fourier Series 373 5.8.6 Sampling 377 x Contents 5.8.7 Karhunen-Loueve Expansion 382 5.9 ?Linear Estimation and Filtering 387 5.9.1 Discrete Time 388 5.9.2 Continuous Time 403 5.10 Problems 407 6 A Menagerie of Processes 424 6.1 Discrete Time Linear Models 425 6.2 Sums of IID Random Variables 430 6.3 Independent Stationary Increment Processes 432 6.4 ?Second-Order Moments of ISI Processes 436 6.5 Speci(cid:12)cation of Continuous Time ISI Processes 438 6.6 Moving-Average and Autoregressive Processes 441 6.7 The Discrete Time Gauss-Markov Process 443 6.8 Gaussian Random Processes 444 6.9 The Poisson Counting Process 445 6.10 Compound Processes 449 6.11 Composite Random Processes 451 6.12 ?Exponential Modulation 452 6.13 ?Thermal Noise 458 6.14 Ergodicity 461 6.15 Random Fields 466 6.16 Problems 467 A Preliminaries 479 A.1 Set Theory 479 A.2 Examples of Proofs 489 A.3 Mappings and Functions 492 A.4 Linear Algebra 493 A.5 Linear System Fundamentals 497 A.6 Problems 503 B Sums and Integrals 508 B.1 Summation 508 B.2 ?Double Sums 511 B.3 Integration 513 B.4 ?The Lebesgue Integral 515 C Common Univariate Distributions 519 D Supplementary Reading 522