Statistical Inference for Engineers and Data Scientists A mathematically accessible and up-to-date introduction to the tools needed to address modern inference problems in engineering and data science, ideal for graduate students taking courses on statistical inference and detection and estimation, and an invaluable reference for researchers and professionals. With a wealth of illustrations and examples to explain the key features of the the- ory and to connect with real-world applications, additional material to explore more advanced concepts, and numerous end-of-chapter problems to test the reader’s knowl- edge, this textbook is the “go-to” guide for learning about the core principles of statistical inference and its application in engineering and data science. The password-protected Solutions Manual and the Image Gallery from the book are available at www.cambridge.org/Moulin. Pierre Moulin is a professor in the ECE Department at the University of Illinois at Urbana-Champaign. His research interests include statistical inference, machine learn- ing, detection and estimation theory, information theory, statistical signal, image, and video processing, and information security. Moulin is a Fellow of the IEEE, and served as a Distinguished Lecturer for the IEEE Signal Processing Society. He has received two best paper awards from the IEEE Signal Processing Society and the US National Science Foundation CAREER Award. He was founding Editor-in-Chief of the IEEE Transactions on Information Security and Forensics. Venugopal V. Veeravalli is the Henry Magnuski Professor in the ECE Department at the University of Illinois at Urbana-Champaign. His research interests include statistical inference and machine learning, detection and estimation theory, and information the- ory, with applications to data science, wireless communications, and sensor networks. Veeravalli is a Fellow of the IEEE, and served as a Distinguished Lecturer for the IEEE Signal Processing Society. Among the awards he has received are the IEEE Browder J. Thompson Best Paper Award, the National Science Foundation CAREER Award, the Presidential Early Career Award for Scientists and Engineers (PECASE), and the Wald Prize in Sequential Analysis. Statistical Inference for Engineers and Data Scientists PIERRE MOULIN University of Illinois, Urbana-Champaign VENUGOPAL V. VEERAVALLI University of Illinois, Urbana-Champaign University Printing House, Cambridge CB2 8BS, United Kingdom One Liberty Plaza, 20th Floor, New York, NY 10006, USA 477 Williamstown Road, Port Melbourne, VIC 3207, Australia 314–321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre, New Delhi – 110025, India 79 Anson Road, #06–04/06, Singapore 079906 Cambridge University Press is part of the University of Cambridge. It furthers the University’s mission by disseminating knowledge in the pursuit of education, learning, and research at the highest international levels of excellence. www.cambridge.org Information on this title: www.cambridge.org/9781107185920 DOI: 10.1017/9781107185920 ±c Cambridge University Press 2019 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2019 Printed in the United Kingdom by TJ International Ltd. Padstow Cornwall A catalogue record for this publication is available from the British Library. ISBN 978-1-107-18592-0 Hardback Additional resources for this publication at www.cambridge.org/Moulin. Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication and does not guarantee that any content on such websites is, or will remain, accurate or appropriate. We balance probabilities and choose the most likely – Sherlock Holmes Brief Contents Preface pagexvii List of Acronyms xx 1 Introduction 1 Part I Hypothesis Testing 23 2 Binary Hypothesis Testing 25 3 Multiple Hypothesis Testing 54 4 Composite Hypothesis Testing 71 5 Signal Detection 105 6 Convex Statistical Distances 145 7 Performance Bounds for Hypothesis Testing 160 8 Large Deviations and Error Exponents for Hypothesis Testing 184 9 Sequential and Quickest Change Detection 208 10 Detection of Random Processes 231 Part II Estimation 257 11 Bayesian Parameter Estimation 259 12 Minimum Variance Unbiased Estimation 280 13 Information Inequality and Cramér–Rao Lower Bound 297 14 Maximum Likelihood Estimation 319 viii Brief Contents 15 Signal Estimation 358 Appendix A Matrix Analysis 384 Appendix B Random Vectors and Covariance Matrices 390 Appendix C Probability Distributions 391 Appendix D Convergence of Random Sequences 393 Index 395 Contents Preface pagexvii List of Acronyms xx 1 Introduction 1 1.1 Background 1 1.2 Notation 1 1.2.1 Probability Distributions 2 1.2.2 Conditional Probability Distributions 2 1.2.3 Expectations and Conditional Expectations 3 1.2.4 Unified Notation 3 1.2.5 General Random Variables 3 1.3 Statistical Inference 4 1.3.1 Statistical Model 5 1.3.2 Some Generic Estimation Problems 6 1.3.3 Some Generic Detection Problems 6 1.4 Performance Analysis 7 1.5 Statistical Decision Theory 7 1.5.1 Conditional Risk and Optimal Decision Rules 8 1.5.2 Bayesian Approach 9 1.5.3 Minimax Approach 10 1.5.4 Other Non-Bayesian Rules 11 1.6 Derivation of Bayes Rule 12 1.7 Link Between Minimax and Bayesian Decision Theory 14 1.7.1 Dual Concept 14 1.7.2 Game Theory 15 1.7.3 Saddlepoint 15 1.7.4 Randomized Decision Rules 16 Exercises 18 References 21 Part I Hypothesis Testing 23 2 Binary Hypothesis Testing 25 2.1 General Framework 25 2.2 Bayesian Binary Hypothesis Testing 26 x Contents 2.2.1 Likelihood Ratio Test 27 2.2.2 Uniform Costs 28 2.2.3 Examples 28 2.3 Binary Minimax Hypothesis Testing 32 2.3.1 Equalizer Rules 33 2.3.2 Bayes Risk Line and Minimum Risk Curve 34 2.3.3 Differentiable V(π ) 35 0 2.3.4 Nondifferentiable V(π ) 35 0 2.3.5 Randomized LRTs 37 2.3.6 Examples 38 2.4 Neyman–Pearson Hypothesis Testing 40 2.4.1 Solution to the NP Optimization Problem 41 2.4.2 NP Rule 42 2.4.3 Receiver Operating Characteristic 43 2.4.4 Examples 44 2.4.5 Convex Optimization 46 Exercises 47 3 Multiple Hypothesis Testing 54 3.1 General Framework 54 3.2 Bayesian Hypothesis Testing 55 3.2.1 Optimal Decision Regions 56 3.2.2 Gaussian Ternary Hypothesis Testing 58 3.3 Minimax Hypothesis Testing 58 3.4 Generalized Neyman–Pearson Detection 62 3.5 Multiple Binary Tests 62 3.5.1 Bonferroni Correction 63 3.5.2 False Discovery Rate 64 3.5.3 Benjamini–Hochberg Procedure 65 3.5.4 Connection to Bayesian Decision Theory 66 Exercises 67 References 70 4 Composite Hypothesis Testing 71 4.1 Introduction 71 4.2 Random Parameter± 72 4.2.1 Uniform Costs Over Each Hypothesis 73 4.2.2 Nonuniform Costs Over Hypotheses 76 4.3 Uniformly Most Powerful Test 77 4.3.1 Examples 77 4.3.2 Monotone Likelihood Ratio Theorem 79 4.3.3 Both Composite Hypotheses 80 4.4 Locally Most Powerful Test 82 4.5 Generalized Likelihood Ratio Test 84