Texts in Statistical Science r i[N inear I eries im:e Theory, Methods, and Applications with R Examples o. h Al A i T III 111 0 J 'k oa T zo no goo 2g ao so 8o tog time time 0 T 00 0% O O O- i Oops 4 O 4- .11 -20 -10 0 10' =Aol =1g. O= no. XI Awl R an, dial D o u c Eric Moulines- David S. Stoffe r CRC Press Taylor & Francis Croup Aa CHAPMA'N' & HALL Nonlinear Time Series Theory, Methods, and Applications with R Examples Randal Douc Eric Moulines David Stoffer CHAPMAN & HALL/CRC Texts in Statistical Science Series Series Editors Francesca Dominici, Harvard School of Public Health, USA Julian J. Faraway, University of Bath, UK Martin Tanner, Northwestern University, USA Jim Zidek, University of British Columbia, Canada Statistical Theory: A Concise Introduction Introduction to Statistical Methods for F. Abramovich and Y. Ritov Clinical Trials T.D. Cook and D.L. DeMets Practical Multivariate Analysis, Fifth Edition A. Afifi, S. May, and V.A. Clark Applied Statistics: Principles and Examples Practical Statistics for Medical Research D.R. Cox and E.J. Snell D.G. Altman Multivariate Survival Analysis and Competing Interpreting Data: A First Course Risks in Statistics M. Crowder A.J.B. Anderson Statistical Analysis of Reliability Data Introduction to Probability with R M.J. Crowder, A.C. Kimber, K. Baclawski T.J. Sweeting, and R.L. Smith Linear Algebra and Matrix Analysis for An Introduction to Generalized Statistics Linear Models, Third Edition S. Banerjee and A. Roy A.J. Dobson and A.G. Barnett Statistical Methods for SPC and TQM Nonlinear Time Series: Theory, Methods, and D. Bissell Applications with R Examples R. Douc, E. Moulines, and D. Stoffer Bayesian Methods for Data Analysis, Third Edition Introduction to Optimization Methods and B.P. Carlin and T.A. Louis Their Applications in Statistics B.S. Everitt Second Edition R. Caulcutt Extending the Linear Model with R: Generalized Linear, Mixed Effects and The Analysis of Time Series: An Introduction, Nonparametric Regression Models Sixth Edition J.J. Faraway C. Chatfield A Course in Large Sample Theory Introduction to Multivariate Analysis T.S. Ferguson C. Chatfield and A.J. Collins Multivariate Statistics: A Practical Approach Problem Solving: A Statistician’s Guide, B. Flury and H. Riedwyl Second Edition C. Chatfield Readings in Decision Analysis S. French Statistics for Technology: A Course in Applied Statistics, Third Edition Markov Chain Monte Carlo: C. Chatfield Stochastic Simulation for Bayesian Inference, Second Edition Bayesian Ideas and Data Analysis: An D. Gamerman and H.F. Lopes Introduction for Scientists and Statisticians Bayesian Data Analysis, Third Edition R. Christensen, W. Johnson, A. Branscum, A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, and T.E. Hanson A. Vehtari, and D.B. Rubin Modelling Binary Data, Second Edition Multivariate Analysis of Variance and D. Collett Repeated Measures: A Practical Approach for Modelling Survival Data in Medical Research, Behavioural Scientists Second Edition D.J. Hand and C.C. Taylor D. Collett Practical Data Analysis for Designed Practical The BUGS Book: A Practical Introduction to Longitudinal Data Analysis Bayesian Analysis D.J. Hand and M. Crowder D. Lunn, C. Jackson, N. Best, A. Thomas, and D. Spiegelhalter Logistic Regression Models J.M. Hilbe Introduction to General and Generalized Linear Models Richly Parameterized Linear Models: H. Madsen and P. Thyregod Additive, Time Series, and Spatial Models Using Random Effects Time Series Analysis J.S. Hodges H. Madsen Statistics for Epidemiology Pólya Urn Models N.P. Jewell H. Mahmoud Stochastic Processes: An Introduction, Randomization, Bootstrap and Monte Carlo Second Edition Methods in Biology, Third Edition P.W. Jones and P. Smith B.F.J. Manly The Theory of Linear Models Introduction to Randomized Controlled B. Jørgensen Clinical Trials, Second Edition J.N.S. Matthews Principles of Uncertainty J.B. Kadane Statistical Methods in Agriculture and Experimental Biology, Second Edition Graphics for Statistics and Data Analysis with R R. Mead, R.N. Curnow, and A.M. Hasted K.J. Keen Statistics in Engineering: A Practical Approach Mathematical Statistics A.V. Metcalfe K. Knight Beyond ANOVA: Basics of Applied Statistics Nonparametric Methods in Statistics with SAS Applications R.G. Miller, Jr. O. Korosteleva A Primer on Linear Models Modeling and Analysis of Stochastic Systems, J.F. Monahan Second Edition Applied Stochastic Modelling, Second Edition V.G. Kulkarni B.J.T. Morgan Exercises and Solutions in Biostatistical Theory Elements of Simulation L.L. Kupper, B.H. Neelon, and S.M. O’Brien B.J.T. Morgan Exercises and Solutions in Statistical Theory Probability: Methods and Measurement L.L. Kupper, B.H. Neelon, and S.M. O’Brien A. O’Hagan Design and Analysis of Experiments with SAS Introduction to Statistical Limit Theory J. Lawson A.M. Polansky A Course in Categorical Data Analysis Applied Bayesian Forecasting and Time Series T. Leonard Analysis Statistics for Accountants A. Pole, M. West, and J. Harrison S. Letchford Statistics in Research and Development, Introduction to the Theory of Statistical Time Series: Modeling, Computation, and Inference Inference H. Liero and S. Zwanzig R. Prado and M. West Statistical Theory, Fourth Edition Introduction to Statistical Process Control B.W. Lindgren P. Qiu Stationary Stochastic Processes: Theory and Sampling Methodologies with Applications Applications P.S.R.S. Rao G. Lindgren Texts in Statistical Science Nonlinear Time Series Theory, Methods, and Applications with R Examples Randal Douc Telecom SudParis Evry, France Randal Douc Eric Moulines Eric Moulines Telecom ParisTech Paris, France David Stoffer David Stoffer University of Pittsburgh Pennsylvania, USA CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2014 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Printed on acid-free paper Version Date: 20131111 International Standard Book Number-13: 978-1-4665-0225-3 (Hardback) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information stor- age or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copy- right.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that pro- vides licenses and registration for a variety of users. For organizations that have been granted a pho- tocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com Contents Preface xiii FrequentlyUsedNotation xvii I Foundations 1 1 LinearModels 3 1.1 Stochasticprocesses 3 1.2 Thecovarianceworld 5 1.2.1 Second-orderstationaryprocesses 5 1.2.2 Spectralrepresentation 9 1.2.3 Wolddecomposition 13 1.3 Linearprocesses 15 1.3.1 WhatarelinearGaussianprocesses? 15 1.3.2 ARMAmodels 16 1.3.3 Prediction 19 1.3.4 Estimation 21 1.4 Themultivariatecases 25 1.4.1 Timedomain 25 1.4.2 Frequencydomain 27 1.5 Numericalexamples 28 Exercises 30 2 LinearGaussianStateSpaceModels 33 2.1 Modelbasics 33 2.2 Filtering,smoothing,andforecasting 36 2.3 Maximumlikelihoodestimation 42 2.3.1 Newton–Raphson 42 2.3.2 EMalgorithm 43 2.4 SmoothingsplinesandtheKalmansmoother 45 2.5 AsymptoticdistributionoftheMLE 47 2.6 Missingdatamodifications 49 2.7 Structuralcomponentmodels 50 2.8 State-spacemodelswithcorrelatederrors 53 2.8.1 ARMAXmodels 54 vii viii CONTENTS 2.8.2 Regressionwithautocorrelatederrors 56 Exercises 56 3 BeyondLinearModels 61 3.1 Nonlinearnon-Gaussiandata 62 3.2 Volterraseriesexpansion 68 3.3 Cumulantsandhigher-orderspectra 69 3.4 Bilinearmodels 72 3.5 Conditionallyheteroscedasticmodels 73 3.6 ThresholdARMAmodels 77 3.7 Functionalautoregressivemodels 78 3.8 Linearprocesseswithinfinitevariance 79 3.9 Modelsforcounts 81 3.9.1 Integervaluedmodels 81 3.9.2 Generalizedlinearmodels 83 3.10 Numericalexamples 84 Exercises 89 4 StochasticRecurrenceEquations 91 4.1 TheScalarCase 93 4.1.1 Strictstationarity 93 4.1.2 Weakstationarity 98 4.1.3 GARCH(1,1) 102 4.2 TheVectorCase 107 4.2.1 Strictstationarity 109 4.2.2 Weakstationarity 111 4.2.3 GARCH(p,q) 114 4.3 Iteratedrandomfunction 118 4.3.1 Strictstationarity 118 4.3.2 Weakstationarity 121 Exercises 123 II MarkovianModels 131 5 MarkovModels:ConstructionandDefinitions 133 5.1 Markovchains:Past,future,andforgetfulness 133 5.2 Kernels 134 5.3 HomogeneousMarkovchain 136 5.4 Canonicalrepresentation 138 5.5 Invariantmeasures 139 5.6 Observation-drivenmodels 142 5.7 Iteratedrandomfunctions 143 5.8 MCMCmethods 152 5.8.1 Metropolis-Hastingsalgorithm 152 5.8.2 Gibbssampling 155 CONTENTS ix Exercises 157 6 StabilityandConvergence 165 6.1 Uniformergodicity 166 6.1.1 Totalvariationdistance 166 6.1.2 Dobrushincoefficient 167 6.1.3 TheDoeblincondition 169 6.1.4 Examples 169 6.2 V-geometricergodicity 173 6.2.1 V-totalvariationdistance 173 6.2.2 V-Dobrushincoefficient 174 6.2.3 Driftandminorizationconditions 175 6.2.4 Examples 180 6.3 Someproofs 186 6.4 Endnotes 188 Exercises 189 7 SamplePathsandLimitTheorems 195 7.1 Lawoflargenumbers 196 7.1.1 Dynamicalsystemandergodicity 196 7.1.2 Markovchainergodicity 203 7.2 Centrallimittheorem 211 7.3 Deviationinequalitiesforadditivefunctionals 218 7.3.1 Rosenthaltypeinequality 218 7.3.2 Concentrationinequality 221 7.4 Someproofs 225 Exercises 231 8 InferenceforMarkovianModels 239 8.1 Likelihoodinference 239 8.2 ConsistencyandasymptoticnormalityoftheMLE 245 8.2.1 Consistency 245 8.2.2 Asymptoticnormality 247 8.3 Observation-drivenmodels 254 8.4 Bayesianinference 263 8.5 Someproofs 271 8.6 Endnotes 274 Exercises 275 III StateSpaceandHiddenMarkovModels 285 9 Non-GaussianandNonlinearStateSpaceModels 287 9.1 Definitionsandbasicproperties 287 9.1.1 Discrete-valuedstatespaceHMM 287 9.1.2 Continuous-valuedstate-spacemodels 295