Springer Texts in Statistics James E. Gentle Matrix Algebra Theory, Computations and Applications in Statistics Second Edition Springer Texts in Statistics Series Editors Richard DeVeaux Stephen E. Fienberg Ingram Olkin More information about this series at http://www.springer.com/series/417 James E. Gentle Matrix Algebra Theory, Computations and Applications in Statistics Second Edition 123 James E. Gentle Fairfax, VA, USA ISSN 1431-875X ISSN2197-4136 (electronic) Springer Texts in Statistics ISBN 978-3-319-64866-8 ISBN 978-3-319-64867-5 (eBook) DOI 10.1007/978-3-319-64867-5 LibraryofCongressControlNumber:2017952371 1stedition:©SpringerScience+BusinessMedia,LLC2007 2ndedition:©SpringerInternational PublishingAG2017 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewhole orpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseof illustrations, recitation, broadcasting, reproduction on microfilmsor in any other physical way,andtransmissionorinformationstorageandretrieval,electronicadaptation,computer software,orbysimilarordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneral descriptive names,registerednames,trademarks, servicemarks,etc. in thispublicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnames areexemptfromtherelevantprotectivelawsandregulationsandthereforefreeforgeneral use. Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinforma- tioninthisbookarebelievedtobetrueandaccurateatthedateofpublication.Neitherthe publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or forany errorsor omissionsthat mayhave been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Printedonacid-freepaper ThisSpringerimprintispublishedbySpringerNature TheregisteredcompanyisSpringerInternational PublishingAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland To Mar´ıa Preface to the Second Edition In this second edition, I have corrected all known typos and other errors; I have (it is hoped) clarified certain passages; I have added some additional material; and I have enhanced the Index. I haveadded a few more comments aboutvectorsand matrices with com- plex elements, although, as before, unless stated otherwise, all vectors and matrices in this book are assumed to have real elements. I have begun to use “det(A)” rather than “|A|” to represent the determinant of A, except in a few cases. I have also expressed some derivatives as the transposes of the expressions I used formerly. I have put more conscious emphasis on “user-friendliness” in this edition. Inabook,user-friendlinessis primarilyafunctionofreferences,bothinternal andexternal,andoftheindex.Asanoldsoftwaredesigner,I’vealwaysthought thatuser-friendlinessisveryimportant.Totheextentthatinternalreferences werepresentinthe firstedition, the positivefeedbackI receivedfromusersof thateditionaboutthefriendlinessofthoseinternalreferences(“Ilikedthefact thatyousaid‘equation(x.xx)onpageyy,’insteadofjust‘equation(x.xx)’”) encouraged me to try to make the internal references even more useful. It’s onlywhenyou’re“eatingyourowndogfood,”thatyoubecomeawareofwhere detailsmatter,andinusingthefirstedition,Irealizedthatthechoiceofentries in the Index was suboptimal. I have spent significant time in organizing it, and I hope that the user will find the Index to this edition to be very useful. I think that it has been vastly improved over the Index in the first edition. The overall organization of chapters has been preserved, but some sec- tions have been changed. The two chapters that have been changed most are Chaps. 3 and 12. Chapter 3, on the basics of matrices, got about 30 pages longer. It is by far the longest chapter in the book, but I just didn’t see any reasonable way to break it up. In Chap. 12 of the first edition, “Software for Numerical Linear Algebra,” I discussed four software systems or languages, C/C++, Fortran, Matlab, and R, and did not express any preference for one vii viii Preface to the Second Edition over another. In this edition, although I occasionally mention various lan- guages and systems, I now limit most of my discussion to Fortran and R. There are many reasonsfor my preference for these two systems. R is ori- ented towardstatistical applications. It is open source and freely distributed. As for Fortran versus C/C++, Python, or other programming languages, I agree with the statement by Hanson and Hopkins (2013, page ix), “... For- tran is currently the best computer language for numerical software.” Many people, however, still think of Fortran as the language their elders (or they themselves) used in the 1970s. (On a personal note, Richard Hanson, who passed away recently, was a member of my team that designed the IMSL C Libraries in the mid 1980s. Not only was C much cooler than Fortran at the time,buttheANSIcommitteeworkingonupdatingtheFortranlanguagewas so fractured by competing interests that approval of the revision was repeat- edly delayed.Manynumericalanalystswho werenotconcernedwithcoolness turned to C because it provided dynamic storage allocation and it allowed flexibleargumentlists,andtheFortranconstructscouldnotbeagreedupon.) Language preferences are personal, of course, and there is a strong “cool- ness factor” in choice of a language. Python is currently one of the coolest languages, but I personally don’t like the language for most of the stuff I do. Although this book has separate parts on applications in statistics and computational issues as before, statistical applications have informed the choices I made throughout the book, and computational considerations have given direction to most discussions. I thank the readers of the first edition who informed me of errors. Two peopleinparticularmadeseveralmeaningfulcommentsandsuggestions.Clark Fitzgeraldnotonlyidentifiedseveraltypos,hemadeseveralbroadsuggestions about organization and coverage that resulted in an improved text (I think). Andreas Eckner found, in addition to typos, some gaps in my logic and also suggested better lines of reasoning at some places. (Although I don’t follow anitemized “theorem-proof”format,I try to givereasonsforany nonobvious statementsImake.)IthankClarkandAndreasespeciallyfortheircomments. Any remaining typos, omissions, gaps in logic, and so on are entirely my responsibility. Again,Ithankmywife,Mar´ıa,towhomthisbookisdedicated,foreverything. I used TEX via LATEX2ε to write the book. I did all of the typing, program- ming, etc., myself, so all misteaks (mistakes!) are mine. I would appreciate receiving suggestions for improvement and notification of errors. Notes on this book, including errata, are available at http://mason.gmu.edu/~jgentle/books/matbk/ Fairfax County, VA, USA James E. Gentle July 14, 2017 Preface to the First Edition I began this book as anupdate of Numerical Linear Algebra for Applications in Statistics, published by Springer in 1998. There was a modest amount of newmaterialtoadd,butIalsowantedtosupplymoreofthereasoningbehind the facts about vectors and matrices. I had used material from that text in some courses, and I had spent a considerable amount of class time proving assertions made but not proved in that book. As I embarked on this project, the character of the book began to change markedly. In the previous book, I apologized for spending 30 pages on the theory and basic facts of linear algebra before getting on to the main interest: numerical linear algebra. In this book, discussion of those basic facts takes up over half of the book. The orientationand perspective of this book remains numerical linear al- gebra for applications in statistics. Computational considerations inform the narrative. There is an emphasis on the areas of matrix analysis that are im- portant for statisticians, and the kinds of matrices encountered in statistical applications receive special attention. This book is divided into three parts plus a set of appendices. The three parts correspond generally to the three areas of the book’s subtitle—theory, computations, and applications—although the parts are in a different order, and there is no firm separation of the topics. Part I, consisting of Chaps. 1 through 7, covers most of the material in linear algebra needed by statisticians. (The word“matrix” in the title of this book may suggest a somewhat more limited domain than “linear algebra”; but I use the former term only because it seems to be more commonly used by statisticians and is used more or less synonymously with the latter term.) The first four chapters cover the basics of vectors and matrices, concen- trating on topics that are particularly relevant for statistical applications. In Chap. 4, it is assumed that the reader is generally familiar with the basics of partial differentiation of scalar functions. Chapters 5 through 7 begin to take on more of an applications flavor, as well as beginning to give more consid- eration to computational methods. Although the details of the computations ix x Preface to theFirst Edition are not covered in those chapters, the topics addressed are oriented more to- ward computational algorithms. Chapter 5 covers methods for decomposing matrices into useful factors. Chapter 6 addresses applications of matrices in setting up and solving linearsystems,including overdeterminedsystems.We shouldnotconfusesta- tistical inference with fitting equations to data, although the latter task is a componentoftheformeractivity.InChap.6,weaddressthemoremechanical aspects of the problemof fitting equations to data. Applications in statistical dataanalysisarediscussedinChap.9.Inthoseapplications,weneedtomake statements (i.e., assumptions) about relevant probability distributions. Chapter 7 discusses methods for extracting eigenvalues and eigenvectors. Therearemanyimportantdetailsofalgorithmsforeigenanalysis,buttheyare beyondthescopeofthisbook.AswithotherchaptersinPartI,Chap.7makes some reference to statistical applications, but it focuses on the mathematical and mechanical aspects of the problem. Althoughthefirstpartison“theory,”thepresentationisinformal;neither definitionsnorfactsarehighlightedbysuchwordsas“definition,”“theorem,” “lemma,” and so forth. It is assumed that the reader follows the natural development. Most ofthe facts have simple proofs, andmost proofsare given naturally in the text. No “Proof” and “Q.E.D.” or “ ” appear to indicate beginning and end; again, it is assumed that the reader is engaged in the development. For example, on page 341: IfAisnonsingularandsymmetric,thenA−1isalsosymmetricbecause (A−1)T =(AT)−1 =A−1. The first part of that sentence could have been stated as a theorem and givena number, and the lastpartofthe sentence couldhavebeen introduced as the proof, with reference to some previous theorem that the inverse and transposition operations can be interchanged. (This had already been shown before page 341—in an unnumbered theorem of course!) Noneoftheproofsareoriginal(atleast,Idon’tthinktheyare),butinmost cases, I do not know the original source or even the source where I first saw them. I would guess that many go back to C. F. Gauss. Most, whether they areasoldasGaussornot,haveappearedsomewhereintheworkofC.R.Rao. Some lengthier proofs are only given in outline, but references are given for the details. Very useful sources of details of the proofs are Harville (1997), especially for facts relating to applications in linear models, and Horn and Johnson(1991),formoregeneraltopics,especiallythoserelatingtostochastic matrices. The older books by Gantmacher (1959) provide extensive coverage andoftenrathernovelproofs.Thesetwovolumeshavebeenbroughtbackinto print by the American Mathematical Society. Ialsosometimesmakesimpleassumptionswithoutstatingthemexplicitly. For example, I may write “for all i” when i is used as an index to a vector. I hope it is clear that “for all i” means only “for i that correspond to indices