Regression Estimators A Comparative Study Marvin H. J. Gruber Department of Mathematics Rochester Institute of Technology Rochester, New York ACADEMIC PRESS, INC. Harcourt Brace Jovanovich, Publishers Boston San Diego New 'fork London Sydney Tokyo Toronto This book is printed on acid-free paper. ® Copyright © 1990 by Academic Press, Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. ACADEMIC PRESS, INC. 1250 Sixth Avenue, San Diego, CA 92101 United Kingdom Edition published by ACADEMIC PRESS LIMITED 24-28 Oval Road, London NW1 7DX Library of Congress Cataloging-in-Publication Data Gruber, Marvin H.J. Regression estimators : a comparative study / Marvin H.J. Gruber. p. cm. — (Statistical modeling and decision science) Includes bibliographical references. ISBN 0-12-304752-8 (alk. paper) 1. Ridge regression (Statistics) 2. Estimation theory. I. Title. II. Series. QA278.2.G78 1990 519.5-dc20 89-29740 CIP Printed in the United States of America 90919293 987654321 Preface The study of different mathematical formulations of ridge regression type estimators points to a very curious observation. The estima- tors can be derived by both Bayesian and non-Bayesian (frequentist) methods. The Bayesian approach assumes that the parameters being estimated have a prior distribution. The derivations using frequen- tist type arguments also use prior information, but it is in the form of inequality constraints and additional observations. The goal of this book is to present, compare, and contrast the development and the properties of the ridge type estimators that result from these two philosophically different points of view. The book is divided into four parts. The first part (Chapters I and II) explores the need for alternatives to least square estimators, gives a historical survey of the literature and summarizes basic ideas in Matrix Theory and Statistical Decision Theory used throughout the book. Although the historical survey is far from exhaustive, the books and research papers cited there should provide the interested reader with many entry points to the vast literature on ridge type estimators. The author hopes that the summaries of results from Matrix Theory and Statistical Decision Theory not usually presented in undergraduate courses will make this specialized book readable to a wider audience. The second part (Chapters III and IV) presents the estimators from both the Bayesian and from the frequentist points of view and explores the mathematical relationships between them. ix X REGRESSION ESTIMATORS The third part (Chapters V-VIII) considers the efficiency of the estimators with and without averaging over a prior distribution. Part IV, the final two chapters IX and X, suggests applications of the methods and results of Chapters III-VII to Kaiman Filters and Analysis of Variance, two very important areas of application. In addition to the theoretical discussion numerical examples are given to illustrate how the theoretical results work in special cases. There are over 100 exercises to help the reader check his or her un- derstanding. The author hopes that this work will prove valuable to professional statisticians and workers in fields that use statistical methods who would like to know more about the analytical properties of ridge type estimators. Any work of this kind is not possible without the help and support of many people. Some of the research that led to the writing of this book began in the author's Ph.D. dissertation at the University of Rochester (Gruber, 1979). The author is very grateful to his thesis advisor, Dr. Poduri S.R.S.Rao, for suggesting dissertation work in this area, his help, encouragement, prodding, and patience. A lot of benefit was also derived from later collaboration with him, especially on the joint paper, Gruber and P.S.R.S.Rao (1982). A number of colleagues at the Rochester Institute of Technol- ogy provided help and support in various ways. These include Dr. John Paliouras, Dr. George Georgantas, Prof. Pasquale Saeva, Prof. Richard Orr, Dr. Patricia Clark, Dr. James Halavin, Prof. David Crystal and Dr. Mao Bautista. The author is grateful for the financial support granted for his research in the form of a Dean's Fellowship in the College of Sci- ence during several summers, and for released time from some of his teaching duties to work on the book. The first draft was prepared by Stephen Beren of Kinko's Copies. The final camera ready version was typed by undergraduate students in the Applied Mathematics, Computational Mathematics, or Ap- plied Statistics programs of the Mathematics Department of Roches- ter Institute of Technology's College of Science. The students in- clude Richard Allan, Christopher Bean, Elaine Hock, Tanya Lomac and Christopher J. Mietlicki. The quality of this book was greatly enhanced by their painstaking efforts. PREFACE xi The author is also indebted to the secretarial staff of the Depart- ment of Mathematics for their help with correspondence, copying, and the like and for their patience. Special thanks for all of this are due Elnora Ayers, Ann Gottorff and Shirley Zavaglia. The staff of Academic Press was very helpful, patient, and sup- portive during the final stages of preparation of the manuscript. A very special "thank you" is due to Publisher Klaus Peters, Assistant Editor Susan Gay, Managing Editor Natasha Sabath and Production Editor Elizabeth Tustian. Special thanks are also due to Ingram Olkin and Gerald J. Lieber- man, the editors of the Statistical Modeling and Decision Science series. Their suggestions really helped make this a better book. Marvin H.J.Gruber Rochester, New York Chapter I Introduction 1.0 Motivation for Writing This Book During the past 15 to 20 years many different kinds of estimators have been proposed as alternatives to least squares estimators for the estimation of the parameters of a linear regression model. Ridge regression type estimators have proven to be very useful for multi- collinear data. As a result, since 1970, many papers and books have been written about ridge and related estimators. There are a number of different mathematical formulations of ridge type estimators that are treated in the literature. Some of the formulations use Bayesian methods. Others employ methods within the context of the frequentist point of view. These include descriptions of the estimators as: 1. a special case of the Bayes estimator (BE); 2. a special case of the mixed estimator; 3. special cases of minimax estimators; 4. the solution to the problem of finding the point on an ellipsoid that is closest to a point in the parameter space. These different approaches yield estimators with similar mathe- matical forms. Furthermore, for certain kinds of prior information and design matrices the estimators are the same. Throughout the 3 4 REGRESSION ESTIMATORS literature, in the author's experience, most articles or books describe the estimators from only one of the above-mentioned points of view. For this reason the author felt that a comparative study of the sim- ilarities and differences amongst ridge type estimators obtained by the different methods should be undertaken. 1.1 Purpose of This Book The objectives of this book are: 1. to give different mathematical formulations of ridge type estima- tors from both the frequentist and Bayesian points of view; 2. to explore the relationship between different kinds of prior infor- mation, the estimators obtained and their efficiencies; 3. to see how the methodology in points 1 and 2 can be used in a few special situations. (Of particular interest will be the Kaiman filter and experimental design models.) The rest of this chapter will contain: 1. a brief historical survey of work done by this and other authors (This survey is by no means exhaustive.); 2. some of the basic ideas about ridge and least square estimators; 3. an outline of the structure of the rest of the book. 1.2 Least Square Estimators and the Need for Alternatives The method of least squares is often used by statisticians to estimate the parameters of a linear regression model. It consists of minimiz- ing the sum of the squares of the differences between the predicted and the observed values of the dependent variables. As will be seen, when there appears to be a linear dependence between the inde- pendent variables, the precision of the estimates may be very poor. Ridge type estimators tend to be more precise for this situation. The derivation of the least square estimator and the derivation of the well-known Hoerl and Kennard ridge regression estimators will INTRODUCTION 5 now be given. Let Y = Χβ + ε (1.2.1) be a linear model. The matrix X is a known n X m matrix of rank s < ra, β is an m dimensional vector and Y and ε are n dimensional random variables. Then ε has mean 0 and dispersion σ21. The least square (LS) estimator is derived by minimizing the sum of the squares of the error terms. Thus, minimize F(ß) = (Y-Xß)'(Y-Xß) (1.2.2) = Y'Y + ß'X'Xß - 2ß'X'Y by differentiation. Then dF(ß) = X'Xß - X'Y. (1.2.3) dß Set (1.2.3) equal to zero. Thus, the normal equation X'Xß = X'Y (1.2.4) is obtained. The solutions to the normal equation (1.2.4) are the least square estimators. Sometimes in practice there may be a linear dependence between the m independent variables represented by the X matrix. Two situations may occur: 1. The case of exact multicoUinearity: The rank of X is strictly less than s. 2. The case of near multicoUinearity: The X'X matrix has at least one very small eigenvalue. Then det X'X is very close to zero. When X is of full rank the solution to (1.2.4) is 6= (X'X^X'F. (1.2.5) When X is of less than full rank the solutions to the normal equations can still be found by finding a matrix G where b = GX'Y. (1.2.6) 6 REGRESSION ESTIMATORS The matrix G is not unique and is called a generalized inverse. Gen- eralized inverses and their properties will be considered in Chapter II. Some of the properties of least square estimators for the less than full rank model will be given in Chapter III. The dispersion of b is D(b) = a2(X'X)-1. (1.2.7) The total variance is s 1 Tr D(b) = σ2Ύτ{Χ'Χ)-λ = σ2 V -^, (1.2.8) <=i Xi where the λ,- are the non-zero eigenvalues of XfX. From the form of (1.2.8) notice that the total variance would be severely inflated if one or more of the λ; was very small. Hoerl and Kennard (1970) suggested using an estimator of the form β = (X'X + kI)~lX'Y, (1.2.9) with k a positive constant. Then the total variance of the ridge estimators of β is i=i Äi + * Clearly, (1.2.10) is less than (1.2.8). The estimator (1.2.9) is called a ridge regression estimator. It is obtained as the solution to the problem of minimizing the distance BfB of a vector of parameters B from the origin subject to the side condition (B - b)'XfX(B -b) = Φ . (1.2.11) 0 This is equivalent to finding the point on the ellipsoid centered at the LS estimator b that is closest to the origin. This ellipsoid may be thought of as a ridge, hence the name ridge regression. The problem is solved by the method of Lagrange multipliers. Differentiate L = B'B+ \[(B - b)'X'X(B - b) - Φ ] (1.2.12) 0 INTRODUCTION 7 to obtain B + \x'X(B - b) = 0. (1.2.13) Solve (1.2.12) for B to obtain (1.2.9) . Example 1.2.1. How Ridge Estimators Are More Precise. Examples of multicollinear data may be found frequently in eco- nomics. The data below is taken from the Economic Report of the President(1988). It represents the relationship between the depen- dent variable,Y (personal consumption expenditures) in billions of dollars, and three other independent variables, Xi,X2, and X3. The variable X\ represents the Gross National Product, X2 represents Personal Income (in billions of dollars), and X3 represents the total number of employed people in the civilian labor force (in thousands). Table 1.2.1 Economic Data OBS YEAR Y ~i Xi X3 1 1965 440.7 705.1 552.0 71088 2 1966 477.3 772.0 600.8 72895 3 1967 503.6 816.4 644.5 74372 4 1968 552.5 892.7 707.2 75920 5 1969 579.9 963.9 772.9 77902 6 1970 640.0 1015.5 831.8 78678 7 1971 691.6 1102.7 894.0 79367 8 1972 757.6 1212.8 981.6 82153 9 1973 837.2 1359.3 1101.7 85064 10 1974 916.5 1472.8 1210.1 86794 11 1975 1012.8 1598.4 1313.4 85846 12 1976 1129.3 1782.8 1451.4 88752 13 1977 1257.2 1990.5 1607.5 92017 14 1978 1403.5 2249.7 1812.4 96048 15 1979 1566.8 2508.2 2034.0 98824 16 1980 1732.6 2732.0 2258.5 99303 17 1981 1915.1 3052.6 2520.9 100397 18 1982 2050.7 3166.0 2670.8 99526 19 1983 2234.5 3405.7 2838.6 100834 20 1984 2430.5 3772.2 3108.7 105005