Recursive identification of smoothing spline ANOVA models Marco Ratto, Andrea Pagano European Commission, Joint Research Centre, Ispra, ITALY July 8, 2009 Introduction We discuss different approaches to the estimation and identification of smoothing splines ANOVA models: • The ‘classical’ approach [Wahba, 1990, Gu, 2002], as improved by Storlie et al. [ACOSSO]; • the recursive approach of Ratto et al. [2007], Young [2001] [SDR]. 1 Introduction: ACOSSO ‘a new regularization method for simultaneous model fitting and variable selection in nonparametric regression models in the framework of smoothing spline ANOVA’. COSSO [Lin and Zhang, 2006] penalizes the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. Storlie et al. introduce an adaptive weight in the COSSO penalty allowing more flexibility in the estimate of important functional components (using heavier penalty to unimportant ones). 2 Introduction: SDR Using the the State-Dependent Parameter Regression (SDR) approach of Young [2001], Ratto et al. [2007] have developed a non-parametric approach very similar to smoothing splines, based on recursive filtering and smoothing estimation [the Kalman Filter, KF, combined with Fixed Interval Smoothing ,FIS, Kalman, 1960, Young, 1999]: • couched with optimal Maximum Likelihood estimation; • flexibility in adapting to local discontinuities, heavy non-linearity and heteroscedastic error terms. 3 Goals of the paper 1. develop a formal comparison and demonstrate equivalences between the ‘classical’ tensor product cubic spline approach and the SDR approach; 2. discuss advantages and disadvantages of these approaches; 3. propose a unified approach to smoothing spline ANOVA models that combines the best of the discussed methods. 4 State Dependent Regressions and smoothing splines: Additive models Denote the generic mapping as z(X), where X ∈ [0, 1]p and p is the number of parameters. The simplest example of smoothing spline mapping estimation of z is the additive model: p X f(X) = f + f (X ) (1) 0 j j j=1 5 To estimate f we can use a multivariate smoothing spline minimization problem, that is, given λ, find the minimizer f(X ) k of: N p Z 1 1 X X 2 00 2 (z − f(X )) + λ [f (X )] dX (2) k k j j j j N 0 k=1 j=1 where a Monte Carlo sample of dimension N is assumed. This minimization problem requires the estimation of the p hyper- parameters λ (also denoted as smoothing parameters): GCV, GML, j etc. (see e.g. Wahba, 1990; Gu, 2002). 6 In the recursive approach by Ratto et al. [2007], the additive model is put into a State-Dependent Parameter Regression (SDR) form of Young [2001]. Consider the case of p = 1 and z(X) = g(X) + e, with e ∼ N(0, σ2), i.e. z = s + e , k k k where k = 1, . . . , N and s is the estimate of g(X ). k k The s is characterized in some stochastic manner, borrowing k from non-stationary time series processes and using the Generalized Random Walk (GRW) class on non-stationary random sequences [see e.g. Young and Ng, 1989, Ng and Young, 1990]. 7 The integrated random walk (IRW) process provides the same smoothing properties of a cubic spline, in the overall State-Space (SS) formulation: Observation Equation: z = s + e k k k State Equations: s = s + d (3) k k−1 k−1 d = d + η k k−1 k where d is the ‘slope’ of s , η ∼ N(0, σ2) and η is independent k k k η k of e . k For the recursive estimate of s , the MC sample has to be sorted k in ascending order of X, i.e. the k and k − 1 subscripts in (3) denote adjacent elements under such ordering. 8 2.5 2 l a n 1.5 g i s k z d e 1 t r o s 0.5 0 0 10 20 30 40 sorted k−ordering (increasing X ) 1 Figure 1: 9