This article was downloaded by: [Universitetsbiblioteket i Bergen] On: 22 April 2015, At: 00:18 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Journal of Statistical Computation and Simulation Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/gscs20 Sensitivity measures,anova- like Techniques and the use of bootstrap G. E. B. Archer a , A. Saltelli a & I. M. Sobol b a Joint Research Centre of the European Commission , Ispra, VA, 21020, Italy b National Centre for Mathematical Modelling of the Russian Academy of Science , 4A Miusskaya Square, 125047, Moscow(CIS) Published online: 20 Mar 2007. To cite this article: G. E. B. Archer , A. Saltelli & I. M. Sobol (1997) Sensitivity measures,anova-like Techniques and the use of bootstrap, Journal of Statistical Computation and Simulation, 58:2, 99-120, DOI: 10.1080/00949659708811825 To link to this article: http://dx.doi.org/10.1080/00949659708811825 PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and- 5 1 conditions 0 2 ril p A 2 2 8 1 0: 0 at ] n e g r e B et i k e ot bli bi s et sit r e v ni U [ y b d e d a o nl w o D J. Statist. Comput. Simul., 1997, Vol. 58, pp.99-120 0 1997 OPA (Overseas Publishers Association) Reprints available directly from the publisher Amsterdam B.V. Published in The Netherlands under Photocopying permitted by license only license by Gordon and Breach Science Publishers Printed in India 5 SENSITIVITY MEASURES, ANOVA-LIKE 1 0 2 TECHNIQUES AND THE USE ril p OF BOOTSTRAP A 2 2 18 G. E. B. ARCHERa , A. SALTELLIa and I. M. SOBOL~ 0: 0 at aJoint Research Centre of the European Commission, 21020 Zspra (VA),I taly; ] b~ationaCl entre for Mathematical Modelling of the Russian Academy n e of Science, 4A Miusskaya Square, 125047 Moscow (CIS) g r e B et i (Received 3 September 1996; In final form 21 January 1997) k e ot bli Souobtpoul't ,s aernes ictoivmitpya irneddi cweist,h utsheed Ainn avlyasriisa nocf eV baarisaendc eg lionb calla ssseincsailt fivacittyo rainala ldyessisig onf. Mmoondetel sbi Carlo computation of Sobol' indices is described briefly, and a bootstrap approach is et presented, which can be used to produce a confidence interval for the true, unknown sit indices. r e v ni Keywords: Analysis of variance decomposition; bootstrap; sensitivity indices U [ y b d e ad 1. INTRODUCTION o nl w o In the context of numerical experiments, sensitivity analysis (SA) aims D to quantify the relative importance of input variables X = (XI. ,. . , X,) in determining the value of an assigned output variable Y = f (X). (Note that Y could in fact be vector-valued without affecting any of the following results). More specifically, global SA tries to quantify output uncertainty due to the uncertainty in the input variables, both singly and in combination with one another. Variance based SA techniques are intended to estimate how much ouput variability is dependent on each of the input variables (again, taken singly and in combination with one another). This note deals with a recently developed method for global sensitivity analysis of model output: Sobol' sensitivity indices. The 100 G. E. B. ARCHER et al. method is based on decomposing the variance of model output into terms of increasing dimensionality, as in the classical Analysis of Variance (ANOVA) of factorial experimental designs. Monte-Carlo (MC) techniques are used to expedite their construction, and a bootstrap method is presented which produces reliable interval 5 1 estimates for the true, unknown index values. The use of the bootstrap 0 2 ril increases the usefulness of the indices by reducing the computational p effort required to estimate their variability. A 2 The layout of the paper is as follows. In Section 2 we introduce the 2 8 Sobol' indices, and discuss briefly the history of the ANOVA- 1 0: decomposition, and explain how to construct the indices using MC 0 at methods. In Section 3 the bootstrap method for interval estimation is n] explained, and Section 4 presents a numerical example of all the work. e rg Section 5 contains the conclusions drawn from the experiments, and e B ideas for future work. et i k e ot bli 2. THE SOBOL' SENSITIVITY INDICES bi s et 2.1. Motivation sit r ve The sensitivity indices described in this note were developed by Sobol' Uni (1990a), based on his earlier work on the Fourier Haar series (1969). [ y The indices were developed for the purpose of Sensitivity Analysis b d (SA), that is, to estimate the sensitivity of a function f (X) with respect e d to different variables or subgroups of variables. In SA terminology, a o nl Y = f (X) is the (possibly vector valued) output variable, while the X w o are the input variables. The method is outlined briefly: D Let the function Y = f (X) = f (XI . . . Xn) be defined on the n- dimensional unit cube It is possible to decompose f(X) into summands of increasing dimensions: SENSITIVITY MEASURES 101 provided that fo is a constant and the integral of every summand over any of its own variables is zero: 5 1 0 Consequences of (1) and (2) are that all the functions which appear 2 ril within the summands in (1) are orthogonal, and thatfo = J,, f (X)dX. p A In Sobol' (1969) the representation (1, 2) is constructed from 2 consideration of Fourier Haar series; in his 1990 article a more 2 8 general developement is offered. Iff (X) is integrable in the unit cube, 1 00: then all of the functions which appear within the summands in (1) are at also integrable, as follows: ] n e g r e B et i k e ot bli bi s and so on, where the convention is used that dX-{ij,,,,) indicates et sit integration over all variables with the exception of those within the r ve subscript parenthesis. These integrals will be remarked upon in Sec- ni U tion 2.2, where we explore the history of this form of decomposition. y [ The total variance off (X) can be written as D = J,, f2(x)dx- b d f &while e d a o nl w o D is the contribution to the total variance from term ft,,,,isinth e series development. At this point the sensitivity estimates Si,,,,i, can be introduced: In Sobol' (1990a) it is shown that the total variance can be partitioned in the same way as the original function: 102 G. E. B. ARCHER et al. from which it follows that the sum of all the sensitivity indices-over all possible combinations of indices-must be 1. We write this as X'' Si,.,,i=3 1. This decomposition is useful for SA because the terms Si,,.,gisiv e the fraction of the total variance of f(X) which is due to any individual 5 1 input variable or combination of input variables. In this way, for 0 2 ril example, S1 is the main effect of varaible XI, S12 is the interaction p effect, i.e., that part of the output variation due to variables X1,X2 A 2 which cannot be explained by the sum of the effect of the two 2 8 variables alone. Finally, the last term S12...i,s that fraction of the 1 0: output variance which cannot be explained by summing terms of lower 0 at order. ] n This decomposition is not unique in the analysis of numerical e g r experiments (see next section): a variance decomposition identical to e B (5) is suggested by Cukier et al. (1978) when using the Fourier et i Amplitude Sensitivity Test (FAST) method, based on the Fourier k ote transform, for sensitivity analysis. The FAST indices are identical to bli the Sobol' ones in all but computation (Saltelli & Bolado, 1996); bi s however their calculation is usually limited to only those indices which et sit refer to "main effect", that is, individual Xi terms (Liepman & r e Stephanopoulos, 1985). v ni Decompositions similar to (1) and (4) are discussed in Cotter (1979), U y [ Cox (1982), Efron and Stein (1981), and Sacks et al. (1989). In this last b d article the plots of the individual f terms in the development (1) are e d used for the purpose of SA. a o nl Other investigators (Iman & Hora (1990), Krzykacz (1990), Saltelli w et al., (1993), McKay (1995)) have developed sensitivity measures o D (called importance measures or correlation ratios) which are also based on the fractional contribution of the total variance of individual input variables. In an SA exercise, these measures would produce the same importance ranking as that gained by consideration of the single term Si values. For a discussion of the various measures, see also Homma & Saltelli (1994). Practically, in order to apply Sobol' sensitivity estimates one must evaluate the multidimensional integrals (such as Equation (3)) using MC methods. Each term in the series development (1) is a separate integral, and the number of terms is equal to 2" 1, far too many to be - computed even for moderate model dimension n. An MC technique SENSITIVITY MEASURES 103 designed to obviate this difficulty - reducing the number of MC + calculations to n 1- is explained in Section 2.3. In fractionally replicated designs it is customary to assume that higher order interactions are zero, in order to leave sufficient degrees of freedom for variance estimation, but in SA experiments, where the 5 models are usually nonlinear and the variation in the response much 1 20 wider, it may happen that the higher order terms are the most ril important (for example, see Saltelli & Sobol' (1995)) and so their p A estimation is crucial. These sensitivity indices allow such effects to be 2 2 estimated easily, by partitioning X, and treating subsets of variables as 8 1 new variables; for example, X could be partitioned into (U, V), where 0: at 0 U = (XI . . . Xk), and V (Xn++l ,...,X ,). The variance of f (X) can ] then be decomposed into D = DU+ Dv+ DUV. Using this results, we n ge define a "total effect" term for each variable, by letting U=Xi, Ber V = (XI,.. . , Xi-l, Xi+l . . . X,) and declaring the total effect of vari- et i able i to be given by k e ot bli bi s et In this way the total contribution of each variable to the output variation sit is estimated. r e v The general conclusion to this subsection is that the Sobol' ni U formulation of the sensitivity indices is very general and includes as [ y a particular case most of what has been done previously in SA, using b d decompositions like (3) or (5), as well as those sensitivity measures e d a based on fractional contributions to the output variance. Sobol' o nl indices of the first order are identical to FAST coefficients, and differ w o from the other tests mentioned above only by a scale factor. Further, D Sobol' indices allow the interaction and higher order interaction terms to be computed straightforwardly; this capacity makes Sobol' sensiti- vity analysis similar to the Analysis of Variance in factorial design, as shall be discussed in the next section. 2.2. The ANOVA Decomposition The decomposition (1) or (5) has a long history which, in this section, is examined, to help endow the Sobol' indices with their true statistical definitions. The decomposition has an interesting pedigree which links 104 G. E. B. ARCHER et al. naturally to U-statistics and resampling methodology. The lemma which describes the decomposition in its most general form is given in Efron & Stein (1981), and concerns functions of independent random variables (Xi) which are not necessarily identically distributed, although when they are, the result can be stated more concisely. 5 Suppose, as above, f(X1,. . . , X,) is some statistic defined on the 01 product measure generated by X = (XI,.. . , X,). Then f (X)m ay be 2 ril decomposed into a grand mean fo = E [f( X)], i' th main effect p A J;:(Xi)= E [f( XIXi = xi)]- fo; ij'th interaction Jl(Xi,X j) = E [f( X)I 2 2 Xi=~i,~=~j]-E[f(X)IXi=~i]-E[f(X)IXj=~ja]n+d fSoO, 8 1 on. Given these definitions, the decomposition in Section 2.1 follows 0: 0 immediately, as the case n = 2 easily demonstrates: at ] n e g r e B et i k e ot bli bi s et Using the law of iterated expectations. it is straightforward to see all sit er the random variables on the right hand side have zero mean and are v ni mutually uncorrelated. For example E [J;:(Xi)=] E [E{ f (XIX i = xi) U w1 y [ -h)l = Elf -fo = 0. 0 b This decomposition of a statistic is the same as is typically deployed d e for data collected from a factorial experiment (Fisher, 1958). One of d a o our aims is to discuss the similarities and the differences between SA nl w and ANOVA. o D One of the earliest to write about this lemma was Hoeffding (1948), who was concerned with estimators f (X)w hich are U-statistics. Such a U-statistic is defined as where, as in Section 2.1, the sum in the numerator is carried out over all permutations (al,. . . , a,) of m different integers. If the {Xi) are independent and identically distributed (i.i.d.) according to some distribution function G, then U is an unbiased estimator of SENSITIVITY MEASURES 105 8(G) = J . . . Jf (xl, . . . , xm)dG(xl). . . dG(xm). In other words U is the average of all the possible values off (X,, , . . . , Xam)d rawn from the realisation of (XI, . . . , X,), m < n. This will strike chords with readers who are practioners of resampling methodologies (start with Efron 1979), which have become steadily more popular as cheap and 5 powerful computing facilities become more widely available. In 1 0 bootstrapping, resamples are drawn from the full n set of data 2 ril (xl, . . . , x,), while the U-statistics are even more redolent of the p A jackknife (Chapter 5 of Efron & Tibshirani (1993)), which estimates 2 2 functionals of G by forming a weighted average of the set of estimates 8 1 obtained by deleting one (or more) data points at a time. 0: 0 Both the jackknife and the bootstrap use as a rationale the ] at substitution of the sample distribution function G (which places n e probability mass lln on each xi) for the unknown population g r e distribution G. Substituting in the function 8(G) above gives B et i k e ot bli bi s et Hoeffding shows that O(G)i s a linear function of U-Statistics with rsit E [B(G)]= 0(G) +0(nP1) and therefore B(G) -+ 8(G) as n -+ m. This e v fact-that as sample size increases, so an estimator calculated on a ni U random sample will tend towards the population parameter-justifies [ y the bootstrap also. Interestingly enough, iff is a linear statistic, i.e., all b d terms on the right hand side of the decomposition are zero apart from e d a those which only involve individual Xi,t hen the jackknife and o nl bootstrap estimates of 0 agree, whilst if there are higher-order effects w o then the bootstrap estimates are more accurate. D Much of the rest of Hoeffding's rich paper is a mathematical tour- de-force, in which it is shown that many commonly used statistics, such are the rank correlation coefficient, are examples of U-statistics. The most important result is that when the {Xi) are i.i.d. and f is a plug-in estimator (that is, it does not depend on n), then asymptotically fi(U - 0) has a normal distribution. The relationship between U-statistics and the jackknife is explored in Efron & Stein (1981), who consider the case where C EG( Xi) = and var~(Xi=) a2vi 106 G. E. B. ARCHER et al. and show that the plug-in estimator of sample variance 1 " x=nC S(Xl,...,Xn)=-E(Xi-T)', where Xi i=l i= 1 (y) 5 has grand mean: fo = 2, 201 main effects:J ;:(Xi) = 9[( xi - C) 2 - o 2], Vi, pril and pairwise interaction effects: Jij(Xi, 4)= 5 (xi - C) ( ~-j C ), V(i,j ) A and all higher order terms are zero. 2 8 2 A more interesting application to SA is the realisation that the 0:1 decomposition (I), which powers the Sobol' indices, is exactly the same at 0 as that used to construct an ANOVA of the results of a factorial ] experimentation. In this sense, ANOVA is SA, and SA is ANOVA. n e g For consider a response variable X, measured under different r e B conditions of two factors A (with I levels) and B (with J levels). Each et i particular combination of factors is replicated K times, and so Xqk is k e the k'th replicate of the experiment at the (I, J)'th level of factors A ot bli and B. (Attention is restricted to the case of the fully replicated, two bi factor design only for ease of discussion). In statistical terminology, A s et and B are factors, Xis a response variable; in SA, A and B are input rsit variables, and X is the output variable. But the Sobol' indices, to e v ni measure the importance of A, B, or their interaction (written A B) uses U exactly the same decomposition as the ANOVA F-tests of significance. [ y b This is seen if we write down the total sum of squares about the mean, d e used in ANOVA to perform the F-test. We have: d a o nl w o D (A dot in the subscript indicates that the average has been taken over that index.) In comparison with (5), the first two terms on the right hand side correspond to the single D terms (N= 2 here), with the third right hand side term corresponding to D12.T he final term on the right