StatisticalScience 2007,Vol.22,No.3,401–406 DOI:10.1214/07-STS239 (cid:13)c InstituteofMathematicalStatistics,2007 Multivariate Meta-Analysis: Contributions of Ingram Olkin Betsy Jane Becker 8 0 Abstract. Theresearchonmeta-analysis andparticularly multivariate 0 meta-analysis hasbeengreatly influencedbythework ofIngramOlkin. 2 This paper documents Olkin’s contributions by way of citation counts n and outlines several areas of contribution by Olkin and his academic a descendants. An academic family tree is provided. J 8 Key words and phrases: Meta-analysis, multivariate. 2 ] E 0. INTRODUCTION univariate case—one endpoint per study. Such end- M points can be represented by correlations, mean dif- Much of the research on statistical methods for ferences, proportions, odds ratios (or log odds) and t. meta-analysis in the last three decades has been in- a even observed probabilities. fluenced by Ingram Olkin, either through his direct t s Thefirstof Olkin’s contributions tometa-analysis contributions or through the work of his students [ (Hedges and Olkin, 1980) examined the intuitively andtheiracademicdescendants.Weindicatetheex- 1 appealing vote-counting methods used in many re- tent of this influence and present a tree of Olkin’s v search syntheses and traditional literature reviews. 7 academic descendants who have made, or are mak- Votecountingentailscountingthenumberofstudies 0 ing, contributions to research in meta-analysis. We 2 that have statistically significant results in support then consider the outcome metrics that have been 4 of, and counter to, a particular hypothesis, as well . used in the multivariate meta-analysis context and 1 as those with nonsignificant results. The category brieflyreviewkey resultsforeach metric,thusshow- 0 with the most votes (or more than some specific 8 ing Olkin’s seminal influence on this important sub- proportion of votes) “wins” and the set of all re- 0 field of meta-analysis. : sults is then characterized as supporting that view v (e.g., if half of the studies have significant tests in i 1. OLKIN’S INFLUENCE ON META-ANALYSIS X favor of a hypothesis, the studies are viewed as sup- r Meta-analysis is a set of methods for combining porting the hypothesis). Hedges and Olkin showed a and analyzing results from series of related stud- that the statistical properties of this approach were ies. Glass (1976) coined the term “meta-analysis,” problematic—in that more evidence can lead to but the idea of summarizing study results is much poorer decisions. older, with references dating to the turn of the last Since then Olkin has authored or co-authored century (e.g., Pearson, 1904). Much of the litera- 39 more articles or book chapters and one ture on methods for meta-analysis deals with the book on meta-analysis. The influence of his work is shown by the fact that these documents Betsy Jane Becker is Professor of Measurement and have generated over 5600 citations. (Based on Statistics, College of Education, Florida State searches of the Web of Science at University, Tallahassec, Florida 32306, USA e-mail: http://80isi4.isiknowledge.com.proxy.lib.fsu.edu/ [email protected] using the author names “Olkin I*,” “Hedges L*,” “Gleser L*” and “Sampson A*.”) His book Statis- This is an electronic reprint of the original article tical Methods for Meta-analysis with Larry Hedges published by the Institute of Mathematical Statistics in Statistical Science, 2007, Vol. 22, No. 3, 401–406. This (Hedges and Olkin, 1985) is something of a cita- reprint differs from the original in pagination and tion classic, having been cited at least 3270 times. typographic detail. However, Olkin’sarticles andbookchapters arealso 1 2 B. J. BECKER highly cited, with the number of citations per work 2. CONTRIBUTIONS TO META-ANALYSIS OF OLKIN’S ACADEMIC DESCENDANTS rangingfrom0to916withameancountof62.5cita- tions (SD=157.4) and a median count of 20.5 cita- BesidesOlkin’sowncontributionstometa-analysis, tionsperarticle.[Asistypicalofcitation counts,the individuals that he has mentored and trained have also made many contributions to this literature— distribution of citation counts per article is highly some writing dissertations on meta-analysis topics. skewed (skewness coefficient = 4.8), suggesting that (ApologiesaremadetoanystudentsofOlkinandhis the median citation count per article is the more descendants who have inadvertently been omitted appropriate measure of central tendency.] The ma- from this analysis.) All first-generation descendants were students at StanfordUniversity, though not all jority of this work is collaborative—30 of these pa- earned degrees in the Department of Statistics. In pers are co-authored, with the mean number of co- addition, students of those students are considered, authors across all 39 documents being 2.74 (SD= and so on, through several generations of Olkin aca- 3.5). As might be expected from the recipient of the demic “descendants.” These individuals are displayed in Figure 1, the Elizabeth L. Scott Award from the Committee of Olkinmeta-analytic family tree. Theyears shownin PresidentsofStatisticalSocieties(in1998),overhalf the figure are the graduation dates for each person; (17) of Olkin’s 30 co-authored works were written dissertations concerning meta-analysis methods are with at least one female co-author. included in the reference list as well. The tree shows Fig. 1. The Olkin meta-analytic family tree. 3 MULTIVARIATEMETA-ANALYSIS on the bottom-most branches three former students exhibit subtle dependencies because of common in- of Olkin who wrote dissertations on meta-analysis. strumentation, treatments and the like, their out- They are Hedges (1980), Holmgren (1989) and Yen comesdonothaveacorrelationstructurethatiseas- (1997). In addition, three other former students of ily characterized. Hedges and Olkin (1985) firstpre- Olkinareshown.Gleser,PerlmanandSampsoneach sented methods for dealing with multivariate data have contributed to the literature on meta-analysis in meta-analysis. Their Chapter 10 dealt with stan- or synthesis of results, though none wrote a disser- dardized mean differences that are dependent be- tation on the topic. Relevant works include Gleser cause p (dependent)responsevariables are observed and Olkin (1994, 1996), Koziol and Perlman (1978) within each primary study. and Olkin and Sampson (1998), among others. We denote the results as T , where i indexes the ij The next set of leaves shows students of Olkin’s study and j the outcome. Across studies we may students—perhaps we can call these Olkin’s meta- have analytic “grandchildren.” Here are listed seven who T11 ... T1p wrotedissertations on meta-analytic methods.Abu- T21 ... T2p Libdeh (1984), Becker (1985), Champney (1983), . .  . .  Konstantopoulos (2003), Pigott (1992) and Zhang  . .    (1993)weredissertationswrittenbystudentsofHed- Ti1 ... Tip   ges, and Sylvester (2001) and Sezer (2006) were dis-  .. ..   . .    sertations directed by Gleser. Two other students of Tk1 ... Tkp Hedges (Vevea and Friedman) have contributed to for k studies and upto p outcome indices. The p de- the literature on meta-analytic methods after com- pendent indices arise when p response variables are pleting a dissertation using meta-analytic methods observed, when contrasts are dependent (e.g., com- or on another topic (e.g., Friedman, 1989, 2000; Hedges and Vevea, 1996, 1998). mon controls, multiple proportions), when multiple Finally we reach the currentends of the branches. indices involve each response variable (e.g., correla- Six additional students are listed who worked with tion matrices), and when multivariate analyses ap- Becker on meta-analytic methods (Chang, 1992; pear within a primary study. The possible metrics Chiu, 1999; Cho, 2000; Fahrbach, 2001; Schram, include multivariate standardized mean differences, 1996;Wu,2006)andtwowhowerestudentsofVevea correlations and proportions (or odds ratios). Each and who have either written dissertations on meta- such metric will be considered in turn. analysis methods (Hafdahl, 2001) or contributed to themeta-analyticliterature(VeveaandWoods,2005; 4. MULTIVARIATE STANDARDIZED MEAN Woods et al., 2002) while writing a dissertation on DIFFERENCES a different topic. We can be assured that others will This metric may be the most thoroughly inves- follow. tigated of all those for which multivariate analy- ses have been proposed. Gleser and Olkin (1994) 3. OVERVIEW OF MULTIVARIATE dealt with multiple treatment studies and multiple META-ANALYSIS endpoint studies for standardized mean differences. We next turn to the topic of multivariate meta- Some studies combine both of these multivariate as- analysis and explore Olkin’s fundamental contribu- pects.Evidencethatmultivariateeffect-sizedataare tions to this domain. (See Becker, 2000, and van common is found in the fact that Gleser and Olkin Houwelingen et al., 2002, for overviews of the topic (1994)hasbeencitedover100times,infieldssuchas of multivariate meta-analysis.) Multivariate meta- psychology, education, medicine, ecology and crim- analysis occurs when more than one (dependent) inal justice. Similarly, an early paper by Rauden- outcome is measured in a study. This can occur bush, Becker and Kalaian (1988) dealt with multi- when subjects are measured on several outcomes or variate standardized-mean-difference data. atseveraltimepoints(multipleendpointstudies),or 4.1 Multiple Treatment Studies whenstudyindicesarecomputedusingsharedtreat- mentorcontrolgroups(multipletreatmentstudies). Multiple treatment studies are illustrated here Thesecases donot typically includestudies with re- with an example of studies with a common control sults for multiple samples. While such samples may group. Further elaborations of this scenario (e.g., 4 B. J. BECKER with three or more treatment groups or multiple relevantreferencesincludeArends,VokoandStijnen controlgroups)leadtomoreoutcomes,buttheprin- (2003) and Nam, Mengersen and Garthwaite (2003) ciples underlying these methods can be illustrated which concern analyses of multiple log-odds ratios. with this simplest scenario. Additional forthcoming work will undoubtedly ad- Supposeastudyhastwotreatmentgroups,T1 and dress this issue. T2, and one control group C. Then if we define X¯A to represent the mean of group A and S to be the 6. MULTIVARIATE CORRELATIONS AND pooled within-groups standard deviation across all SLOPES groups, we can compute The topic of synthesis of correlation matrices has T1=(X¯T1 −X¯C)/S and T2=(X¯T2 −X¯C)/S seen increasing activity in the past few years. This increase in interest is likely related to the increas- for each study. If we index these outcomes as Ti1 ingly complex models investigated in primary re- and Ti2 with i for the ith study, we will have search, at least in the social sciences. Researchers T11 T12 want to be able to statistically model the effects of T21 T22 multiple predictors as well as to control for poten-  .. ..  tial confounding variables, and this is done by in-  . .    cluding such variables in complex models. Results Ti1 Ti2 of such techniques as structural equation modeling,  .. ..   . .  factor analysis and multiple regression have often   Tk1 Tk2 been omitted from meta-analyses because of a lack ofmethodsforsynthesizingindicesfromtheseanaly- whichhasamultivariatestructure.GleserandOlkin ses.While Olkin has not contributed directly to this (1994) gave two formulas for Cov(T ,T ) for mul- ij ij′ area of synthesis methods, his work is fundamental tiple treatment studies. More recent work by Cook because most of the analyses proposed to date are (2004) presents a formula tailored to small-sample asymptoticandrelyonthelarge-sampledistribution cases. theory presented by Olkin and Siotani in 1967. 4.2 Multiple Endpoint Studies Themultivariateworkinthisrealmofmeta-analysis has involved the synthesis of correlation matrices, Gleser and Olkin (1994) also cover dependence of and the use of those summaries in further modeling standardized mean differences due to multiple re- of linear models, structural equation models, and sponse variables (expanding on Hedges and Olkin, even factor analysis (G. Becker, 1996). B. Becker 1985). If we define T to represent an effect size for ij andhercollaborators (B.Becker,1992,1995;Becker outcome measure j (j=1 to p) in study i, we have and Fahrbach, 1994; Becker and Schram, 1994) be- Tij =(Y¯iTj −Y¯iCj )/Sij gan this stream of work by presenting methods for the synthesis of correlation matrices, specifically es- for i = 1 to k studies and j = 1 to p measures. timates of mean matrices under fixed- and random- This was labeled the multiple endpoint design. The effects models and tests of the homogeneity of the effect-size data structure is identical to that shown seriesofmatrices underreview.Atroughlythesame above but the covariances between the multiple ef- time, applications of like methods appeared in the fects from each study differ from those in the mul- personnelpsychologyliterature(e.g.,Schmidt,Hunter tiple treatment case. andOuterbridge,1986).Beckeralsopresentedmeth- ods for estimating linear models based on the mean 5. MULTIVARIATE PROPORTIONS correlationmatricesandtestingcomponentsofthose Lesshasbeenpublishedonthemultivariate meta- composite models. Others have pursued this work analysis of proportions. One contribution is Gleser andinvestigatedtheuseofmeanmatriceswithstruc- and Olkin’s (2000) chapter on multiple treatment tural equation modeling software (e.g., Cheung and studies with outcomes expressed as two-by-two ta- Chan,2005;FurlowandBeretvas,2005).Allofthese bles. Gleser and Olkin present large-sample gener- works rely on the fundamental result derived by alized least squares methods for dealing with risk OlkinandSiotani(1976,page238) ofthecovariance differences, log odds ratios, and arcsine transformed among correlations from a single sample. Specifi- proportions from multiple treatment studies. Other cally, the large-sample covariance, σ , between ist,iuv 5 MULTIVARIATEMETA-ANALYSIS population correlations ρist and ρiuv within study i Becker, B. J. (1995). Correction to “Using results from is replicated studies to estimate linear models.” J. Educa- tional and Behavioral Statistics 20 100–102. 2 2 2 2 σ =[0.5ρ ρ (ρ +ρ +ρ +ρ ) Becker, B. J. (2000). Multivariate meta-analysis. In Hand- rist,riuv ist iuv isu isv itu itv book of Applied Multivariate Statistics and Mathematical +ρisuρitv+ρisvρitu Modeling (H. E. A. Tinsley and S. D. Brown, eds.) 499– −(ρ ρ ρ +ρ ρ ρ 525. Academic Press, San Diego, CA. MR1823752 ist isu isv its itu itv Becker, B. J. and Fahrbach, K. (1994). A comparison of +ρ ρ ρ +ρ ρ ρ )]/n , approaches to the synthesis of correlation matrices. Paper ius iut iuv ivs ivt ivu i presented at the annual meeting of the American Educa- wheren isthesamplesizeinstudy iand s,t,u and tional Research Association, New Orleans, LA. i v index the variables within study i that are corre- Becker, B. J. and Schram, C. M. (1994). Examining ex- planatorymodelsthroughresearchsynthesis.InTheHand- lated. That is, ρ is the correlation between vari- ist book of Research Synthesis (H. M. Cooper and L. V. ables X and X within studyi.This resultwas also s t Hedges,eds.)357–381.RussellSageFoundation,NewYork. used by Hafdahl (2001) who examined exploratory Becker,G.(1996).Themeta-analysisoffactoranalyses:An factor analysis methods based on synthesized ma- illustration based on the cumulation of correlation matri- trices, and papers by Olkin and other collaborators ces. Psychological Methods 1 341–353. Champney, T. F. (1983). Adjustments for selec- (e.g., Olkin and Finn, 1976, 1990; Olkin and Saner, tion/publication bias in quantitative research. Ph.D. 2001) also rely on this fundamental result. dissertation, Univ.Chicago. Chang,L.(1992).Apoweranalysisofthetestofhomogene- 7. CONCLUSION ity in effect-size meta-analysis. Ph.D. dissertation, Michi- gan StateUniv. It is safe to say that much of the work on meta- Cheung, M. W. L. and Chan, W. (2005). Meta-analytic analysis, and especially multivariate issues in meta- structural equation modeling: A two-stage approach. Psy- analysis, has its genesis in the contributions of In- chological Methods 10 40–64. Chiu, C. W.-T. (1999). Scoring performance assessments gram Olkin. The review of research in this paper based on judgements: Utilizing meta-analysis to estimate shows the significant impact of Olkin’s work. The variance components in generalizability theory for unbal- familytreeillustratesthatcontributionsfromOlkin’s ancedsituations.Ph.D.dissertation,MichiganStateUniv. academic descendants are numerous and will con- Cho, J. M.(2000).Hypothesesandlimitingdistributionsfor tinue to be forthcoming. p-value summaries to compare studies in meta-analysis. Ph.D. dissertation, Michigan StateUniv. Cook,S.R.(2004).Anoteontestingforhomogeneityamong ACKNOWLEDGMENTS effect sizes sharing a common controlgroup. Psychological Methods 9 446–452. This work was supported by National Science Fahrbach, K. R. (2001). An investigation of methods for FoundationGrantsREC-0335656andREC-0634013. mixed-modelmeta-analysisinthepresenceofmissingdata. Parts of this work were presented in the sympo- Ph.D. dissertation, Michigan StateUniv. sium “Multivariate Analysis: In Celebration of In- Friedman, L. (1989). Mathematics and the gender gap: A gramOlkin’s80thBirthday,”attheJointStatistical meta-analysisofrecentstudiesonsexdifferencesinmathe- maticaltasks.Review ofEducational Research 59185–213. Meetings, Toronto, Canada, August 2004. Friedman, L. (2000). Estimators of random effects variance components in meta-analysis. J. Educational and Behav- ioral Statistics 25 1–12. REFERENCES Furlow, C. F. and Beretvas, S. N. (2005). Meta-analytic Abu-Libdeh,O.I.(1984).Strengthofassociationinthesim- methods of pooling correlation matrices for structural ple general linear model: A comparative study of Hays’ equationmodelingunderdifferentpatternsofmissingdata. omega-squared. Ph.D. dissertation, Univ.Chicago. Psychological Methods 10 227–254. Arends, L. R., Vokoˇ, Z. and Stijnen, T. (2003). Com- Glass, G. V. (1976). Primary,secondary,and meta-analysis bining multiple outcome measures in a meta-analysis: An of research. Educational Researcher 5 3–8. application. Statistics in Medicine 22 1335–1353. Gleser,L.J.andOlkin,I.(1994).Stochasticallydependent Becker, B. J. (1985). Applying tests of combined signifi- effect sizes. In The Handbook of Research Synthesis (H. cance:Hypothesesandpowerconsiderations.Ph.D.disser- M. Cooper and L. V. Hedges, eds.) 339–355. Russell Sage tation, Univ.Chicago. Foundation,New York. Becker, B. J. (1992). Using results from replicated studies Gleser, L. J. and Olkin, I. (1996). Models for estimating toestimatelinearmodels.J.EducationalStatistics 17341– the number of unpublished studies. Statistics in Medicine 362. 15 2493–2507. 6 B. J. BECKER Gleser, L. J. and Olkin, I. (2000). Meta-analysis for 2×2 and Statistics Chapter 16 (S. Ikeda, ed.) 235–251. Shinko tableswithmultipletreatmentgroups.InMeta-Analysisin Tsusho, Tokyo. MR0603847 MedicineandHealthPolicy (D.A.BerryandD.K.Stangl, Pearson, K.(1904).Reportoncertainentericfeverinocula- eds.) 339–355. Dekker,New York. tions. British Medical Journal 2 1243–1246. Hafdahl, A. R. (2001). Multivariate meta-analysis for Pigott, T. D. (1992). The application of normal maximum exploratory factor analytic research. Ph.D. dissertation, likelihoodmethodstomissingdatainmeta-analysis.Ph.D. Univ.North Carolina at Chapel Hill. dissertation, Univ.Chicago. Hedges, L. V.(1980).Combiningtheresultsofexperiments Raudenbush, S. W., Becker, B. J. and Kalaian, H. using different scales of measurement. Ph.D. dissertation, (1988). Modeling multivariate effect sizes. Psychological Stanford Univ. Bulletin 103 111–120. Hedges,L.V.andOlkin,I.(1980).Vote-countingmethods Schmidt, F., Hunter, J. E. and Outerbridge, A. N. in research synthesis. Psychological Bulletin 88 359–369. (1986). Impactofjob experienceandability onjob knowl- Hedges, L. V. and Olkin, I. (1985). Statistical Methods for edge,work sample performance and supervisory ratingsof Meta-Analysis. AcademicPress,Orlando,FL.MR0798597 job performance. J. Applied Psychology 71 432–439. Hedges, L. V. and Vevea, J. L. (1996). Estimating effect Schram, C. M. (1996). A comparison of methods for cor- size under publication bias: Small sample properties and robustness of a random effects selection model. J. Educa- recting multivariate data for attenuation with application tional and Behavioral Statistics 21 299–332. to synthesizing correlation matrices. Ph.D. dissertation, Hedges,L.V.andVevea,J.L.(1998).Fixed-andrandom- Michigan StateUniv. effects models in meta-analysis. Psychological Methods 3 Sezer, A. (2006). Reporting uncertainty by spline function 486–504. approximation of log-likelihood. Ph.D. dissertation, Univ. Holmgren,E.B.(1989).Utilizingp–pplotsinmeta-analysis Pittsburgh. as general measures of treatment effects. Ph.D. disserta- Sylvester,M.A.,II.(2001).Estimationofacommonmean tion, Stanford Univ. through a series of similar interlaboratory experiments. Konstantopoulos, S.(2003).Variancecomponentsestima- Ph.D. dissertation, Univ.Pittsburgh. tioninmeta-analysisemployingmaximumlikelihoodmeth- van Houwelingen, H. C., Arends, L. R. and Stijnen, ods. Ph.D. dissertation, Univ.Chicago. T. (2002). Tutorial in biostatistics. Advanced methods in Koziol, J. A. and Perlman, M. D. (1978). Combining in- meta-analysis:Multivariateapproachandmeta-regression. dependent chi-squared tests. J. Amer. Statist. Assoc. 73 Statistics in Medicine 21 589–624. 753–763. Vevea,J. L.andWoods, C. M.(2005).Publicationbiasin Nam, I. S., Mengersen, K. and Garthwaite, P. (2003). researchsynthesis:Sensitivityanalysisusingaprioriweight Multivariatemeta-analysis.StatisticsinMedicine222309– functions. Psychological Methods 10 428–443. 2333. Woods, C. M., Vevea, J. L., Chambless, D. L. and Olkin, I. and Finn, J. D. (1976). Correlations redux. Psy- chological Bulletin 118 155–164. Bayen, U. J. (2002). Are compulsive checkers impaired in memory? A meta-analytic review. Clinical Psychology: Olkin, I.andFinn, J. D.(1990).Testingcorrelatedcorrela- tions. Psychological Bulletin 108 330–333. Science and Practice 9 353–366. Olkin, I. and Sampson, A. (1998). Comparison of meta- Wu, M. J. (2006). Methods of meta-analyzing regression analysis versus analysis of variance of individual patient studies: Applications of generalized least squares and fac- data. Biometrics 54 317–322. MR1626809 toredlikelihoods.Ph.D.dissertation,MichiganStateUniv. Olkin,I.andSaner,H.(2001).Approximationsfortrimmed Yen, J. H.(1997). Robustestimation ofeffectsizes inmeta- Fisherproceduresinresearchsynthesis.StatisticalMethods analysis. Ph.D. dissertation, Stanford Univ. in Medical Research 10 267–276. Zhang,J.(1993).Statisticalsignificancepublicationbias,its Olkin,I.andSiotani,M.(1976).Asymptoticdistributionof determinationandstatisticaladjustmentsinmeta-analysis. functions of a correlation matrix. In Essays in Probability Ph.D. dissertation, Univ.Chicago.