DOCUMENT RESUME ED 445 076 TM 031 753 AUTHOR Hines, Joy L.; Foil, Carolyn R. TITLE Covariance Corrections: What They Are and What They Are Not. PUB DATE 2000-01-28 NOTE 27p.; Paper presented at the Annual Meeting of the Southwest Educational Research Association (Dallas, TX, January 27-29, 2000) . PUB TYPE Reports Descriptive (141) -- Speeches/Meeting Papers (150) EDRS PRICE MF01/PCO2 Plus Postage. DESCRIPTORS *Analysis of Covariance; *Research Methodology ABSTRACT Researchers are often faced with the dilemma of having to use intact groups. Analysis of covariance (ANCOVA) is sometimes used as a means of leveling unequal groups. However, only after five assumptions have been met is it safe to use ANCOVA. Part and partial correlations also invoke statistical correlation procedures. This paper reviews these covariance correction procedures, explains the relevant assumptions and potential pitfalls related to the analysis, and discusses research situations when their use is warranted or ill-advised. (Contains 1 figure, 5 tables, and 13 references.) (Author/SLD) Reproductions supplied by EDRS are the best that can be made from the original document. Covariance Corrections 1 Running Head: COVARIANCE CORRECTIONS Covariance Corrections: What They Are and What They Are Not Joy L. Hines and Carolyn R. Foil The University of Southern Mississippi U.S. DEPARTMENT OF EDUCATION PERMISSION TO REPRODUCE AND Office of Educational Research and Improvement DISSEMINATE THIS MATERIAL HAS EDUCATIONAL RESOURCES INFORMATION BEEN GRANTED BY CENTER (ERIC) IZI,Ahis document has been reproduced as received from the person or organization ,1_64 1-1-k yve s originating it. Minor changes have been made to improve reproduction quality. TO THE EDUCATIONAL RESOURCES Points of view or opinions stated in this INFORMATION CENTER (ERIC) document do not necessarily represent 1 official OERI position or policy. O 2 Paper presented at the annual meeting of the Southwest Educational Research Association, Dallas, TX, January 2%, 2000. BEST COPY AVAILABLE Covariance Corrections 2 Abstract Researchers are often faced with the dilemma of having to use intact groups. ANCOVA is sometimes employed as a means of leveling unequal groups. However, only after five assumptions are met is it safe to use ANCOVA. Part and partial correlations also invoke statistical correction procedures. The purpose of the present paper is to (a) review these covariance correction procedures, (b) explain the relevant assumptions and potential pitfalls related to the analyses, and (c) discuss research situations when their use is both warranted and ill-advised. Covariance Corrections 3 Covariance Corrections: What They Are and What They Are Not Researchers often find themselves at the mercy of intact groups (e.g., classrooms, schools, etc.) that limit, if not eliminate, the possibility of conducting true, randomized experimental research (Henson, 1998). For example, given a particular intervention for special education instruction, it is arguably unethical to randomly assign students into treatment and control group participants. Accordingly, there remains a great need for experimental validation of educational programs in general (cf. Welch & Walberg, 1974). One result of this dynamic is the tendency (perhaps necessity) to either (a) use existing groups (e.g., two special education classrooms) and attempt to statistically control for preexisting differences between groups or (b) use one treatment group and attempt to statistically control for preexisting variables that may impact the final outcome(s). These types of analyses are all related and go by various names such as analysis of covariance (ANCOVA) and part and partial correlation. Covariance corrections vary in their frequency of use. ANCOVA, for example, appeared in about 4% of the published literature in American Educational Research Journal articles (Goodwin & Goodwin, 1985). More recently, Baumberg and Bangert (1996) reported that the use of ANCOVA accounted for 7% of the statistics used in the research found in the Journal of Learning Disabilities for 1989 through 1993. Similarly, in a review of 17 well-respected education and psychology journals, Keselman and colleagues (1998) found that univariate ANCOVA was used in 7 % of the articles examined (45 out of 651). Furthermore, this hit rate does not include applications of regression analyses with covariates or multivariate ANCOVA. Thompson (1988, 1994) found frequent use among doctoral students. Furthermore, it is arguable that other related corrections, such as part and partial correlation, are more frequently utilized in Covariance Corrections 4 the literature, particularly when researchers attempt to control for variables that may influence the outcome of a single treatment group, such as using a pretest correction on posttest scores. Like most analytic methods, it is important to realize that these corrections have certain benefits and limitations. ANCOVA, for example, is generally inappropriate when used with intact groups such as established classrooms of students. And yet, ANCOVA does continue to be utilized in such situations. Of the 45 uses of ANCOVA reported, by Keselman and colleagues (1998), full two-thirds of the studies "involved non randomization of the experimental units" (p. 373). As suggest by Henson (1998), Perhaps the lingering use of ANCOVA is due to the mystical promise that ANCOVA is a statistical correction of all pre-treatment problems and that it will provide increased power against Type II error. Such an argument is particularly compelling to doctoral students who find themselves aggressively seeking and even praying for statistically significant results! Unfortunately, ANCOVA has multiple assumptions that must be met before it can be accurately utilized. (p. 4) Similarly, part and partial correlations should be used only when it is meaningful to do so. While these correlational procedures may not have the same assumption as ANCOVA, there are important ramifications of their misuse. The purpose of the present paper is to (a) review these covariance correction procedures, (b) explain the relevant assumptions and potential pitfalls related to the analyses, and (c) discuss research situations when their use is both warranted and ill-advised. To make the discussion concrete, heuristic data sets are presented and analyzed to illustrate appropriate and inappropriate use of covariance corrections. Covariance Corrections 5 A Review of Procedures ANCOVA is a form of statistical control used to determine whether some variable, other than the intervention, is responsible for the difference on the dependent variable between groups (Borg & Gall, 1989; Shavelson, 1996). To do so, a continuously scaled variable that is highly correlated with the outcome measure is identified that contains information on individual subjects and is collected before the intervention begins. This variable is called the covariate. Covariates may take many forms and are best determined by the theoretical model of the researcher. Examples of covariates in the social and behavioral sciences may include a pretest, aptitude, age, experience, previous training, motivation, and grade point average (Ruiz-Primo, Mitchell, & Shavelson, 1996). Importantly, ANCOVA is designed to be used when individuals have been randomly assigned to control and experimental groups. That is, the ANCOVA statistical analysis is appropriate when the study has an experimental design. It cannot make a study's design experimental when, in fact, it is not. As Henson (1998) noted, In a researcher's zealous attempts to have an experimental or quasi-experimental design, the use of ANCOVA comes to be seen as a way of making such a design happen. In fact, a design is either experimental or it is not. Statistical analyses employed on the data obtained from the design do not magically transform a study into a true experiment . . . This thinking (or lack thereof) reflects confusion regarding the cooperative but separate roles of methodological design and statistical analysis. (p. 6-7) In an ANCOVA, covariate regression first removes the covariate variance from the dependent variable ignoring treatment group membership, thus reducing the error variance from within groups or the error caused by systematic differences among subjects. The idea is that this gives a more accurate estimate of error when asking if two or more groups differ on a dependent Covariance Corrections 6 variable. If the covariate and the dependent variable are linearly related, the use of the covariate in the analysis will serve to reduce the unexplained variation between groups. The greater the magnitude of the correlation, the greater the reduction in dependent variable variance due to within groups variation. Next, the error (or residualized) scores that remain from this regression of the covariate and dependent variable become the new dependent variable, and an analysis of variance (ANOVA) is performed using this residualized dependent variable (Henson, 1998). Since removal of the covariate ideally reduces the variance of the error scores in this ANOVA analysis, the size of the ANOVA F statistic purportedly increases, as does the likelihood of obtaining statistical significance. If error variance is reduced in this manner (which assumes that certain assumptions have been met), the ANCOVA will be more powerful than the ANOVA (Shavelson, 1996). In sum, ANCOVA partitions the total variability among the scores on the dependent variable into three sources of variability (Ruiz-Primo et al., 1996) but does so in a two step sequence. First, variance attributable to the covariate is removed from the dependent variable. This is done for all subjects, regardless of treatment group membership. Second, a classical ANOVA partitions the residualized dependent variable into variance attributable to the treatment group (or explained) variance and error. As previously stated, one reason ANCOVA is used concerns its "promise" of power against Type II error. However, there are several conditions that must be met for this to occur. Statistically, ANCOVA requires one degree of freedom be lost from the error term for each covariate used (Shavelson, 1996). This actually results in a larger critical value for the F statistic and makes it more difficult to find a significant F statistic. In order for power to increase, the variance attributable to the covariate must be sufficient enough to compensate for the loss of Covariance Corrections 7 power due to degree of freedom manipulations. This compensation occurs by lowering the sum of squares error (because some of the variance was due to the covariate) and thereby increasing power in the ANOVA. Thus, power is fostered only when the covariate has a sufficiently large correlation with the dependent variable. If the covariate is not highly correlated with the dependent variable, the use of ANCOVA may actually lower statistical power. A similar procedure of covariance correction is employed in part and partial correlations, which are used to identify unique relationships between two variables. Part correlation removes the influence of a third variable from one of two variables being considered (Borg & Gall, 1989). Here, a regression is performed between the third variable and one of the two variables being considered. The remaining error scores (residuals) are then correlated with the second variable in consideration, yielding the part correlation. Theoretically, the analysis is intended to explain the unique relationship between the two variables of interest after removing the spurious influence of some identified third variable. Partial correlation involves a similar dynamic but removes the influence of a third variable from both of the variables being considered. Here, two regression analyses are conducted between a third variable and both variables in consideration. Then, the remaining error scores from both of these analyses are correlated, yielding the partial correlation. Theoretically, a partial correlation explains the relationship between the variables after removing the spurious effect of the third variable from both variables of interest. For example, in considering factors that influence end-of-the-year test scores, one would have to consider what the student knew coming into the class (beginning-of-the-year test scores) as well as the teaching method that was used. Naturally, the beginning-of-the-year test scores would influence the teacher's decision of which method of teaching to use. Teachers with high achievers will probably instruct them differently than teachers of low achievers. By using partial Covariance Corrections 8 correlation analysis, the influence of the beginning-of-the-year achievement level can be removed from the end-of-the-year scores and from teacher instructional factors. This same scenario could employ part correlation analysis to remove the beginning-of-the-year achievement from just one variable, such as the end-of-the-year scores, to purportedly account for differences between students prior to instruction. In this case, if a large correlation were then observed between teaching method and the residualized end-of-the-year scores, the researchers may be more inclined to believe that the instructional method had some real effect on student achievement. (Note that this assertion could only ultimately be supported through an experimental study.) However, in such pre/posttest cases, there are serious conceptual problems with using part and partial correlations that will be discussed later. Relevant Assumptions and Potential Pitfalls If ANCOVA is to be used appropriately, then multiple conditions must be satisfied. Misinterpretation and misapplication of ANCOVA data may result from ignoring required assumptions. Researchers, both novice and experienced, should heed warnings against using this analysis as a "correction" of the dependent variable scores to adjust for data set problems (Buser, 1995), particularly when treatment groups were not randomly assigned. Loftin and Madison (1991) listed several assumptions underlying the successful use of ANCOVA. They are as follows: The covariate (or covariates) should be an independent variable highly correlated 1. with the dependent variable. 2. The covariate should be uncorrelated with the independent variable or variables. With respect to the dependent variable, (a) the residualized dependent variable (Y*) is 3. assumed to be normally distributed for each level of the independent variable, and (b) Covariance Corrections 9 the variances of the residualized dependent variable (Y*) for each level of the independent variable are assumed to be equal. 4. The covariate and the dependent variable must have a linear relationship, at least in conventional ANCOVA analyses. 5. The regression slopes between the covariate and the dependent variable must be parallel for each independent variable group. (p. 134) Each assumption has important ramifications. For example, Assumption 1 states the need for high correlation between the dependent variable (Y) and the covariate (X). The covariate will not do much to reduce the error sum of squares if the correlation is not high, and power may actually be reduced due to the loss of error degree(s) of freedom as the price to use the covariate(s). To demonstrate this point, Table 1 represents three sets of summary tables illustrating a regular ANOVA without a covariate, an ANCOVA in which the covariate (X) has a small correlation with the dependent variable (Y), and an ANCOVA involving a covariate with a large correlation with the dependent variable. The covariate is perfectly uncorrelated with the treatment conditions. INSERT TABLE 1 ABOUT HERE. Notice that when the covariate has a large correlation with the dependent variable, the statistical adjustments include the loss of one degree of freedom (df) for each covariate, a larger mean square (MS) error, and larger calculated F values. While the lowering of degrees of freedom error actually increases the critical F, the decrease in error sum of squares counterbalances the loss of degrees of freedom in error due to the high correlation between the 10