DOCUMENT RESUME ED 462 419 TM 033 683 Li, Jun Corser; Woodruff, David J. AUTHOR Bayesian Statistical Inference for Coefficient Alpha. ACT TITLE Research Report Series. INSTITUTION American Coll. Testing Program, Iowa City, IA. ACT-RR-2002-2 REPORT NO 2002-01-00 PUB DATE NOTE 27p. AVAILABLE FROM ACT Research Report Series, P.O. Box 168, Iowa City, IA 52243-0168. Tel: 319-337-1028; Web site: http://www.act.org. PUB TYPE Reports Evaluative (142) EDRS PRICE MF01/PCO2 Plus Postage. *Bayesian Statistics; Markov Processes; Monte Carlo Methods; DESCRIPTORS *Reliability; *Statistical Inference; Statistics *Alpha Coefficient IDENTIFIERS ABSTRACT Coefficient alpha is a simple and very useful index of test reliability that is widely used in educational and psychological measurement. Classical statistical inference for coefficient alpha is well developed. This paper presents two methods for Bayesian statistical inference for a single sample alpha coefficient. An approximate analytic method based on conjugate distributions is derived. This method is easy to compute. A second method uses Markov Chain Monte Carlo (MCMC) methodology as implemented by the computer program WinBUGS. WinBUGS may be downloaded for free from the Internet, and this paper includes WinBUGS code for making Bayesian inferences about coefficient alpha. Pseudo-randomly generated data are used to compare the two Bayesian methods to each other and both of these methods to the classical method. The results indicate that the two Bayesian methods work well as long as the number of items and examinees are not too small. Appendixes discuss the derivation of the marginal likelihood, contain WinBUGS code, and present the figures. (Contains 1 table, 9 figures, and 11 references.) (Author/SLD) Reproductions supplied by EDRS are the best that can be made from the original document. 2002.,2 s Bayesian Statistical Inference for c, Coefficient Alpha jun Corser Li David J. Woodruff PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY fax- reAJ\A- TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) 1 U.S. DEPARTMENT OF EDUCATION office of Educational Research and Improvement EDUçrIONAL RESOURCES INFORMATION CENTER (ERIC) This document has been reproduced as received from the person or organization originating it. 0 Minor changes have been made to improve reproduction quality. 00 Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. 2 BEST COPY AVAILABLE ACT 2002 anuary For additional copies write: ACT Research Report Series P.O. Box 168 Iowa City, Iowa 52243-0168 ©2002 by ACT, Inc. All rights reserved. Bayesian Statistical Inference for Coefficient Alpha Jun Corser Li David J. Woodruff 4 Abstract Coefficient alpha is a simple and very useful index of test reliability that is widely used in educational and psychological measurement. Classical statistical inference for coefficient alpha is well developed. This paper presents two methods for Bayesian statistical inference for a single sample alpha coefficient. An approximate analytic method based on conjugate distributions is derived. This method is easy to compute. A second method uses MCMC methodology as implemented by the computer program WinBUGS. WinBUGS may be downloaded for free from the Internet, and this paper includes WinBUGS code for making Bayesian inferences about coefficient alpha. Psuedo-randomly generated data are use to compare the two Bayesian methods to each other and both of those methods to the classical method. The results indicate that the two Bayesian methods work well so long as the number of items and examinees are not too small. Acknowledgements The authors thank the following persons for their comments, suggestions, Dr. Jim and support: Dr. Hiroshi Watanabe, Dr. Shin-ichi Mayekawa, and Sconing. 6 v Bayesian Statistical Inference for Coefficient Alpha Frequentist inferential procedures for coefficient alpha are well developed. The paper by Fe ldt, Woodruff, and Salih (1987) presents a summary of the different methods, gives complete references to the area, and also discusses the robustness of the procedures to violations of their assumptions. A more recent paper by Hakstian and Barchard (2000) also evaluates the robustness of the studies on the robustness of the other recent procedures and references procedures. The methods depend on normal distribution theory for random and Even though the methods are often employed using mixed effects ANOVA. dichotomous item response data, the procedures generally perform well when there are reasonable numbers of items and examiness. Two papers by Fe ldt and Ankenmann (1998, 1999) contain classical power curves and tables for testing the difference between two independent sample alpha coefficients or two dependent sample alpha coefficients. A more recent paper on the sampling distribution of coefficient alpha is VanZyl, Neudecker, and Nel (2000). Their paper takes a multivariate approach instead of the ANOVA approach used in the earlier papers. The present paper derives two Bayesian procedures for making inference also depend on normal about a single alpha coefficient. Our procedures distribution theory for random and mixed effects ANOVA. Our first procedure is an approximate analytic method based on conjugate distributions. This method is relatively simple and easy to compute. Our second method uses MCMC methodology as implemented by the WinBUGS computer program. Example WinBUGS code is included in Appendix B. We compare the two Bayesian procedures to each other and both to a frequentist method. One advantage gained from using Bayesian techniques is the ability to incorporate disparate but relevant information into the analysis by way of the prior distribution. Another advantage is the ability to combine data from different analyses by using the posterior distribution obtained from a previous analysis as the prior distribution for the next analysis. These properties can be useful to test developers creating new tests. If the new tests are related to 7 2 previous tests, then information about the reliability of the earlier tests can be used to develop an initial prior distribution for the new tests. In addition, test developers initially may have to use shorter versions of the tests and administer these versions at different times and to different small groups of students. In these situations inference can be updated by using the posterior distribution obtained from an earlier analysis as the prior distribution for the next analysis. Test users may want to calculate the reliability of a test in a specific group of examinees of special interest to them, and the test may be administered to only a few examinees at any one time, but at regular intervals over time. The Bayesian could prove process of updating the inference from prior to posterior to new prior convenient in such situations. Feldt's inferential procedures for alpha is based on the demonstration by Hoyt (1941) that the sample alpha coefficient can be computed from the observed student mean square and the observed interaction mean square of a two-way students by items ANOVA with one observation per cell. Hoyt's result is an is true for any two-way table of numbers with one It algebraic identity. observation per cell. Using Hoyt's 1941 result, Fe 1dt (1965) developed a frequentist procedure for theory. He a single alpha coefficient based on ANOVA normal distribution considered the two-way examinees by items random effects ANOVA model with mixed model. In one observation per cell, but his results also can be valid under a practice examinees often are randomly sampled from a large population of examinees and the same is sometimes true for items, though the sampling of items is not always strictly random. In these situations the random effects ANOVA model is most appropriate. In other situations only the items actually administered are of interest so items are treated as a fixed effect and the mixed ANOVA model should be used. VanZyl, Neudecker, and Nel (2000) derive the same result as Fe ldt, but their derivation is based on the multivariate form of the mixed ANOVA model that assumes a compound-symmetric covariance matrix. The derivation of our analytic Bayesian method is similar to the derivation used by VanZyl et. al. (2000), but we use the two-way random effects ANOVA model with WinBUGS. 3 Methodology In what follows, the true and sample values of coefficient alpha can take values between oo and one. Negative values for coefficient alpha can occur when the inter-item correlations are negative. See VanZyl, Neudecker, and Nel (2000) for additional discussion of this issue. We denote coefficient alpha by p and we take as prior distribution for p (13T a' oo < p <1. p(p) = (1) > 0, > 0, p)(o' (1 (1-P) 13' (a It is convenient This is a gamma type distribution and we denote it Gp (OL' . , to indicate our prior knowledge about p by specifying a likely value for p. We then can indicate our confidence in this value by specifying a hypothetical prior sample size upon which the value is based. We denote the prior mean of p as and we assign it our prior estimate of the value of p. We next define n' + 1 as the size of a hypothetical prior sample that indicates the strength of our prior n12 and O' Using E, M, and V to ni[2(1 r')] belief. Finally we take = = . find that denote mean, mode, and variance we (2) n' 2 21 r , and M (p) = (3) , ri 2 = V (p) (4) . Having specified values for n' and r' we then can compute from equations (3) and (4) the prior mode and prior variance for p. If n' is less than three the prior density will be J shaped and without a well-defined mode. Taking n' = 0 Note that coefficient yields an improper non-informative prior density for p . alpha cannot by computed from a sample of size one. We recommend specifying the prior mean of p and not the prior mode of p. When n' is small the prior mean and prior mode can have quite different values and then better results are obtained by specifying a value for the prior mean rather than the prior mode. 4 Before combining our prior with the likelihood, it is convenient to transform This yields as prior distribution for T p to T where T = 1/(1 p) . (7. )MC, +1, e-fl 1), T > 0. ( p(T) (5) r (oL )" This is an inverse gamma distribution and we denote it IG, 13' . , As previously mentioned Feldt (1965) and VanZyl, Neudecker, and Nel (2000) derived a frequentist distribution theory for Let alpha. coefficient r) where r denotes the sample alpha coefficient. For a sample that has t = 1/(1 n-4-1 examinees and m+1 items they showed that t p) (1 F r) "m T (1 From a Bayesian perspective this F distribution is a marginal or integrated likelihood (Bernardo & Smith, 1994), and we derive it under a Bayesian model in Appendix A. For nm large the distribution is well approximated by a xn2/n Fryzin this based and method distribution on our Bayesian inference of is approximation. Considered as a function of T we write the xn2 based likelihood as = k er (n,t)r-npe-nt/(2-r) (7) where k (n,t) is a function of n and t only and does not depend on T. Inspection of equation (7) shows that it is the kernel of an inverse gamma distribution for T. The inverse gamma distribution is closed under multiplication; consequently, because our prior distribution for T is also inverse gamma, it follows that application of Bayes theorem yields a posterior inverse gamma distribution for -r. We denote the posterior IGT (a." 03"). Multiplying (5) and (7) together as dictated by Bayes theorem gives the following values for cc" and On : n) (n" a = + and (8) 2 (9) 2 (1 r)1. 1 1 0