The Effects of Test Accommodation on Test Performance: A Review of the Literature1 Stephen G. Sireci, Shuhong Li, and Stanley Scarpati University of Massachusetts Amherst 1 Center for Educational Assessment Research Report no. 485. Amherst, MA: School of Education, University of Massachusetts Amherst. This paper was commissioned by the Board on Testing and Assessment of the National Research Council of the National Academy of Sciences. Effects of Test Accommodations 2 Abstract Over 150 studies pertaining to test accommodations were identified in the literature and 40 studies that empirically studied the effects of test accommodations on the performance of students with disabilities or English language learners were reviewed. The results of these studies are discussed as are the internal and external validity of the authors’ conclusions. All studies were critiqued with respect to the interaction hypothesis that test accommodations should improve the test scores for targeted groups, but should not improve the scores of examinees for whom the accommodations are not intended. Data are provided regarding the types of accommodations studied. In general, consistent conclusions were not found across studies due to the wide variety of accommodations, the ways in which they were implemented, and the heterogeneity of students to whom they were given. However, a fairly consistent finding was that the accommodation of extended time improved the performance of students with disabilities more than it improved the performance of students without disabilities. In light of this finding and similar results in some studies focusing on other accommodations, a revision of the interaction hypothesis is proposed. Directions for future research and for improved test development and administration practices are also proposed. Effects of Test Accommodations 3 The Effects of Test Accommodation on Test Performance: A Review of the Literature Standardized tests are a common part of educational systems throughout the United States. However, some aspects of standardized testing make the administration of these tests infeasible or unfair to certain students, particularly students with disabilities or students who are not native speakers of English. To address this problem, many tests are altered, or the test administration conditions are adjusted, to “accommodate” the special needs of these students. This practice is designed to level the playing field so that the format of the test or the test administration conditions do not unduly prevent such students from demonstrating their “true” knowledge, skills, and abilities. The practice of accommodating standardized tests for certain groups of students is often heralded as promoting equity in assessment. However, the resulting oxymoron—an accommodated standardized test—is not without controversy. At least two questions fuel the debate on the value of test accommodations. One question is “Do the test scores that come from nonstandard test administrations have the same meaning as test scores resulting from standard administrations?” A related question is “Do current test accommodations lead to more valid test score interpretations for certain groups of students?” These questions, and many related ones, have presented significant challenges for psychometricians, educational researchers, and educational policy makers for decades. The professional literature contains numerous published and unpublished empirical and non-empirical studies in the area of test accommodations. This literature is vast and passionate. In many cases, researchers argue against test accommodations in the name of fairness to the majority of examinees who must take the tests under perceivably stricter, standardized Effects of Test Accommodations 4 conditions. In many other cases, researchers argue that test accommodations are the only way to validly measure the knowledge, skills, and abilities of significant numbers of students. In this paper, we critically evaluate the literature on test accommodations focusing on those studies that evaluated the effects of test accommodations on students’ test performance. Several fundamental questions guided our review including (a) Do test accommodations affect the test performance of students with disabilities? (b) Do test accommodations affect the test performance of students who are non-native speakers of English? (c) What specific types of test accommodations best facilitate valid score interpretations for specific types of students? Many other important questions are addressed in this review; however, our major purpose is to critically evaluate the empirical research in this area to inform the educational measurement and policy communities about the pros and cons of test accommodations. Willingham et al. (1988) classified students with disabilities (SWD) into one of four categories: visually impaired, hearing impaired, physically disabled, or learning disabled. Currently, test accommodations are used for students classified into one or more of these categories as well as for students who are non-native speakers of English (hereafter referred to as English language learners or ELL). Our review looks at the literature associated with the first four categories (i.e., SWD) as well as with ELL. Understanding the “Interaction Hypothesis” A fundamental tenet underlying the provision of test accommodations to certain examinees is that some examinees need them, while most do not. Put in the perspective of test validity theory, some features of a standardized test administration introduce construct-irrelevant Effects of Test Accommodations 5 variance for some students. For example, a student who learned Spanish as her first language may do worse on a math test administered in English than on a parallel math test administered in Spanish. In this case, English proficiency may be considered extraneous to the math construct targeted by the test, but would certainly affect her test performance on the English language version of the test. Similarly, the ability to maneuver test materials may introduce construct- irrelevant variance for examinees with motor disabilities and the ability to see would obviously present construct-irrelevant difficulties for a blind student taking a standard math exam. From these examples it is clear that test accommodations are designed to promote fairness in testing and to lead to more accurate interpretations of students’ test scores. However, if the accommodation leads to an unfair advantage for the students who get them, for example, if everyone would benefit from the accommodation, then the scores from accommodated exams may be invalidly inflated, which would be unfair to students who do not receive accommodations. For this reason, an interaction hypothesis has been proposed to justify the use of test accommodations. This hypothesis states that test accommodations will lead to improved test scores for students who need the accommodation, but not for students who do not need the accommodation (Malouf, 2001, cited in Koenig, 2002; Shepard, Taylor, & Betebenner, 1998; Zuriff, 2000). That is, it is hypothesized that there is an interaction between accommodation condition (accommodated versus standard test administration) and type of student (e.g., students with disabilities versus students without disabilities) with respect to test performance. This hypothesis has also been called the maximum potential thesis (Zuriff, 2000). In our review, we look for evidence in support of this hypothesis. Effects of Test Accommodations 6 Psychometric Issues in Test Accommodations Before discussing our review, a brief discussion of the psychometric issues associated with test accommodations is necessary. Many of these issues have been clearly elucidated in the literature (e.g., Geisinger, 1994; Green & Sireci, 2000; Koretz & Hamilton, 2000; Phillips, 1994; Pitoniak & Royer, 2001; Scarpati, 1991, 2003; Sireci & Geisinger, 1998; Willingham et al., 1988), including guidelines presented in the Standards for Educational and Psychological Testing (American Educational Research Association (AERA), American Psychological Association, & National Council on Measurement in Education, 1999). Psychometric issues in test accommodations stress the need to remove construct- irrelevant barriers to test performance while maintaining integrity to the construct being measured by the test. In situations where individuals who take accommodated versions of tests may be compared to those who take the standard version, an additional validity issue is the comparability of scores across the different test formats. This issue was addressed in some depth in the Standards, for example Score equivalence is easiest to establish when different forms are constructed following identical procedures and then equated statistically. When that is not possible…additional evidence may be required to establish the requisite degree of score equivalence for the intended context and purpose…Some testing accommodations may only affect the dependence of test scores on capabilities irrelevant to the construct the test is intended to measure. Use of a large-print edition, for example, assures that performance does not depend on the ability to perceive standard-size print. In such cases, relatively modest studies of professional judgment may be sufficient to support claims of score equivalence. (AERA, et. al., p. 57) Effects of Test Accommodations 7 Although we do not address studies of score equivalence or comparability in this review, it is important to bear in mind that test accommodations are made in the pursuit of more valid test score interpretations. Review Process Several sources were used to locate research on the effects of test accommodations on test performance. First, we received a set of papers from the Board on Testing and Assessment of the National Research Council, who commissioned this review. Second we searched two electronic databases: ERIC and PsychInfo. We also contacted several researchers whose work was frequently cited or appeared numerous times in our electronic searches. We sent these researchers a list of the citations of their work that we had and we asked them to send us anything else they may have done in this area. Finally, we searched the web sites of the Center for Research on Evaluation, Standards, and Student Testing (CRESST) and the National Center on Educational Outcomes. These activities yielded over 150 papers, many of which were unpublished. This extensive list was closely reviewed to identify papers that explicitly addressed the effects of test accommodations on test performance. We initially identified 94 documents that fit this description, but there were a small number of papers that we were unable to locate. After reviewing the papers we did locate, we concluded that only 46 studies actually focused on the effects of test accommodations, and only 38 involved empirical analysis. Appendix A is an annotated bibliography of many of the studies we initially found in our scan of the literature. This appendix provides annotations for all of the studies included in our review, as well as for those studies that were relevant, but deemed less important due to methodological concerns or extreme lack of generalizability. A listing of all studies located in our initial scan of the literature, including those that were not annotated, is presented in Appendix B. Effects of Test Accommodations 8 Organization of the Review Our review is structured using three primary criteria: group intended to be helped by the accommodation (SWD or ELL), type of accommodation, and research design. However, this organization is not perfect, since many studies involved multiple accommodations and some looked at both SWD and ELL. Moreover, the term “students with disabilities” includes students with learning disabilities, motor disabilities, and sensory disabilities. The term “English language learners” (ELL) includes examinees who come from a variety of linguistic backgrounds and are in various stages of becoming proficient in English. Although we use these labels to describe general classifications of students for the purpose of accommodation, we do not consider individuals within these groups to be homogeneous, nor do we consider these general labels appropriate for all individuals with a group. As for type of accommodation, a wide variety of assessment accommodations exist. In their review of the literature, Thompson, Blount, and Thurlow (2002) categorized eleven types of accommodation into one of four general categories: presentation, response, setting, or timing/scheduling. Borrowing this categorization scheme, we defined test presentation accommodations to include oral administration (i.e., “read aloud” protocols where the test directions and/or items are read to the test taker), changes in test content (e.g., simplified language), and changes in test format (e.g., Braille, large print). Timing/scheduling accommodations included allowing extended or unlimited time and breaking up test administration into separate sessions (in a single day or over multiple days). Response accommodations included allowing students to write directly into test booklet or dictate their answers. Setting accommodations typically involved administering the tests individually or in a separate room. Effects of Test Accommodations 9 Our discussion of the literature first focuses on accommodations for SWD. Subsequently, we focus on accommodations designed for ELL. Within each of these student groups, we distinguish between literature reviews, experimental studies, and non-experimental studies, stratified by type of accommodation, where possible. Descriptive Characteristics of Studies Due to the fact that the literature on the effects of test accommodations is vast, some general descriptions of the number and types of studies that we found are helpful for understanding this body of research. The number of studies that are relevant to specific types of accommodation are presented in Table 1, stratified by student group (SWD or ELL). As illustrated in Table 1, the majority of research in this area has focused on SWD (75%) and the most common accommodations studied were oral administration (31%) and the provision of extra time (20%). These findings are similar to a recent review of the literature conducted by Thompson, Blount, and Thurlow (2002) who found that studies investigating oral administration were the most common, followed closely by studies investigating extended time. In another recent review of the literature, Chiu and Pearson (1999) found that extended time was the most frequently investigated accommodation and setting and response format were least frequently investigated. It should be noted that oral presentation is often given with extended time and so separation of the effects of these two variables is not always possible. Another point to bear in mind about Table 1 is that many studies, such as those that looked at scores from large-scale test administrations, analyzed scores from many different types of accommodations, while other studies focused on just a single accommodation. Extended time and oral presentation were the most common accommodations investigated in those studies that focused on a single accommodation. Effects of Test Accommodations 10 Table 1 General Description of Studies Included in the Review Study Focused On Type(s) of Accommodation SWD ELL Total Presentation: Oral* 22 1 23 Dictionary/Glossary -- 9 9 Linguistic modification of test -- 3 3 Dual-language -- 2 2 Paraphrase 2 -- 2 Technological 2 -- 2 Braille/Large Print 1 1 Sign Language 1 -- 1 Encouragement 1 -- 1 Cueing 1 -- 1 Spelling assistances 1 -- 1 Manipulatives 1 -- 1 Timing: Extended time 12 3 15 Multi day/sessions 1 -- 1 Separate sessions 1 -- 1 Response: Scribes 2 -- 2 In booklet vs. answer sheet 1 -- 1 Mark task book to maintain place 1 -- 1 Transcription 1 -- 1 Setting: Separate room 1 -- 1 No specifics listed*** 4 1 5 Total 56 19 75 Notes: *Includes read aloud, audiotape, or videotape, and screen-reading software. Literature reviews and issues papers are not included.