NAEP Math Performance and Test Accommodations: Interactions With Student Language Background CSE Technical Report 536 Jamal Abedi, Carolyn Hofstetter, and Eva Baker CRESST/University of California, Los Angeles Carol Lord California State University/Long Beach February 2001 Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing Graduate School of Education & Information Studies University of California, Los Angeles Los Angeles, CA 90095-1522 (310) 206-1532 National Assessment of Educational Progress, Technical Review Panel for Assessing the Validity of the National Assessment of Educational Progress TRP Task 3D: Language Background Study—Phase 2. Jamal Abedi, Project Director, CRESST/University of California, Los Angeles Copyright © 2001 The Regents of the University of California The work reported herein was supported in part under National Center for Education Statistics Contract No. RS90159001 as administered by the U.S. Department of Education. The findings and opinions expressed in this report do not reflect the position or policies of the National Center for Education Statistics or the U.S. Department of Education. CONTENTS LIST OF TABLES ....................................................................................................................................iv ACKNOWLEDGMENTS.......................................................................................................................vi EXECUTIVE SUMMARY.....................................................................................................................vii INTRODUCTION.....................................................................................................................................1 LITERATURE REVIEW...........................................................................................................................3 Math Performance Among Language Minority Students.......................................................4 Impact of Background Factors.....................................................................................................5 Linguistic Variables Affecting Math Performance...................................................................7 Effect of Accommodations..........................................................................................................11 PURPOSE.................................................................................................................................................14 RESEARCH HYPOTHESES..................................................................................................................14 METHOD..................................................................................................................................................16 Participants....................................................................................................................................16 Design.............................................................................................................................................16 Instruments....................................................................................................................................17 Procedure.......................................................................................................................................20 Linguistic Modification of Math Items.....................................................................................20 Development of Glossary Accommodation.............................................................................22 Categorization of LEP and Non-LEP students........................................................................22 FINDINGS................................................................................................................................................24 Sample Descriptives.....................................................................................................................24 Math Performance by Accommodation...................................................................................26 Reading Performance by Accommodation..............................................................................29 Impact of Reading Proficiency on Math Performance...........................................................30 Teacher and School Effects..........................................................................................................32 Analyses of the Background Questionnaire............................................................................34 Designation of LEP Status...........................................................................................................46 Predictors of Math and Reading Performance........................................................................46 Differential Impact of Accommodation Strategies on LEP Students..................................49 SUMMARY..............................................................................................................................................56 REFERENCES..........................................................................................................................................58 APPENDIX: Student Background Questionnaire/Teacher Classroom Questionnaire.............65 iii LIST OF TABLES 1. Two Categories of Accommodations for English Language Learners...............................12 2. Test Booklets Administered in Study.......................................................................................17 3. Results of Interrater Reliability Studies for Open-Ended Math and ..................................19 Reading Test Items 4. Mean NAEP Math Achievement Scores for 8th-Grade Students........................................27 5. ANOVA Results for Math Scores by Accommodation and LEP Status ............................28 6. Mean NAEP Reading Achievement Scores for 8th-Grade Students...................................29 7. ANOVA Results for Reading Scores by Accommodation and LEP Status ......................30 8. ANCOVA Results for Math Scores by Accommodation and LEP Status, ........................31 Using Reading Score as a Covariate 9. ANOVA Results for Math Scores by School............................................................................33 10. ANOVA Results for Reading Scores by School......................................................................33 11. ANOVA Results for Math Scores by Teacher.........................................................................33 12. ANOVA Results for Reading Scores by Teacher....................................................................33 13. Selected Background Variables by Question Number..........................................................35 14. Correlation Among Background (Composite) Variables......................................................36 15. Internal Consistency Coefficients of Background (Composite) Variables.........................37 16. Correlation Among Composite Variables for LEP Students................................................38 17. Correlation Among Composite Variables for Non-LEP Students.......................................39 18. Internal Consistency Coefficients of Composite Variables for ...........................................40 LEP Students 19. Internal Consistency Coefficients of Composite Variables for Non-LEP...........................41 Students 20. Correlation Coefficients Between Composite Variables and Math and ............................42 Reading Scores 21. Correlation Coefficients Between Composite Variables and Math and ............................43 Reading Scores for LEP Students 22. Correlation Coefficients Between Composite Variables and Math and ............................44 Reading Scores for Non-LEP Students 23. Correlation Coefficients Between Individual Variables and Math and ............................45 Reading Scores for All Students 24. Comparison of LEP Status..........................................................................................................46 25. Results of Multiple Regression Analysis Predicting Math Scores From............................47 Students’ Background Information (All Students) 26. Results of Multiple Regression Analysis Predicting Math Scores From ...........................48 Students’ Background Information (LEP Students) 27. Results of Multiple Regression Analysis Predicting Reading Scores From .....................49 Students’ Background Information (All Students) iv 28. Results of Multiple Regression Analysis Predicting Reading Scores From ......................49 Students’ Background Information (LEP Students) 29. Background Variables for Grouping Students........................................................................50 30. Full Model......................................................................................................................................52 31. Restricted Model...........................................................................................................................53 32. Impact of Accommodations on Average Math Performance, by Math Class...................54 33. Impact of Accommodations on the Average Math Performance, by .................................55 Language of Instruction v ACKNOWLEDGMENTS The generous efforts of several people informed the development of this study. We would like to thank all the researchers—faculty, staff, and students—who participated in and informed the study. We are especially indebted to Christy Kim, Joan Herman, Frances Butler, and Robin Stevens for their support and insightful comments on an earlier draft of this report. We appreciate the assistance of: Cici Bianchi Debra LeRoux Gina Cogswell Lynn Winters Kris Waltman Alfredo Artiles Barry Gribbons Zenaida Aguirre-Muñoz Rory Constancio Katharine Fry We greatly appreciate the cooperation and assistance of the Los Angeles Unified School District, the Long Beach Unified School District, and all the test administrators, teachers, school staff, and students who helped us. vi NAEP MATH PERFORMANCE AND TEST ACCOMMODATIONS: INTERACTIONS WITH STUDENT LANGUAGE BACKGROUND Jamal Abedi, Carolyn Hofstetter, and Eva Baker National Center for Research on Evaluation, Standards, and Student Testing/UCLA Carol Lord California State University, Long Beach EXECUTIVE SUMMARY Legislation has mandated the inclusion of students with limited English proficiency in large-scale academic assessments administered in English. Many states permit accommodations in the testing of limited English proficient (LEP) students, and various approaches to accommodation were tried in the 1996 NAEP administration. At present, research on the effect of accommodations is limited, yet policymakers and educators must make decisions about whether to use accommodations, which types of accommodation to use, and which students should receive testing accommodations. We report here on a study that addresses the following questions: • What student background factors affect math performance? • What accommodation strategies have the greatest impact on student performance? • What effect do testing accommodations have for students with limited English proficiency? • Does the impact of accommodations vary with student background factors? During the spring of 1997, 946 students in 8th-grade math classes in urban schools in southern California were given tests including 35 items from the 1996 NAEP Grade 8 Bilingual Mathematics booklet. Five different forms of the test booklet were randomly distributed to the students. One booklet contained the math test items in their original English form. Each of the other four booklets incorporated accommodations in test form or in testing procedure, specifically: • the linguistic structures in the items were modified; mathematical terms were retained, but non-math vocabulary was simplified, and complex syntactic structures were reduced; or vii • the original wording was retained, but a glossary was provided; the margins of the test booklet pages included definitions for non-math vocabulary items that might be difficult or unfamiliar; or • extra time was given for the test; or • both a glossary and extra time were provided. In addition to a math test, each student completed a reading test and a language background questionnaire. The reading test was a two-page story with 11 questions from the NAEP 1992 Grade 8 Reading assessment. The language background questionnaire consisted of 45 items, primarily from the 1996 NAEP Grade 8 Bilingual Mathematics booklet and an earlier CRESST study (Abedi, Lord, & Plummer, 1995). Over half of the students in the study were designated limited English proficient (LEP). Only about 17% were initially fluent in English (IFE); the remainder, about 30%, had transitioned from LEP programs and were designated fluent English proficient (FEP). Most (85%) spoke another language besides English, and for most of those, the other language was Spanish (82%). Initial analyses suggest that test accommodations affected student math scores. For the entire sample, providing extra time for the math test resulted in a 1-point increase in student scores (mean scores of 14.68 on the original items, and 15.64 on the original items with extra time allowed, out of a total of 35 items). When a glossary and extra time were provided, the mean scores were more than 2 points higher (mean 17.08). For the entire sample, no significant difference was found when items were linguistically modified (mean 14.23) or a glossary was provided without extra time (mean 14.53). In fact, non-LEP students actually scored slightly lower on the modified English version than they did on the original version. When we compare the scores of LEP and non-LEP students, we find differences: on average, non-LEP students scored more than 5 points higher overall. The greatest difference between LEP and non-LEP scores was found on the glossary- plus-extra-time accommodation (6.38 points difference); the least difference between LEP and non-LEP scores was found on the linguistically modified version (3.31 points difference). In other words, the modified English accommodation enabled the LEP students to achieve scores most comparable to those of non-LEP students. If we look at the performance of the 473 LEP students in the sample, we find that they benefited from three of the accommodations—Modified English, Extra Time, and Glossary plus Extra Time—with the latter showing the greatest benefit. (Mean scores: original items 12.07, Modified English 12.63, Extra Time 12.93, and viii Glossary plus Extra Time 13.69.) The LEP students did not benefit from the glossary accommodation without extra time; a possible explanation for this is that it took extra minutes to consult the glossary, and therefore the glossary did not help to increase scores unless extra time was provided for it. Student scores on the reading test correlated, in general, with scores on the math tests. This is consistent with earlier research on student performance in math and reading. LEP students scored lower overall; the LEP mean was 3.92 out of 11, and the non-LEP mean was 6.35. All students took the same reading test. However, there were small differences in mean reading test scores for different accommodation groups; this was not an expected result, since the booklets were distributed randomly within classes. For the LEP student group, math scores on original, Modified English, Extra Time, and Glossary plus Extra Time booklets were 12.07, 12.63, 12.93, and 13.69, respectively; these same groups showed reading scores of 3.78, 3.84, 3.93, and 4.48, respectively. If the reading scores represent real differences between groups, these trends may imply that reading skills and math skills tend to go together, or that the poorer readers got low math scores because they did not understand the English language of the items. We are investigating these possibilities. After controlling for students’ reading scores, there were still significant differences in students’ math test scores, by type of accommodation. When LEP and non-LEP groups were compared on their math performance without controlling for reading proficiency, a coefficient of determination of 0.15 was obtained. When the reading score was entered as a covariate, however, this coefficient was reduced to 0.05. That is, two thirds of the variance in math scores between LEP and non-LEP students was explained by differences in level of reading proficiency in English. Analyses of students’ responses on the language background questionnaire showed that the best predictor of math scores was the length of time the student has lived in the United States. Other predictors were questions about how far the student expects to go in school, how good at math the student is, and how many times the student changed schools. Some major findings of this study include the following: • Students designated LEP by their schools scored, on average, more than 5 points lower than non-LEP students on a 35-item math test. • In comparison with scores on the original NAEP items, the greatest score improvements, by both LEP and non-LEP students, were on the accommodation version that included a glossary explaining potentially unfamiliar or difficult words plus extra time. ix • LEP students’ scores were higher on all types of accommodation except Glossary only; LEP students were helped by Modified English, Extra Time, and Glossary plus Extra Time. • Most accommodations helped both LEP and non-LEP students; the only type of accommodation that narrowed the score difference between LEP and non-LEP students was Modified English. • Students who were better readers, as measured by reading test scores, achieved higher math scores. The results of this study indicate that there are relationships among student background variables and test performance under different types of accommodation. We are currently conducting further analyses to clarify these relationships. Among the specific variables we are investigating are: student English proficiency level; math proficiency level; reading skill level; first language; recency of arrival in the United States; self-reported data including attitudes, English proficiency, and first-language proficiency; the consistency and reliability of self- reported data and school-reported data as sources of information on language proficiency; and appropriateness of different types of accommodation with different subgroups of students. Test accommodations can result in higher math scores for both LEP and non- LEP students, and some types of accommodation have greater impact than others. Furthermore, certain accommodations may help LEP students more than non-LEP students. These differences and relative impacts need to be considered and investigated further before accommodation strategies are adopted for large-scale assessments. x