Descriptive Study of State Assessment Policies for Accommodating English Language Learners Lynn Shafer Willner, Charlene Rivera, and Barbara D. Acosta October 2008 GW-CEEE The George Washington University Center for Equity and Excellence in Education 1555 Wilson Boulevard, Suite 515 Arlington, VA 22209 Tel: 703.528.3588 Fax: 703.528.5973 http://ceee.gwu.edu ii The George Washington University Center for Equity and Excellence in Education The mission of The George Washington University Center for Equity and Excellence in Education (GW-CEEE) is to advance education reform so all students achieve to high standards. GW-CEEE conducts policy and applied research, designs and implements program evaluations, and provides professional development and technical assistance. The Center’s clients include state education agencies, school districts, schools, foundations and federal agencies. For over a decade, GW-CEEE has conducted research on the inclusion and accommodation of English language learners (ELLs) in high stakes testing, including periodic reviews of state assessment policies for ELLs. GW-CEEE is currently conducting an applied research project with support from the Bill & Melinda Gates Foundation to study the academic language demands of ELLs in middle and high schools. GW-CEEE also provides technical assistance and conducts professional development for clients in states, districts, and schools. With funding from the U.S. Department of Education, GW-CEEE operates the Mid-Atlantic Comprehensive Center (MACC) and the Mid-Atlantic Equity Assistance Center, both of which provide technical assistance to states in the Mid-Atlantic region. Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu iii Acknowledgements This publication was supported by the U.S. Department of Education’s (ED) LEP Partnership, a partnership of the states with the Deputy Secretary’s Office and the Office of English Language acquisition at ED, the National Clearinghouse for English Language Acquisition, the Council of Chief State School Officers, the National Council of La Raza, and the Mexican American Legal Defense Fund. As part of Partnership meetings, external assessment experts are made available to states to discuss the quality of the assessment process for English language learners (ELLs) and to examine how state assessments can be made more accessible to ELLs. This particular project grew out of discussions with state education agency staff including assessment, Title I, and III directors who had concerns about selecting and using accommodations for ELLs. Identifying accommodations for ELLs that meet requirements of both effectiveness and validity is a challenging task. In the absence of a strong research base, ED charged The George Washington University Center for Equity and Excellence in Education (GW-CEEE) to develop a Guide for Refining State Assessment Policies for Accommodating ELLs. To prepare the foundation for the Guide, GW-CEEE conducted two studies, a Descriptive study of state polices and a Delphi study of practice. We are grateful for the collaboration among ED offices that believed it was important to fund work in this area. In particular we would like to thank Ray Simon, Deputy Secretary, Kathryn Doherty, Special Assistant Office of the Deputy Secretary, Richard Smith, Acting Director for the Office of English Language Acquisition (OELA), Jenelle Leonard, Director, School Support and Technology Programs, and Fran Walter, Group Leader, Technical Assistance Programs. In particular, the GW-CEEE research team is appreciative to staff from the 51 SEAs who cooperated in providing state policies and verifying the accuracy of state policy data. Their feedback on drafts of the Descriptive Study was invaluable. We are indebted to the members of the expert panel. With their help, the GW-CEEE research team sorted and ranked ELL-responsive accommodations and mapped them to ELP levels. They provided valuable feedback on drafts of the two study reports. We would also like to recognize the contributions of Ellen Forte, President of edCounts and Charles Stansfield, President of Second Language Testing, Inc. Special recognition goes to GW-CEEE staff who provided important support throughout the project. Diane Staehr Fenner coordinated the development of an English language proficiency rubric for the Delphi Study, provided assistance during the data collection process, and gave feedback on the first draft of the Descriptive Study. Roshaun Tyson provided design and layout expertise. Kristina Anstrom, Abhee Brahmnalkar, Janet Brown, Marilyn Muirhead, James Ruff, and Tracy Runfola helped pilot the Delphi questionnaires and provided feedback on drafts of the two studies and the Guide; Jennifer Lynch and Gamaliel Gabriento provided administrative support; and Charlotte Blane provided expert editorial assistance. Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu iv Expert Panel Jamal Abedi, Ph.D. Lorena Llosa, Ph.D. Professor Assistant Professor School of Education New York University University of California, Davis Carlos Martinez, M.A. Linda Carstens, Ph.D. Assistant Secretary of Education Director New Mexico Public Stanford LEADS Network Education Department Stanford University Teddi Predaris, M.S., M.A. H. Gary Cook, Ph.D. Director Researcher Office of ESOL Services Wisconsin Center for Education Research Fairfax County Public Schools University of Wisconsin Maria Pennock-Roman, Ph.D. Richard Duran, Ph.D. Research Consultant Professor MPR Psychometric & Statistical Research University of California, Santa Barbara Consulting Margo Gottlieb, Ph.D. Raquel Sinai, M.Ed. Director of Assessment and Evaluation Coordinator Illinois Resource Center Bilingual/ESL Education New Jersey Department of Education Robin Lisboa, M.S. Administrator Division of English Language Learning Illinois State Board of Education Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu v Table of Contents Acknowledgments.......................................................................................................................... iii Expert Panel ................................................................................................................................... iv Table of Contents .............................................................................................................................v List of Tables ................................................................................................................................ vii List of Figures ............................................................................................................................... vii Executive Summary ..................................................................................................................... viii Introduction ......................................................................................................................................1 Review of literature..........................................................................................................................1 Inclusion of ELLs in state assessments ........................................................................................2 What is an accommodation? ........................................................................................................3 Reviews of state assessment policies ...........................................................................................3 Focus on inclusion ...................................................................................................................3 Focus on inclusion and accommodations ................................................................................4 Research on specific accommodations ........................................................................................6 Research on accommodations designated for specific content assessments ...............................7 Research on guidance for assigning accommodations ................................................................7 Research on monitoring the use of accommodations ..................................................................9 Method ...........................................................................................................................................10 Study limitations ........................................................................................................................12 Findings..........................................................................................................................................12 To what extent are state assessment policies responsive to ELLs’ unique linguistic needs? ....13 Overview of accommodations allowed ..................................................................................13 ELL-responsive accommodations with a research base, in state policies .............................15 Non-ELL-responsive accommodations in state policies ........................................................17 Accommodation approach used in state policies allowing only ELL-responsive accommodations ....................................................................................................................19 Number and type of accommodations allowed to ELLs per state policy ..............................20 Indicators of ELL-responsiveness within state policies .........................................................22 Indicators of use of a disabilities framework within state policies ........................................24 Relationship between the number of accommodations in state policy and ELL- responsiveness........................................................................................................................26 Summary of key findings to research question #1 .................................................................26 To what extent are state policies guiding decision making and monitoring practices? .............27 Guidance to decision makers for assigning accommodations ...............................................27 Models of practice for assigning accommodations ................................................................28 Policy requirements for use of accommodations prior to testing ..........................................29 Policy requirements for monitoring the use of accommodations ..........................................30 Summary of key findings for research question #2 ...............................................................32 What are the most frequently allowed ELL-responsive accommodations? ..............................32 Frequency with which content for accommodations is specified in state policy ...................32 Frequency of ELL-responsive accommodations allowed for mathematics assessments.......34 Frequency of ELL-responsive accommodations allowed for reading/language arts assessments ............................................................................................................................36 Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu vi Extent to which states allow accommodations with a research base .....................................37 Summary of key findings for research question #3 ...............................................................38 In what ways have state assessment policies for accommodating ELLs changed since the inception of NCLB? ...................................................................................................................39 Number of states with an assessment policy addressing ELLs ..............................................39 Number and percent of ELL-responsive accommodations allowed across states .................39 Number and proportion of total accommodations allowed by each state ..............................40 Comparative influence of a disabilities framework in state policies .....................................44 Comparison of most frequently-specified accommodations for mathematics assessments for ELLs between 2000-01 and 2006-07 ...............................................................................45 Summary of key findings for research question #4 ...............................................................46 Discussion ......................................................................................................................................46 Ways in which state assessment policies for the accommodation of ELLs have improved ......46 Ways in which state policies need improvement .......................................................................47 Policy design ..........................................................................................................................47 The systematic selection of ELL-responsive accommodations .............................................48 Procedures to improve the quality of accommodations .........................................................49 Future Research .............................................................................................................................49 Recommendations ..........................................................................................................................50 Improve state assessment policy design ....................................................................................50 Improve the selection of ELL-responsive accommodations ......................................................51 Improve the quality of accommodations ...................................................................................51 References ......................................................................................................................................52 Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu vii List of Tables Table 1. Year of publication for 2006-07 state assessment policies ............................................... 13 Table 2. Written and oral ELL-responsive accommodations in English and the native language…………………………….................................. ............................................................ 16 Table 3. Classification of non-ELL-responsive accommodations in state policies…….....… ...... 18 Table 4. Number and type of ELL and non-ELL responsive accommodations available to ELLs in each state policy ................................................................................................................ 21 Table 5. Indicators of ELL-responsiveness in each state policy ..................................................... 23 Table 6. Indicators of use of a disabilities framework in ELL state assessment policies ............... 25 Table 7. Making a decision to administer CSAP using oral translation and/or translated scribing ............................................................................................................................................ 29 Table 8. Monitoring requirements described in state assessment policies ..................................... 31 Table 9. Number of specified and unspecified ELL-responsive accommodations by content assessed ........................................................................................................................................... 33 Table 10. Frequency of ELL-responsive accommodations allowed for mathematics assessments in state policies ........................................................................................................... 34 Table 11. Frequency of ELL-responsive accommodations allowed for reading/language arts assessments in state policies ........................................................................................................... 36 Table 12. Frequency of ELL-responsive accommodations with a research base allowed in state policies .................................................................................................................................... 38 Table 13. Change in number and percent of ELL-responsive accommodations over five years ... 39 Table 14. Change in number of accommodations allowed for ELLs in state policies .................. 41 Table 15. Change in number and percent of ELL-responsive accommodations in state policies ............................................................................................................................................ 43 Table 16. Change in use of disabilities taxonomy in state policies to organize ELL accomodations, 2000-01 and 2006-07 ............................................................................................ 44 Table 17. Change in use of combined list of ELL and disabilities accommodations in state policies in 2000-01 and 2006-07..................................................................................................... 45 Table 18. Comparison of highest frequency ELL-responsive mathematics accommodations, 2000-01 and 2006-07 ...................................................................................................................... 45 List of Figures Figure 1. Total accommodations contrasted to number of ELL-responsive accommodations by category ..........................................................................................................................................14 Figure 2. Decision flow chart for selecting additional LEP accommodations ..............................28 Figure 3. Numbers of accommodations for ELLs in 2000-01 state policies .................................40 Figure 4. Numbers of accommodations for ELLs in 2006-07 .......................................................40 Figure 5. Percent of accommodations in 2000-01 state policies that were ELL-responsive .........42 Figure 6. Percent of accommodations in 2006-07 state policies that were ELL-responsive .........42 Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu viii Executive Summary The Descriptive Study examines state assessment policies for accommodating English language learners (ELLs) in the 50 U.S. states and the District of Columbia. Qualitative and quantitative methods were used to address the following research questions:  To what extent are state assessment policies responsive to ELLs’ unique linguistic needs?  To what extent are state policies guiding decision making and monitoring practices?  What are the most frequently allowed ELL-responsive accommodations and to what extent are these accommodations research based?  In what ways have state assessment policies for accommodating ELLs changed since the inception of NCLB? When taking assessments, English-proficient students apply automatized language processing skills and knowledge of academic English to focus on test content. In contrast, for students who are still in the process of acquiring English, a content test in English is likely to introduce construct-irrelevant variance which may impede the student from being able to demonstrate what he/she knows and can do (Abedi, 2005; Kopriva, 2008; LaCelle-Peterson & Rivera, 1994; Rivera, Collum, Shafer Willner, & Sia Jr., 2006). An accommodation for ELLs involves changes to testing procedures, testing materials, or the testing situation in order to allow students meaningful participation in an assessment. Effective accommodations for ELLs address the unique linguistic and socio-cultural needs of the student without altering the test construct. Accommodated scores should be sufficiently equivalent in scale that they can be pooled with unaccommodated scores. From July through December 2007, the research team collected electronic editions of the 2006- 07 State Education Agency (SEA) assessment policies for the 50 states and the District of Columbia, and identified relevant sections related to the accommodation of ELLs. The research team conducted 30-60 minute verification phone calls with 48 of the 51 SEA Assessment and Title III staff to verify their policy document(s) and gain their input. Accommodations in state polices were counted and examined for ELL-responsiveness. An ELL- responsive accommodation was operationalized as one that is likely to reduce construct- irrelevant variance due to language. Applying the operational definition, the research team distinguished ELL-responsive accommodations from non-ELL-responsive accommodations and grouped the latter into one of three categories: (a) accommodations designed for students with disabilities; (b) test administration practices or adjustments, such as small group administration, that can support the administration of an accommodation; and (c) qualifications and activities that prepare students and staff for test administration. The research team organized ELL-responsive accommodations according to Rivera et al.’s (2006) taxonomy of direct and indirect linguistic support and by the language of the assessment (i.e., English or native language) for which they were intended. The team also counted the accommodations in each state policy by the content assessment for which they were specified. Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu ix To investigate the extent to which accommodations were responsive to the needs of ELLs, the research team identified indicators in the policies to quantify the extent to which a state assessment policy was ELL-responsive. The research team codified if state policy guidance to districts included; (1) criteria for selecting accommodations for ELLs; (2) requirements for using accommodation prior to testing; and/or (3) criteria for monitoring the accommodations provided. Finally, to assess the extent to which state policies for accommodating ELLs had changed between 2000-01 and 2006-07, the research team compared data for the two points in time. To answer research question one, to what extent are state assessment policies responsive to ELLs’ unique linguistic needs, the following areas were examined: (1) the distribution of accommodations within and across state policies, (2) indicators of ELL-responsiveness within state policies, and (3) the extent to which the policies for accommodating ELLs were based on policies designed for students with disabilities. More than half (64 of 104) of the accommodations allowed across all 51 state policies did not meet the criteria for ELL-responsiveness. The range in the number (2 to 57) of ELL-responsive accommodations suggests different understandings among states about which accommodations are ELL-responsive. While many state policies are responsive to the linguistic needs of ELLs, there are some states’ policies that are more ELL-responsive than others. Lastly, a regression analyses indicated that the more accommodations a state includes in its policy, the less likely the policy is ELL-responsive. To examine the second research question, the extent to which state policies guide decision making and monitoring practices, the research team investigated in policies the criteria or guidance provided to local decision makers to assign accommodations to ELLs. A few state policies include criteria or guidance for assigning specific kinds of accommodations to ELLs with specific background characteristics, particularly for translated or dual language tests or for tests that are sight translated. However, the majority of states provide little or no guidance to assist decision makers in matching accommodations to student background characteristics. With regard to monitoring accommodation decisions, two-third of the state policies (34 of 51) include a requirement to monitor the accommodation decision making process. Some policies require data to be collected and maintained at the local level. Other policies require the decision to accommodate to be tracked and reported to the state. About a third of states require tracking of the specific accommodations administered. It is unclear from the state policies in what ways or to what extent these data are used by the state to make improvements to policy and/or practice. The third research question examined the most frequently allowed ELL-responsive accommodations and the extent to which these accommodations were research based. Two-thirds of state policies specify the content areas for which all accommodations for ELLs are intended, while just under one-third of state policies specify the content area for some, but not all accommodations. Only three state policies do not specify a content area for any accommodation. Most states appear to use the same accommodations allowed for mathematics assessments for reading/language arts. Of the states policies that specify the content for which the accommodation is appropriate, all but two specify an equal or greater number for mathematics Descriptive Analysis of State Assessment Policies 2008 © GW-CEEE | www.ceee.gwu.edu