IES PRACTICE GUIDE WHAT WORKS CLEARINGHOUSE IIImmmppprorrvooinvvgii nnAdggo lAAesddceoonlltee Lssitccereeanncytt: L Liitteerraaccyy:: Effective Classroom and EEffffeeccttiivvee CCllaassssrroooomm aanndd Intervention Practices IInntteerrvveennttiioonn PPrraaccttiicceess NCEE 2008-4027 U.S. DEPARTMENT OF EDUCATION The Institute of Education Sciences (IES) publishes practice guides in education to bring the best available evidence and expertise to bear on the types of systemic challenges that cannot currently be addressed by single interventions or programs. Authors of practice guides seldom conduct the types of systematic literature searches that are the backbone of a meta-analysis, although they take advantage of such work when it is already published. Instead, authors use their expertise to identify the most important research with respect to their recommendations, augmented by a search of recent publications to ensure that research citations are up-to-date. Unique to IES-sponsored practice guides is that they are subjected to rigorous exter- nal peer review through the same office that is responsible for independent review of other IES publications. A critical task for peer reviewers of a practice guide is to determine whether the evidence cited in support of particular recommendations is up-to-date and that studies of similar or better quality that point in a different di- rection have not been ignored. Because practice guides depend on the expertise of their authors and their group decisionmaking, the content of a practice guide is not and should not be viewed as a set of recommendations that in every case depends on and flows inevitably from scientific research. The goal of this practice guide is to formulate specific and coherent evidence-based recommendations that educators can use to improve literacy levels among adoles- cents in upper elementary, middle, and high schools. The target audience is teach- ers and other school personnel with direct contact with students, such as coaches, counselors, and principals. The guide includes specific recommendations for edu- cators and the quality of evidence that supports these recommendations. IES PRACTICE GUIDE Improving Adolescent Literacy: Ef fective Classroom and Intervention Practices August 2008 Panel Michael L. Kamil (Chair) STANFORD UNIVERSITY Geoff rey D. Borman UNIVERSITY OF WISCONSIN—MADISON Janice Dole UNIVERSITY OF UTAH Cathleen C. Kral BOSTON PUBLIC SCHOOLS Terry Salinger AMERICAN INSTITUTES FOR RESEARCH Joseph Torgesen FLORIDA STATE UNIVERSITY Staff Xinsheng “Cindy” Cai Fiona Helsel Yael Kidron Elizabeth Spier AMERICAN INSTITUTES FOR RESEARCH NCEE 2008-4027 U.S. DEPARTMENT OF EDUCATION This report was prepared for the National Center for Education Evaluation and Re- gional Assistance, Institute of Education Sciences under Contract ED-02-CO-0022. Disclaimer The opinions and positions expressed in this practice guide are the authors’ and do not necessarily represent the opinions and positions of the Institute of Education Sci- ences or the U.S. Department of Education. This practice guide should be reviewed and applied according to the specific needs of the educators and education agency using it, and with full realization that it represents the judgments of the review panel regarding what constitutes sensible practice, based on the research that was available at the time of publication. This practice guide should be used as a tool to assist in decisionmaking rather than as a “cookbook.” Any references within the document to specific education products are illustrative and do not imply endorse- ment of these products to the exclusion of other products that are not referenced. U.S. Department of Education Margaret Spellings Secretary Institute of Education Sciences Grover J. Whitehurst Director National Center for Education Evaluation and Regional Assistance Phoebe Cottingham Commissioner August 2008 This report is in the public domain. While permission to reprint this publication is not necessary, the citation should be: Kamil, M. L., Borman, G. D., Dole, J., Kral, C. C., Salinger, T., and Torgesen, J. (2008). Improving adolescent literacy: Effective classroom and intervention practices: A Prac- tice Guide (NCEE #2008-4027). Washington, DC: National Center for Education Evalu- ation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc. This report is available on the IES Web site at http://ies.ed.gov/ncee/wwc. Alternative Formats On request, this publication can be made available in alternative formats, such as Braille, large print, audiotape, or computer diskette. For more information, call the Alternative Format Center at (202) 205–8113. Improving Adolescent Literacy: Effective Classroom and Intervention Practices Contents Introduction 1 The What Works Clearinghouse standards and their relevance to this guide 3 Improving Adolescent Literacy: Effective Classroom and Intervention Practices 4 Overview 4 Scope of the practice guide 8 Checklist for carrying out the recommendations 9 Recommendation 1. Provide explicit vocabulary instruction 11 Recommendation 2. Provide direct and explicit comprehension strategy instruction 16 Recommendation 3. Provide opportunities for extended discussion of text meaning and interpretation 21 Recommendation 4. Increase student motivation and engagement in literacy learning 26 Recommendation 5. Make available intensive and individualized interventions for struggling readers that can be provided by trained specialists 31 Conclusion 37 Appendix A. Postscript from the Institute of Education Sciences 38 Appendix B. About the Authors 41 Appendix C. Disclosure of potential conflicts of interest 42 Appendix D. Technical information on the studies 43 References 52 ( iii ) IMPROVING ADOLESCENT LITERACY: EFFECTIVE CLASSROOM AND INTERVENTION PRACTICES List of tables 1. Institute of Education Sciences levels of evidence for practice guides 2 2. Recommendations and corresponding levels of evidence to support each 7 ( iv ) Introduction reading instruction, we use this informa- tion to make broader points about im- The goal of this practice guide is to present proving practice. In this guide we have specific and coherent evidence-based rec- tried to take findings from research or ommendations that educators can use to practices recommended by experts and improve literacy levels among adolescents describe how recommendations might ac- in upper elementary, middle, and high tually unfold in school settings. In other schools. The panel purposefully included words, we aim to provide sufficient detail students in 4th and 5th grades within the so that educators will have a clear sense realm of adolescents because their in- of the steps necessary to make use of the structional needs related to literacy have recommendations. more in common with those of students in middle and high school than they do A unique feature of practice guides is the with students in early elementary grades. explicit and clear delineation of the qual- Many students in grades 4 and up experi- ity—as well as quantity— of evidence that ence difficulty acquiring the advanced lit- supports each claim. To do this, we used eracy skills needed to read in the content a semi-structured hierarchy suggested by areas.1 The target audience for the practice IES. This classification system uses both guide is teachers and other school person- the quality and the quantity of available nel who have direct contact with students, evidence to help determine the strength of such as coaches, counselors, and princi- the evidence base grounding each recom- pals. The practice guide includes specific mended practice (table 1). recommendations for educators along with a discussion of the quality of evidence that Strong refers to consistent and generaliz- supports these recommendations. able evidence that a practice causes bet- ter outcomes for students in measures of We, the authors, are a small group with reading proficiency.2 expertise on this topic. The range of evi- dence we considered in developing this Moderate refers either to evidence from guide is vast, ranging from experimental studies that allow strong causal conclu- studies in which reading was the depen- sions but cannot be generalized with as- dent variable, to trends in the National As- surance to the population on which a rec- sessment of Educational Progress (NAEP) ommendation is focused (perhaps because data, to correlational and longitudinal the findings have not been widely repli- studies, again with reading as the major cated) or to evidence from studies that variable of interest. For questions about are generalizable but have more causal what works best, high-quality experimen- ambiguity than offered by experimental tal and quasi- experimental studies—such designs (statistical models of correlational as those meeting the criteria of the What data or group comparison designs for Works Clearinghouse (http://www.what- which equivalence of the groups at pretest works.ed.gov)—have a privileged posi- is uncertain). tion. In all cases we pay particular atten- tion to findings that are replicated across Low refers to expert opinion based on rea- studies. sonable extrapolations from research and theory on other topics and evidence from Although we draw on evidence about the effectiveness of specific practices in 2. Following What Works Clearinghouse guide- lines, we consider a positive, statistically signifi- 1. Biancarosa and Snow (2004); Heller and Green- cant effect or large effect size (greater than 0.25) leaf (2007). as an indicator of positive effects. ( 1 ) INTRODuCTION Table 1. Institute of Education Sciences levels of evidence for practice guides In general, characterization of the evidence for a recommendation as strong requires both studies with high internal validity (i.e., studies whose designs can support causal conclusions) and studies with high external validity (i.e., studies that in total include enough of the range of participants and settings on which the recommendation is focused to support the conclusion that the results can be generalized to those participants and settings). Strong evidence for this practice guide is operationalized as: • A systematic review of research that generally meets the standards of the What Works Clearing- house (WWC) (see http://ies.ed.gov/ncee/wwc/) and supports the effectiveness of a program, prac- tice, or approach with no contradictory evidence of similar quality; OR Strong • Several well-designed, randomized controlled trials or well designed quasi- experiments that gen- erally meet the WWC standards and support the effectiveness of a program, practice, or approach, with no contradictory evidence of similar quality; OR • One large, well-designed, randomized controlled, multisite trial that meets the WWC standards and supports the effectiveness of a program, practice, or approach, with no contradictory evi- dence of similar quality; OR • For assessments, evidence of reliability and validity that meets the Standards for Educational and Psychological Testing.a In general, characterization of the evidence for a recommendation as moderate requires studies with high internal validity but moderate external validity, or studies with high external validity but mod- erate internal validity. In other words, moderate evidence is derived from studies that support strong causal conclusions but where generalization is uncertain, or studies that support the generality of a relationship but where the causality is uncertain. Moderate evidence for this practice guide is opera- tionalized as: • Experiments or quasi- experiments generally meeting the WWC standards and supporting the ef- fectiveness of a program, practice, or approach with small sample sizes and/or other conditions of implementation or analysis that limit generalizability and no contrary evidence; OR • Comparison group studies that do not demonstrate equivalence of groups at pretest and there- Moderate fore do not meet the WWC standards but that (a) consistently show enhanced outcomes for par- ticipants experiencing a particular program, practice, or approach and (b) have no major flaws related to internal validity other than lack of demonstrated equivalence at pretest (e.g., only one teacher or one class per condition, unequal amounts of instructional time, highly biased outcome measures); OR • Correlational research with strong statistical controls for selection bias and for discerning influ- ence of endogenous factors and no contrary evidence; OR • For assessments, evidence of reliability that meets the Standards for Educational and Psychological Testingb but with evidence of validity from samples not adequately representative of the popula- tion on which the recommendation is focused. In general, characterization of the evidence for a recommendation as low means that the recommenda- tion is based on expert opinion derived from strong findings or theories in related areas and/or expert Low opinion buttressed by direct evidence that does not rise to the moderate or strong levels. Low evidence is operationalized as evidence not meeting the standards for the moderate or high levels. a. American Educational Research Association, American Psychological Association, and National Council on Measure- ment in Education (1999). b. Ibid. ( 2 ) INTRODuCTION studies that do not meet the standards for studies with no design flaws and ran- moderate or strong evidence. domized controlled trials that have problems with randomization, attri- The What Works Clearinghouse tion, or disruption. standards and their relevance to this guide • Does Not Meet Evidence Screens for studies that do not provide strong evi- In terms of the levels of evidence indicated dence of causal validity. in table 1, we rely on What Works Clearing- house (WWC) evidence standards to assess Appendix D provides more technical in- the quality of evidence supporting educa- formation about the studies and our de- tional programs and practices. The WWC cisions regarding the level of evidence addresses evidence for the causal validity for each recommendation. To illustrate of instructional programs and practices the types of studies reviewed, we de- according to WWC standards. Informa- scribe one study for each recommenda- tion about these standards is available at tion. Our goal in doing this is to provide http://ies.ed.gov/ncee/wwc. The technical interested readers with more detail about quality of each study is rated and placed the research designs, the intervention into one of three categories: components, and the way impact was measured. • Meets Evidence Standards for random- ized controlled trials and regression Dr. Michael Kamil discontinuity studies that provide the Dr. Geoffrey D. Borman strongest evidence of causal validity. Dr. Janice Dole Cathleen C. Kral • Meets Evidence Standards with Res- Dr. Terry Salinger ervations for all quasi- experimental Dr. Joseph Torgesen ( 3 ) Improving Adolescent attention to the challenges of improving Literacy: Effective reading instruction in upper elementary, middle, and high school. Yet reading in- Classroom and struction as a formal part of the curricu- Intervention Practices lum typically decreases as students move beyond upper elementary grades. Overview To acquire the skills they need, students must work hard to refine and build upon Data from the 2007 National Assessment their initial reading skills, and teachers of Educational Progress (NAEP) in read- in upper elementary grades and in mid- ing report that 69 percent of 8th grade dle and high school classes should help students fall below the proficient level in students acquire more advanced skills their ability to comprehend the meaning once they understand the demands that of text at their grade level.1 Equally alarm- content area tasks actually present, es- ing, 26 percent of students read below the pecially to students who struggle with basic level, which means that they do not reading.7 However, many teachers re- have sufficient reading ability to under- port feeling unprepared to help their stu- stand and learn from text at their grade dents or do not think that teaching read- level. When these data are coupled with ing skills in content-area classes is their reports showing that even high school responsibility.8 students with average reading ability are currently unprepared for the literacy de- For more than 50 years9 the realities of stu- mands of many workplace and postsec- dent reading difficulties and teacher lack ondary educational settings, the need for of preparation to address them have been improved literacy instruction of adoles- met by calls for more instruction in higher- cents is apparent.2 level reading skills for adolescents and for professional development in content- Reading ability is a key predictor of achieve- area reading instruction for middle and ment in mathematics and science,3 and the high school teachers. Although the debate global information economy requires to- about the role of content-area teachers in day’s American youth to have far more ad- reading instruction continues,10 the time vanced literacy skills than those required has come to consider seriously the support of any previous generation.4 However, as that needs to be given to struggling read- long-term NAEP data5 and other studies ers and the role that every teacher needs show,6 improvements in the literacy skills to play in working toward higher levels of of older students have not kept pace with literacy among all adolescents, regardless the increasing demands for literacy in the of their reading abilities. workplace. These studies, and those men- tioned earlier, suggest the need for serious A significant difficulty in working toward higher levels of literacy involves struc- tural barriers at the middle and high 1. Lee, Griggs, and Donahue (2007). school levels that need to be overcome. 2. Pennsylvania Department of Education (2004); Williamson (2004). 7. Heller and Greenleaf (2007). 3. ACT (2006). 8. Heller and Greenleaf (2007). 4. Snow, Burns, and Griffin (1998). 9. Artley (1944); Moore, Readence, and Rickman 5. Perie and Moran (2005). (1983). 6. ACT (2006). 10. Heller and Greenleaf (2007). ( 4 )