Reading Framework for the 2015 National Assessment of Educational Progress National Assessment Governing Board U.S. Department of Education W I NAEP? HAT S The National Assessment of Educational Progress (NAEP) is a continuing and nationally representative measure of trends in academic achievement of U.S. elementary and secondary students in various sub- jects. For nearly four decades, NAEP assessments have been conducted periodically in reading, mathe- matics, science, writing, U.S. history, civics, geography, and other subjects. By collecting and reporting information on student performance at the national, state, and local levels, NAEP is an integral part of our nation’s evaluation of the condition and progress of education. THE 2014–2015 NATIONAL ASSESSMENT GOVERNING BOARD The National Assessment Governing Board was created by Congress to formulate policy for NAEP. Among the Governing Board’s responsibilities are developing objectives and test specifications and designing the assessment methodology for NAEP. Members Rebecca Gagnon Dale Nowlin School Board Member Teacher and Mathematics Department Terry Mazany, Chair Minneapolis Public Schools Chair President and CEO Minneapolis, Minnesota Bartholomew Consolidated School The Chicago Community Trust Corporation Chicago, Illinois Shannon Garrison Columbus, Indiana Fourth-Grade Teacher Susan Pimentel, Vice Chair Solano Avenue Elementary School Father Joseph O’Keefe, S.J. Education Consultant Los Angeles, California Professor Hanover, New Hampshire Boston College Lynch School of Honorable James Geringer Education Andrés Alonso Director of Policy and Public Sector Chestnut Hill, Massachusetts Professor of Practice Strategies Harvard Graduate School of Education Environmental Systems Research W. James Popham Cambridge, Massachusetts Institute (ESRI) Emeritus Professor Cheyenne, Wyoming University of California, Los Angeles Honorable Mitchell D. Chester Los Angeles, California Commissioner of Elementary and Doris R. Hicks Secondary Education Principal and CEO B. Fielding Rolston Massachusetts Department of Dr. Martin Luther King, Jr. Charter Chairman Elementary and Secondary Education School for Science and Technology Tennessee State Board of Education Malden, Massachusetts New Orleans, Louisiana Kingsport, Tennessee Lucille E. Davy Andrew Dean Ho Cary Sneider President and CEO Professor Associate Research Professor Transformative Education Solutions, Harvard Graduate School of Education Portland State University LLC Cambridge, Massachusetts Portland, Oregon Pennington, New Jersey Honorable Terry Holliday Chasidy White Louis M. Fabrizio Commissioner of Education Eighth-Grade Teacher Data, Research and Federal Policy Kentucky Department of Education Brookwood Middle School Director Lexington, Kentucky Brookwood, Alabama North Carolina Department of Public Instruction Tonya Matthews Ex-officio Member Raleigh, North Carolina President and CEO Sue Betka Michigan Science Center Acting Director Frank K. Fernandes Detroit, Michigan Institute of Education Sciences Principal U.S. Department of Education Kaimuki Middle School Tonya Miles Washington, D.C. Honolulu, Hawaii General Public Representative Mitchellville, Maryland Honorable Anitere Flores Senator Honorable Ronnie Musgrove Florida Senate Founding Member Miami, Florida Musgrove/Smith Law Jackson, Mississippi National Assessment Governing Board Terry Mazany Chair Susan Pimentel Vice Chair Cornelia S. Orr Executive Director Mary Crovo Deputy Executive Director Developed for the National Assessment Governing Board under contract number ED–02– R–0007 by the American Institutes for Research For further information, contact: National Assessment Governing Board 800 N. Capitol St. NW Suite 825 Washington, DC 20002-4233 www.nagb.org January 2015 For sale by the U.S. Government Printing Office Superintendent of Documents, Mail Stop: SSOP, Washington, DC 20402-0328 TABLE OF CONTENTS List of Exhibits ........................................................................................................................ ii Executive Summary ............................................................................................................... iii Text Types ....................................................................................................................... iv Meaning Vocabulary Assessment ................................................................................... iv Item Design ..................................................................................................................... iv 12th Grade NAEP .............................................................................................................. v Reporting Results .............................................................................................................. v Reporting Trend Data ........................................................................................................ v Sample Items ................................................................................................................... vi Preface by the National Assessment Governing Board ..................................................... vii NAEP Reading Project Staff and Committees .................................................................... ix Chapter One: Overview ..........................................................................................................1 NAEP Overview ................................................................................................................ 1 Overview of NAEP Reading Assessment ......................................................................... 6 Chapter Two: Content and Design of NAEP in Reading ...................................................15 Texts on the NAEP Reading Assessment to Be Included ............................................... 15 Literary Text .................................................................................................................... 16 Informational Text ........................................................................................................... 22 Characteristics of Texts Selected for Inclusion ............................................................... 28 Vocabulary on the NAEP Reading Assessment .............................................................. 32 Cognitive Targets ............................................................................................................ 36 Item Types ....................................................................................................................... 41 Chapter Three: Reporting Results .......................................................................................43 No Child Left Behind Provisions for NAEP Reporting .................................................. 43 Achievement Levels ........................................................................................................ 43 Reporting NAEP Results ................................................................................................. 44 Reporting State NAEP Results ........................................................................................ 45 Reporting Trend Data ...................................................................................................... 45 Bibliography ...........................................................................................................................49 Definition of Reading ...................................................................................................... 49 Text Types, Matrices, and Cognitive Targets ................................................................. 49 Vocabulary Assessment .................................................................................................. 52 Appendix A: Glossary of Terms ...........................................................................................57 Appendix B: NAEP Reading Achievement Level Definitions ...........................................63 NAEP Reading Achievement Levels—Grade 4 .............................................................. 63 NAEP Reading Achievement Levels—Grade 8 .............................................................. 64 NAEP Reading Achievement Levels—Grade 12 ............................................................ 66 Appendix C: Special Studies: NAEP Reading Framework ...............................................69 NAEP 2015 READING FRAMEWORK i L E IST OF XHIBITS Exhibit 1. Percentage distribution of literary and informational passages ....................... 11 Exhibit 2. Similarities and differences: 1992–2007 and 2009–2015 NAEP reading frameworks ....................................................................................................... 14 Exhibit 3. Literary text matrix: Fiction ............................................................................. 17 Exhibit 3 (continued). Literary text matrix: Literary nonfiction ....................................... 19 Exhibit 3 (continued). Literary text matrix: Poetry .......................................................... 21 Exhibit 4. Informational text matrix: Exposition .............................................................. 23 Exhibit 4 (continued). Informational text matrix: Argumentation and persuasive text .... 25 Exhibit 4 (continued). Informational text matrix: Procedural texts and documents ......... 27 Exhibit 5. Passage lengths for grades 4, 8, and 12 ............................................................ 29 Exhibit 6. Considerations for selecting stimulus material ................................................ 32 Exhibit 7. Considerations for selecting vocabulary items and distractors ........................ 35 Exhibit 8. Cognitive targets .............................................................................................. 40 Exhibit 9. Percentage distribution of cognitive targets by grade ...................................... 41 Exhibit 10. Percentage distribution of time to be spent on specific item types ................ 42 Exhibit 11. Generic NAEP achievement levels ................................................................ 44 Exhibit 12. Years of administration of NAEP reading assessments ................................. 46 NAEP 2015 READING FRAMEWORK ii E S XECUTIVE UMMARY As the ongoing national indicator of what American students know and can do, the National Assessment of Educational Progress (NAEP) in Reading regularly collects achievement information on representative samples of students in grades 4, 8, and 12. Through The Nation’s Report Card, the NAEP Reading Assessment reports how well students perform in reading various texts and responding to those texts by answering multiple-choice and constructed-response questions. The information NAEP provides about student achievement helps the public, educators, and policymakers understand strengths and weaknesses in student performance and make informed decisions about ed- ucation. The 2015 NAEP Reading Assessment will measure national, regional, state, and sub- group achievement in reading but is not designed to report individual student or school performance. The assessment will measure students’ reading comprehension and their ability to apply vocabulary knowledge to assist them in comprehending what they read. The reading assessment will use the same framework used in 2009. The public will have access to performance results and released questions through NAEP reports and websites. This document, the Reading Framework for the 2015 National Assessment of Education- al Progress, presents the conceptual base for, and discusses the content of, the assessment. It is intended for a broad audience. A more detailed technical document, the Reading Assessment and Item Specifications for the National Assessment of Educational Progress, is available on the Web. The specifications will provide information to guide passage selection, item development, and other aspects of test development. Both the framework and the specifications documents are available to the public at www.nagb.org/publications/frameworks.htm. The Governing Board, the policymaking body for NAEP, has stated that the NAEP Read- ing Assessment will measure reading comprehension by asking students to read passages written in English and to answer questions about what they have read. The framework “shall not endorse or advocate a particular pedagogical approach … but shall focus on important, measurable indicators of student achievement” (Governing Board 2002). Alt- hough broad implications for instruction may be inferred from the assessment, NAEP does not specify how reading should be taught; nor does it prescribe a particular curricu- lar approach to teaching reading. The NAEP Reading Framework results from the work of many individuals and organiza- tions involved in reading and reading education, including researchers, policymakers, ed- ucators, and other members of the public. Their work was guided by scientifically based literacy research that conceptualizes reading as a dynamic cognitive process as reflected in the following definition of reading. NAEP 2015 READING FRAMEWORK iii Reading is an active and complex process that involves: • Understanding written text. • Developing and interpreting meaning. • Using meaning as appropriate to type of text, purpose, and situation. This definition applies to the assessment of reading achievement on NAEP and is not intended to be an inclusive definition of reading or reading instruction. TEXT TYPES This framework recognizes that reading behaviors such as recognizing and using features of text, making sense of sentences and paragraphs, and comprehending vocabulary occur regardless of text type. However, other reading behaviors vary with the type of text en- countered by a reader. Thus, the NAEP Reading Framework includes two types of texts on the assessment: literary texts, which include fiction, literary nonfiction, and poetry; and informational texts, which include exposition, argumentation and persuasive text, and procedural text and documents. MEANING VOCABULARY ASSESSMENT The NAEP Reading Framework includes a more systematic approach to vocabulary assessment than the NAEP Reading Framework used from 1992 through 2007. Vocabu- lary assessment will occur in the context of a passage; that is, vocabulary items will function both as a measure of passage comprehension and as a test of readers’ specific knowledge of the word’s meaning as intended by the passage author. A sufficient number of vocabulary items at each grade will provide reliable and valid information about students’ vocabulary knowledge. ITEM DESIGN The framework includes the following cognitive targets, or behaviors and skills, for items from both literary and informational texts: locate/recall, integrate/interpret, and critique/ evaluate. These cognitive targets illustrate the complex nature of the reading process whereas the corresponding behaviors highlight the different behaviors elicited by differ- ent text types. To measure these cognitive skills, students will respond to both multiple- choice and constructed-response items with varying distributions of question type by grade level. Students in grade 4 will spend approximately half of the assessment time responding to multiple-choice items and half responding to constructed-response items. Students in grades 8 and 12 will spend a greater amount of time on constructed-response items. NAEP 2015 READING FRAMEWORK iv 12TH GRADE NAEP In May 2005, the Governing Board adopted a policy regarding NAEP and 12th-grade preparedness. The policy states that NAEP will pursue assessment and reporting on 12th- grade student achievement as it relates to preparedness for postsecondary education and training. This policy resulted from recommendations of the Board’s National Commis- sion on NAEP 12th Grade Assessment and Reporting in March 2004. Subsequent studies and deliberations by the Board took place during 2004 and 2005. In reading, the Board adopted minor modifications to the 2009 NAEP Reading Frame- work at grade 12 based on a comprehensive analysis of the framework conducted by Achieve. The current version of the reading framework incorporates these modifications at grade 12 to enable NAEP to measure and report on preparedness for postsecondary en- deavors. The 2015 NAEP Reading Assessment will use the same framework used in 2009. REPORTING RESULTS Results are reported in two ways: as average scores for groups of students on the NAEP 0–500 scale and as percentages of students who attain each of the three achievement lev- els (Basic, Proficient, and Advanced, according to definitions adopted by the Governing Board). NAEP scores are always reported at the aggregate level; scores are not produced for individual schools or students. REPORTING TREND DATA The Reading Framework for the 2009 National Assessment of Educational Progress replaced the framework used first for the 1992 reading assessment and then for subse- quent reading assessments through 2007. Compared with the previous framework, the 2009 reading framework included more emphasis on literary and informational texts, a redefinition of reading cognitive processes, a new systematic assessment of vocabulary knowledge, and the addition of poetry to grade 4. The 2009 NAEP Reading Report Card included trend data on student reading perfor- mance from 1992 to 2009. Results from special analyses determined the 2009 reading assessment results could be compared with those from earlier assessment years. These special analyses started in 2007 and included in-depth comparisons of the frameworks and the test questions, as well as an examination of how the same students performed on the 2009 assessment and the earlier assessment. A summary of these special analyses and an overview of the differences between the previous framework and the 2009 framework are available on the Web at nces.ed.gov/nationsreportcard/reading/trend_study.asp. The 2015 NAEP Reading Report Card will report trends in student reading performance from 1992 to 2015. NAEP 2015 READING FRAMEWORK v SAMPLE ITEMS Sample passages and items released to the public may be viewed on the NAEP Questions Tool at nces.ed.gov/nationsreportcard/itmrlsx/. The Questions Tool also includes perfor- mance results, scoring rubrics, and student responses to NAEP reading items. NAEP 2015 READING FRAMEWORK vi