What Is NAEP? The National Assessment of Educational Progress (NAEP) is a continuing and nationally representative measure of trends in academic achievement of U.S. elementary and secondary students in various sub- jects. For nearly four decades, NAEP assessments have been conducted periodically in reading, mathe- matics, science, writing, U.S. history, civics, geography, and other subjects. By collecting and reporting information on student performance at the national, state, and local levels, NAEP is an integral part of our nation’s evaluation of the condition and progress of education. THE 2016–2017 NATIONAL ASSESSMENT GOVERNING BOARD The National Assessment Governing Board was created by Congress to formulate policy for NAEP. Among the Governing Board’s responsibilities are developing objectives and test specifications and designing the assessment methodology for NAEP. Members Honorable James Geringer Honorable Jeanette Nuñez Director of Policy and Public Sector Legislator Terry Mazany, Chair Strategies Florida House of Representatives President and CEO Environmental Systems Research Miami, Florida The Chicago Community Trust Institute (ESRI) Chicago, Illinois Cheyenne, Wyoming Father Joseph O’Keefe, S.J. Professor Lucille E. Davy, Vice Chair Doris R. Hicks Boston College Lynch School of President and CEO Principal and CEO Education Transformative Education Solutions, Dr. Martin Luther King, Jr. Charter Chestnut Hill, Massachusetts LLC School for Science and Technology Pennington, New Jersey New Orleans, Louisiana W. James Popham Emeritus Professor Alberto Carvalho Andrew Dean Ho University of California, Los Angeles Superintendent Professor Los Angeles, California Miami-Dade County Public Schools Harvard Graduate School of Education Miami, Florida Cambridge, Massachusetts B. Fielding Rolston Chairman Honorable Mitchell D. Chester Carol Jago Tennessee State Board of Education Commissioner of Elementary and Associate Director Kingsport, Tennessee Secondary Education California Reading and Literature Massachusetts Department of Project at UCLA Linda Rosen Elementary and Secondary Education Oak Park, Illinois CEO Malden, Massachusetts Change the Equation Tonya Matthews Washington, DC Frank K. Fernandes President and CEO Principal Michigan Science Center Cary Sneider Kaimuki Middle School Detroit, Michigan Associate Research Professor Honolulu, Hawaii Portland State University Tonya Miles Portland, Oregon Honorable Anitere Flores General Public Representative Senator Mitchellville, Maryland Ken Wagner Florida Senate Commissioner for Elementary and Miami, Florida Honorable Ronnie Musgrove Secondary Education Founding Member Rhode Island Department of Education Rebecca Gagnon Musgrove/Smith Law Providence, Rhode Island School Board Member Jackson, Mississippi Minneapolis Public Schools Chasidy White Minneapolis, Minnesota Dale Nowlin Eighth-Grade Teacher Teacher and Mathematics Department Brookwood Middle School Shannon Garrison Chair Brookwood, Alabama Fourth-Grade Teacher Bartholomew Consolidated School Solano Avenue Elementary School Corporation Joe Willhoft Los Angeles, California Columbus, Indiana Consultant Tacoma, Washington National Assessment Governing Board Terry Mazany Chair Lucille E. Davy Vice Chair William Bushaw Executive Director Lisa Stooksberry Deputy Executive Director Developed for the National Assessment Governing Board under contract number ED–02– R–0007 by the American Institutes for Research For further information, contact: National Assessment Governing Board 800 N. Capitol St. NW Suite 825 Washington, DC 20002-4233 www.nagb.org January 2017 Publication Note The 2017 NAEP Reading Framework is the same framework first developed for the 2009 NAEP Reading Assessment, which includes 2009 modifications for 12th grade to support NAEP reporting on academic preparedness for postsecondary endeavors. Continuity in the NAEP Reading Framework enables reporting of student achievement trends over time. To reflect this continuity, this edition reflects updated dates and refer- ences to legislation, National Assessment Governing Board actions, and NAEP activities, including the 2017 transition to digital-based assessment. For sale by the U.S. Government Printing Office Superintendent of Documents, Mail Stop: SSOP, Washington, DC 20402-0328 TABLE OF CONTENTS List of Exhibits ........................................................................................................................ ii Executive Summary ............................................................................................................... iii Text Types ....................................................................................................................... iv Meaning Vocabulary Assessment ................................................................................... iv Item Design ..................................................................................................................... iv 12th Grade NAEP .............................................................................................................. v Reporting Results .............................................................................................................. v Reporting Trend Data ........................................................................................................ v Sample Items ................................................................................................................... vi Preface by the National Assessment Governing Board ..................................................... vii NAEP Reading Project Staff and Committees .................................................................... ix Chapter One: Overview .......................................................................................................... 1 NAEP Overview ................................................................................................................ 1 Overview of NAEP Reading Assessment ......................................................................... 6 Chapter Two: Content and Design of NAEP in Reading ................................................... 15 Texts on the NAEP Reading Assessment to Be Included ............................................... 15 Literary Text .................................................................................................................... 16 Informational Text ........................................................................................................... 22 Characteristics of Texts Selected for Inclusion ............................................................... 28 Vocabulary on the NAEP Reading Assessment .............................................................. 32 Cognitive Targets ............................................................................................................ 36 Item Types ....................................................................................................................... 41 Chapter Three: Reporting Results ....................................................................................... 43 Legislative Provisions for NAEP Reporting ................................................................... 43 Achievement Levels ........................................................................................................ 43 Reporting NAEP Results ................................................................................................. 44 Reporting State NAEP Results ........................................................................................ 45 Reporting Trend Data ...................................................................................................... 45 Bibliography ........................................................................................................................... 49 Definition of Reading ...................................................................................................... 49 Text Types, Matrices, and Cognitive Targets. ................................................................. 49 Vocabulary Assessment ................................................................................................... 52 Appendix A: Glossary of Terms ........................................................................................... 57 Appendix B: NAEP Reading Achievement Level Definitions............................................ 63 NAEP Reading Achievement Levels—Grade 4 .............................................................. 63 NAEP Reading Achievement Levels—Grade 8 .............................................................. 64 NAEP Reading Achievement Levels—Grade 12 ............................................................ 66 Appendix C: Special Studies: NAEP Reading Framework ............................................... 69 Developmental Study: Meaning Vocabulary Assessment. .............................................. 69 Special Study: English Language Learners ..................................................................... 70 Special Study: Gender Differences .................................................................................. 71 NAEP 2017 READING FRAMEWORK i L E IST OF XHIBITS Exhibit 1. Percentage distribution of literary and informational passages ....................... 11 Exhibit 2. Similarities and differences: 1992–2007 and 2009–2017 NAEP reading frameworks ................................................................................................................ 14 Exhibit 3. Literary text matrix: Fiction ............................................................................. 17 Exhibit 3 (continued). Literary text matrix: Literary nonfiction ....................................... 19 Exhibit 3 (continued). Literary text matrix: Poetry .......................................................... 21 Exhibit 4. Informational text matrix: Exposition .............................................................. 23 Exhibit 4 (continued). Informational text matrix: Argumentation and persuasive text .... 25 Exhibit 4 (continued). Informational text matrix: Procedural texts and documents ......... 27 Exhibit 5. Passage lengths for grades 4, 8, and 12 ............................................................ 29 Exhibit 6. Considerations for selecting stimulus material ................................................ 32 Exhibit 7. Considerations for selecting vocabulary items and distractors ........................ 35 Exhibit 8. Cognitive targets .............................................................................................. 40 Exhibit 9. Percentage distribution of cognitive targets by grade ...................................... 41 Exhibit 10. Percentage distribution of time to be spent on specific item types ................ 42 Exhibit 11. Generic NAEP achievement levels ................................................................ 44 Exhibit 12. Years of administration of NAEP reading assessments ................................. 46 NAEP 2017 READING FRAMEWORK ii E S XECUTIVE UMMARY As the ongoing national indicator of what American students know and can do, the National Assessment of Educational Progress (NAEP) in Reading regularly collects achievement information on representative samples of students in grades 4, 8, and 12. Through The Nation’s Report Card, the NAEP Reading Assessment reports how well students perform in reading various texts and responding to those texts by answering selected-response and constructed-response questions. The information NAEP provides about student achievement helps the public, educators, and policymakers understand strengths and weaknesses in student performance and make informed decisions about ed- ucation. The 2017 NAEP Reading Assessment will measure national, regional, state, and sub- group achievement in reading but is not designed to report individual student or school performance. The assessment will measure students’ reading comprehension and their ability to apply vocabulary knowledge to assist them in comprehending what they read. The reading assessment will use the same framework used in 2009. The public will have access to performance results and released questions through NAEP reports and websites. This document, the Reading Framework for the 2017 National Assessment of Education- al Progress, presents the conceptual base for and discusses the content of the assessment. It is intended for a broad audience. A more detailed technical document, the Reading As- sessment and Item Specifications for the National Assessment of Educational Progress, is available on the Web. The specifications will provide information to guide passage selec- tion, item development, and other aspects of test development. Both the framework and the specifications documents are available to the public at www.nagb.org/publications/frameworks.htm. The Governing Board, the policymaking body for NAEP, has stated that the NAEP Read- ing Assessment will measure reading comprehension by asking students to read passages written in English and to answer questions about what they have read. The framework “shall not endorse or advocate a particular pedagogical approach … but shall focus on important, measurable indicators of student achievement” (Governing Board 2002). Al- though broad implications for instruction may be inferred from the assessment, NAEP does not specify how reading should be taught; nor does it prescribe a particular curricu- lar approach to teaching reading. The NAEP Reading Framework results from the work of many individuals and organiza- tions involved in reading and reading education, including researchers, policymakers, ed- ucators, and other members of the public. Their work was guided by scientifically based literacy research that conceptualizes reading as a dynamic cognitive process as reflected in the following definition of reading. NAEP 2017 READING FRAMEWORK iii Reading is an active and complex process that involves: • Understanding written text. • Developing and interpreting meaning. • Using meaning as appropriate to type of text, purpose, and situation. This definition applies to the assessment of reading achievement on NAEP and is not intended to be an inclusive definition of reading or reading instruction. TEXT TYPES This framework recognizes that reading behaviors such as recognizing and using features of text, making sense of sentences and paragraphs, and comprehending vocabulary occur regardless of text type. However, other reading behaviors vary with the type of text en- countered by a reader. Thus, the NAEP Reading Framework includes two types of texts on the assessment: literary texts, which include fiction, literary nonfiction, and poetry; and informational texts, which include exposition, argumentation and persuasive text, and procedural text and documents. MEANING VOCABULARY ASSESSMENT The NAEP Reading Framework includes a more systematic approach to vocabulary assessment than the NAEP Reading Framework used from 1992 through 2007. Vocabu- lary assessment will occur in the context of a passage; that is, vocabulary items will function both as a measure of passage comprehension and as a test of readers’ specific knowledge of the word’s meaning as intended by the passage author. A sufficient number of vocabulary items at each grade will provide reliable and valid information about students’ vocabulary knowledge. ITEM DESIGN The framework includes the following cognitive targets, or behaviors and skills, for items from both literary and informational texts: locate/recall, integrate/interpret, and critique/ evaluate. These cognitive targets illustrate the complex nature of the reading process whereas the corresponding behaviors highlight the different behaviors elicited by differ- ent text types. To measure these cognitive skills, students will respond to both selected- response and constructed-response items with varying distributions of question type by grade level. Students in grade 4 will spend approximately half of the assessment time responding to selected-response items and half responding to constructed-response items. Students in grades 8 and 12 will spend a greater amount of time on constructed-response items. Starting with the 2017 assessment, students will engage with both selected- response and constructed-response items in a digital platform. NAEP 2017 READING FRAMEWORK iv 12TH GRADE NAEP In May 2005, the Governing Board adopted a policy regarding NAEP and 12th-grade preparedness. The policy states that NAEP will pursue assessment and reporting on 12th- grade student achievement as it relates to preparedness for postsecondary education and training. This policy resulted from recommendations of the Governing Board’s National Commission on NAEP 12th Grade Assessment and Reporting in March 2004. Subse- quent studies and deliberations by the Governing Board took place during 2004 and 2005. In reading, the Governing Board adopted minor modifications to the 2009 NAEP Read- ing Framework at grade 12 based on a comprehensive analysis of the framework con- ducted by Achieve, Inc. The current version of the reading framework incorporates these modifications at grade 12 to enable NAEP to measure and report on preparedness for postsecondary endeavors. The 2017 NAEP Reading Assessment will use the same framework used since 2009. REPORTING RESULTS Results are reported in two ways: as average scores for groups of students on the NAEP 0–500 scale and as percentages of students who attain each of the three achievement lev- els (Basic, Proficient, and Advanced, according to definitions adopted by the Governing Board). NAEP scores are always reported at the aggregate level; scores are not produced for individual schools or students. REPORTING TREND DATA The Reading Framework for the 2009 National Assessment of Educational Progress replaced the framework used first for the 1992 reading assessment and then for subse- quent reading assessments through 2007. Compared with the previous framework, the 2009 reading framework included more emphasis on literary and informational texts, a redefinition of reading cognitive processes, a new systematic assessment of vocabulary knowledge, and the addition of poetry to grade 4. The 2009 NAEP Reading Report Card included trend data on student reading perfor- mance from 1992 to 2009. Results from special analyses determined the 2009 reading assessment results could be compared with those from earlier assessment years. These special analyses started in 2007 and included in-depth comparisons of the frameworks and the test questions, as well as an examination of how the same students performed on the 2009 assessment and the earlier assessment. A summary of these special analyses and an overview of the differences between the previous framework and the 2009 framework are available on the Web at nces.ed.gov/nationsreportcard/reading/trend_study.asp. The 2017 NAEP Reading Report Card will report trends in student reading performance, including achievement results that extend back to1992. NAEP 2017 READING FRAMEWORK v SAMPLE ITEMS Sample passages and items released to the public may be viewed on the NAEP Questions Tool at nces.ed.gov/nationsreportcard/itmrlsx/. The Questions Tool also includes perfor- mance results, scoring rubrics, and student responses to NAEP reading items. NAEP 2017 READING FRAMEWORK vi