d e z ori 45411 h ut A e r u s o cl s Di c bli u P d e z ri o h ut A e r u s National Assessments of Educational Achievement o cl s Di c bli u P VOLUME 2 d Developing Tests and e z ri o h ut A Questionnaires for a e r u s o scl National Assessment Di c bli u P of Educational Achievement d e z ri o h ut A Prue Anderson e r u s George Morgan o cl s Di c bli u P Developing Tests and Questionnaires for a National Assessment of Educational Achievement National Assessments of Educational Achievement VOLUME 2 Developing Tests and Questionnaires for a National Assessment of Educational Achievement Prue Anderson and George Morgan Vincent Greaney and Thomas Kellaghan, Series Editors © 2008 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: [email protected] All rights reserved 1 2 3 4 11 10 09 08 This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The fi ndings, interpretations, and conclu- sions expressed in this volume do not necessarily refl ect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Offi ce of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@world- bank.org. Cover design: Naylor Design, Washington, DC ISBN: 978-0-8213-7497-9 eISBN: 978-0-8213-7498-6 DOI: 10.1596/978-0-8213-7497-9 Library of Congress Cataloging-in-Publication Data Anderson, Prue, 1954– Developing tests and questionnaires for a national assessment of educational achievement / Prue Anderson, George Morgan. p. cm.—(National assessment of educational achievement ; volume 2) Includes bibliographical references and index. ISBN 978-0-8213-7497-9 (alk. paper)—ISBN 978-0-8213-7498-6 1. Educational tests and measurements—United States. 2. Educational evalua- tion—United States. I. Morgan, George, 1945– II. Title. LB3051.A715 2008 371.26'1—dc22 2008002684 CONTENTS PREFACE xi ABOUT THE AUTHORS AND EDITORS xiii ACKNOWLEDGMENTS xv ABBREVIATIONS xvii PART 1 CONSTRUCTING ACHIEVEMENT TESTS CHAPTER 1 INTRODUCTION 3 CHAPTER 2 DEVELOPING AN ASSESSMENT FRAMEWORK 9 The Test Blueprint or Table of Specifi cations 10 Validity 16 Test Language 16 Item Format 17 Student Population to Be Assessed 24 Reporting Results 24 Contexts 25 CHAPTER 3 ITEM WRITING 27 Item Diffi culty 29 Item Bias 30 Stimulus Material 30 Item Format 33 Practice Items 45 v vi | CONTENTS Item Layout and Design 46 The Item-Writing Team 52 Item Panels 55 Other Reviewers 57 Tracking Items 58 CHAPTER 4 PRETESTING ITEMS 61 Designing the Pretest Form 64 Printing and Proofreading the Pretest 68 Implementing the Pretest 70 Scoring the Pretest 71 Reliability 76 CHAPTER 5 SELECTING TEST ITEMS 79 CHAPTER 6 PRODUCING THE FINAL TEST 85 Designing the Final Test 85 Printing and Proofreading 88 CHAPTER 7 HAND-SCORING TEST ITEMS 91 PART 2 CONSTRUCTING QUESTIONNAIRES 97 CHAPTER 8 DESIGNING QUESTIONNAIRES 99 Questionnaire Content 102 Questionnaire Blueprint 105 Questionnaire Items 105 Item Format 108 Language of the Questionnaire 109 Respondents 109 Questionnaire Administration 110 Data Analysis Plan 111 CHAPTER 9 ITEM WRITING FOR QUESTIONNAIRES 113 Questions 114 Statements 114 Response Categories 115 Managing Sensitive Issues 117 Questionnaire Layout 117 Reviewing Questionnaires 119 CHAPTER 10 CODING QUESTIONNAIRE RESPONSES 121 Preparing Questionnaires for Data Entry 123 Coding Missing or Ambiguous Responses 123 CONTENTS | vii CHAPTER 11 MATCHING QUESTIONNAIRES AND TEST DATA 125 Student Questionnaires 125 Parent Questionnaires 126 Teacher and Head-Teacher Questionnaires 126 PART 3 DESIGNING A MANUAL FOR TEST ADMINISTRATION 127 CHAPTER 12 THE TEST ADMINISTRATORS’ MANUAL 129 Contents of the Manual 130 Use of the Manual 131 Features of a Manual 132 How Much Detail Is Necessary? 132 Practice Questions 134 Tryout 134 Review 137 CHAPTER 13 THE TEST ADMINISTRATOR 139 Choice of Test Administrator 139 Following Instructions 140 Quality Assurance 142 Administrator’s Checklist 142 CHAPTER 14 INFORMING SCHOOLS ABOUT THE NATIONAL ASSESSMENT 145 APPENDIX A GLOSSARY 147 APPENDIX B FURTHER READING 153 APPENDIX C EXAMPLES OF TEST AND QUESTIONNAIRE ITEMS AND ADMINISTRATION MANUALS ON CD 157 INDEX 163 BOXES 2.1 Papua New Guinea Mathematics Curriculum 11 2.2 New Zealand English Curriculum 11 2.3 Examples of Multiple-Choice Items 18 2.4 Example of a Closed Constructed-Response Item 19 2.5 Examples of Open-Ended Short-Response Items 19 viii | CONTENTS 2.6 Example of an Essay Prompt 20 3.1 Example of Irrelevant Stimulus Material 32 3.2 Example of an Item with Inaccurate or Misleading Information 33 3.3 Example of a Multiple-Choice Item 34 3.4 Punctuation in Complete Sentences 35 3.5 Punctuation in a List 35 3.6 Minimizing Reading 36 3.7 Item with a Negative Stem 36 3.8 Poorly Paired Distractors 37 3.9 Dealing with Pairs in Distractors 37 3.10 Confusing Open-Ended Item with Unclear Directions 40 3.11 Good Example of a Closed Constructed-Response Item 41 3.12 Item with Partial Credit 42 3.13 Example of an Open-Ended Response Item with Scoring Guide 43 3.14 Example of a Closed-Constructed Item with Scoring Guide 44 3.15 Using Pictures to Reduce Words 48 3.16 Keeping Images Simple 49 3.17 Labeling Graphs Clearly 49 3.18 Labeling Maps Clearly 50 3.19 Leaving Space in the Stimulus Material 51 3.20 Example of Style Sheet for Item Writers 54 4.1 Example of an Item in Multiple-Choice and Open-Ended Format 71 4.2 Example of a Data Entry Sheet for a Pretest 73 6.1 Example of a Test Cover Page 86 8.1 Attitudes and Values Questionnaire Blueprint 106 9.1 Poor Alignment of Boxes and Response Categories 118 9.2 Better Alignment of Boxes and Response Categories 118 10.1 Grayscale Coding Example 122 10.2 Example Treating Items as Separate Categories for Data Entry 122 12.1 Administration Manual Instructions 133 12.2 Information for Teachers and Principals 134 12.3 Administration of Practice Items 135 13.1 Administration Checklist: An Example from the Philippines 143 CONTENTS | ix FIGURES 1.1 National Assessment Organizational Chart 4 1.2 Overview of National Assessment Activities 5 4.1 Example of Circular Linking of Items 65 4.2 Model for Vertically Linking Items 66 C.1 Guide to CD Materials on Tests, Questionnaires, and Manuals 158 TABLES 1.1 National Assessment Stages in Test Development and Questionnaire Design 6 2.1 Blueprint for a Middle Primary Mathematics Test 13 2.2 TIMSS Mathematics Blueprint, Grades 3 and 4 14 2.3 Papua New Guinea Mathematics Content Blueprint 15 2.4 Advantages and Limitations of Item Formats 23 2.5 Papua New Guinea Mathematics Test-Item Formats 24 4.1 Link Items in Two Reading Units 67 4.2 Part of a Spreadsheet for Tracking Items across Forms 68 5.1 Example of Output from Analysis of a Multiple-Choice Item 80 5.2 Example of Output from Analysis of an Open-Ended Partial-Credit Item 81 8.1 Components of Questionnaire Development 101 8.2 Functions of Reading in an International Study: Weights Used to Create Two New Variables, “Reading for a Utilitarian Purpose” and “Reading for Enjoyment” 107