Educational Assessment in a Time of Reform Educational Assessment in a Time of Reform provides background information on large-scale examination systems more generally and the South African exam- ination specifically. It traces the reforms in the education system of South Africa since 1994 and provides a description of the advances in modern test theory that could be considered for future standard setting endeavours. At the heart of the book is the debate on whether the current standard of edu- cation in Africa is good enough. If not, then how can it be improved? The aim of this book is to provide a point of departure for discussions on standard-setting, quality assurance, equating of examinations and assessment approaches. From this point of departure recommendations for practices in general and the exit- level (Grade 12) examination results in particular can be made. This book is ideal reading for principals, teachers, academics and researchers in the fields of educational assessment, measurement and evaluation. Coert Loock is a professor at Department of Education Leadership and Manage- ment, University of Johannesburg, South Africa. Vanessa Scherman is a professor at Department of Psychology of Education, University of South Africa, South Africa. Perspectives on Education in Africa Series Editor – Kerry J. Kennedy Books in the series include: Educational Assessment in a Time of Reform Standards and Standard Setting for Excellence in Education Coert Loock and Vanessa Scherman For more information about the series, please visit www.routledge.com/ Perspectives-on-Education-in-Africa/book-series/EDUAFRICA Educational Assessment in a Time of Reform Standards and Standard Setting for Excellence in Education Coert Loock and Vanessa Scherman First published 2020 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN and by Routledge 52 Vanderbilt Avenue, New York, NY 10017 Routledge is an imprint of the Taylor & Francis Group, an informa business © 2020 Coert Loock and Vanessa Scherman The right of Coert Loock and Vanessa Scherman to be identified as authors of this work has been asserted by them in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data Names: Loock, C. F. (Coert Frederik), author. | Scherman, Vanessa, author. Title: Educational assessment in a time of reform : standards and standard setting for excellence in education / Coert Loock and Vanessa Scherman. Description: Abingdon, Oxon ; New York, NY : Routledge, 2020. | Series: Perspectives on education in Africa | Includes bibliographical references and index. Identifiers: LCCN 2019034552 (print) | LCCN 2019034553 (ebook) | ISBN 9781138394223 (hardback) | ISBN 9780429401329 (ebook) Subjects: LCSH: Educational tests and measurements--South Africa. | Educational change--South Africa. | Education--Aims and objectives--South Africa. Classification: LCC LB3058.S6 L66 2020 (print) | LCC LB3058.S6 (ebook) | DDC 371.26--dc23 LC record available at https://lccn.loc.gov/2019034552 LC ebook record available at https://lccn.loc.gov/2019034553 ISBN: 978-1-138-39422-3 (hbk) ISBN: 978-0-429-40132-9 (ebk) Typeset in Galliard by Cenveo® Publisher Services Contents List of figures viii List of tables ix Series Editors’ introduction – Kerry J. Kennedy, Joseph Divala, Juliet Perumal and Elizabeth Henning x List of abbreviations xii 1 Background and introduction 1 Historical context 1 Aim and objectives of the book 3 Systems theory as a conceptual framework 4 References 5 2 Background to large-scale examination systems 7 Introduction 7 Designing large-scale examination systems 8 Examples of large-scale examination systems 12 Conclusion 20 References 20 3 Perspectives on education assessment reform 23 Standards-based education reform 23 Education assessment reform versus the complexity of assessment standards and academic performance 26 Conclusion 35 References 35 4 Equating examinations and assessment standards in the South African context 37 Introduction 37 Equating high-stakes examinations and testing 39 The history of equating Grade 12 examination marks in South Africa 40 vi Contents Norm-referenced assessment 43 Transition from the Senior Certificate (SC) to the new National Curriculum Statement (NCS) 52 Research related to the comparability of curriculum and assessment demands of the NCS 56 Standardising the new NCS results 57 References 59 5 The management and quality assurance of school-based assessment in South Africa 61 Introduction 61 Social moderation 63 The low reliability of SBA 64 Statistical moderation of SBA marks 66 Statistical moderation 67 Calculation of the subject promotion mark 68 Conclusion 71 References 71 6 Using examination and assessment scores: Different ways of reporting performance 73 Introduction 73 Different ways of reporting performance 73 Reporting performance is directly related to the type and purpose of the test 74 Reporting standards on tests 75 Measuring continuous SBA 76 Comparing results over time – how to measure improvement 77 Strategies for interpreting data 78 Working with examinations and assessment data 80 Conclusion 83 References 83 7 Rethinking standard setting: Drawing from advances in modern test theory 84 Introduction 84 Understanding classical test theory 85 Introduction to IRT 86 The one-parameter model 89 The two-parameter model 90 The three-parameter model 90 Introduction to Rasch-measurement 90 Standard setting methods: Traditional methods and methods drawing on modern test theory 91 Contents vii Linking and equating in modern test theory: Possibilities for linking examinations across years 98 Conclusion 101 References 101 8 Item banking and computer adaptive testing 106 Introduction 106 Computer adaptive testing 106 System design and operations 107 Item pool development and testing 108 Item response theory and calibration 108 Test equating 108 Reliability and measurement precision 108 Validity 108 Advantages and disadvantages of computer adaptive testing 109 Definition and development of item banks 110 Procedure for preparing an item bank 110 Advantages and disadvantages of item banks 111 Validity and reliability considerations 112 Conclusion 115 References 115 9 Developing a computerised system for large-scale computer adaptive testing 117 Introduction 117 Performance benchmarks 117 Design requirements and electronic platform 118 Design logic of a cat-system for the NCS (Grade 12) 118 Conclusion 125 References 125 10 Educational assessment in a time of reform: Where to from here? 126 Introduction 126 Thinking about examination systems within a 21st century skills framework 127 Making use of modern psychometric theory 129 Thinking about moving towards incorporating mixed methods approaches 130 Concluding remarks 131 References 131 Index 134 Figures 2.1 The characteristics of an examination system 8 2.2 The examination process 9 2.3 The standard setting process 12 2.4 Structure of the education system in Kenya 13 2.5 The structure of the curriculum framework 14 2.6 The structure of the Ghanaian education system 16 2.7 The South African education system 18 4.1 Example of a standardisation graph for geography HG (maximum 400 – NATED 550, prior to 2008) 49 6.1 Cohort-to-cohort comparison 78 7.1 Summary illustration of test theories 85 7.2 Summary of IRT statistical models 89 7.3 Summary of the different methods and approaches in standard setting methods 92 7.4 Summary of standard setting procedures using IRT and Rasch measurement 96 8.1 Summary of the process of developing a computer adaptive testing (CAT) system 107 8.2 Criteria for evaluating quality of measurement 113 Tables 3.1 Percentage of progressed learners and the impact on the Grade 12 failure rate in 2015 31 4.1 Norms calculated in 10-symbol intervals (Loock & Grobler, 2004) 47 4.2 Pairs analysis for mathematics 51 4.3 Example: calculation of a norm for geography HG 2001 53 4.4 Mark adjustment table as calculated for Geography HG 2001 54 4.5 Percentage distribution as reflected in Figure 4.1 55 4.6 Determining A grade achievers (physical science) 57 4.7 Determining average achievers (physical science) 57 4.8 Statistical model for the conversion of norms 58 6.1 Test-type comparison (adapted from Boudette et al., 2008) 75 7.1 Differences between classical test theory and item response theory (Barnard, 2009) 88 9.1 Single group types 121 9.2 Group types to categorise groups 121 9.3 Example item A 121 9.4 Example item B 121 9.5 Aspects contacted in questions 122 9.6 Example of an object table 123 9.7 Additional information in the playout screen 124