David Coniam · Peter Falvey Validating Technological Innovation The Introduction and Implementation of Onscreen Marking in Hong Kong Validating Technological Innovation David Coniam • Peter Falvey Validating Technological Innovation The Introduction and Implementation of Onscreen Marking in Hong Kong David Coniam Peter Falvey Department of Curriculum and Instruction Department of Curriculum and Instruction The Education University of Hong Kong The Education University of Hong Kong Tai Po , Hong Kong Tai Po , Hong Kong ISBN 978-981-10-0432-2 ISBN 978-981-10-0434-6 (eBook) DOI 10.1007/978-981-10-0434-6 Library of Congress Control Number: 2016936563 © Springer Science+Business Media Singapore 2016 T his work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifi cally the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfi lms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. T he use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specifi c statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. T he publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. Printed on acid-free paper This Springer imprint is published by Springer Nature The registered company is Springer Science+Business Media Singapore Pte Ltd. Foreword Validating Technological Innovation : T he Introduction and Implementation of Onscreen Marking in Hong Kong is not only a coherent account of a series of vali- dation studies linked to the implementation of onscreen marking (OSM) in Hong Kong, it is a book representing a 35-year research collaboration of the two authors who have in-depth knowledge about the context of the research. Mostly impor- tantly, it is a book about the future of the fi eld. It is challenging to predict the future in the current world of ours with many uncertainties, but I have no doubt that valida- tion research of onscreen marking will increase, and this book will be read by many researchers for years to come. As the new millennium was getting underway, in January 2000, Lyle Bachman’s article, entitled ‘Modern language testing at the turn of the century: assuring that what we count counts’, appeared in L anguage Testing (Volume 17). The article is a 40-plus-page comprehensive review of language testing in the 1980s and 1990s, drawing on more than 200 published works. The personal, retrospective paper focuses on a wide range of areas that developed mainly in the 1990s, including: ‘(a) research methodology; (b) practical advances; (c) factors that affect performance on language tests; (d) authentic, or performance, assessments; and (e) concerns with the ethics of language testing and professionalising the fi eld’ (Bachman 2000, p.4). U nder (b), Bachman lists computer-based assessment as one of practical advances in the fi eld of language testing pointing to the increasing use of computer technol- ogy in the development and delivery of language tests. He stated that ‘On the practi- cal side, advances in the technology of test design and development, along with the availability and use of ever more sophisticated computer- and web-based applica- tions for test administration, scoring and analysis, have resulted in a greater range of test formats and assessment procedures than has ever been available’. In this, Bachman cautions the fi eld that ‘the challenge in applying such technologies to language assessment will be to recognise not only the potential benefi ts but also the limitations of these technologies, and not let the psychometric and technological tails wag the applied linguistics dog’ (p. 9). In discussing the future and what lies ahead, Bachman states that ‘the abstract technical debate about dimensionality and the appropriateness of different IRT models has, I believe, been replaced with a v vi Foreword much more pragmatic focus on practical applications, particularly with respect to performance assessments that involve raters and computer-based tests’ (p. 22). Bachman did not specifi cally discuss onscreen marking, but the above statements clearly direct us to the challenges we are facing in the use of computer-assisted assessment over traditional paper methods. In this sense, this book presenting a coherent series of research studies into onscreen assessment of continuous writing and its impact upon markers’ judgements is, therefore, both timely and important. A s the two authors point out, onscreen marking (the marking of examination scripts on screen instead of on paper) is being used much more frequently as tech- nology develops. Over the past 10 years, there has been a worldwide move towards OSM. Indeed, OSM can offer many benefi ts over traditional paper methods. However, in transferring from one medium to another, it is crucial to understand the extent to which the new medium may alter the nature of traditional assessment prac- tice and affect marking reliability through empirical evidence, evidence that is pro- vided by this book. T his book is unique in a number of ways. First, it charts the development of the validity of OSM in the implementation of OSM across the board in Hong Kong, i.e. in every subject in the public school leaving examination system. The book further provides a series of validation studies over time, which began, in 2009, with the publication of Coniam (2009a), the fi rst description of what became, by 2014/2015, a series of validation studies of OSM in the school public examinations system in Hong Kong, operated by the Hong Kong Examinations and Assessment Authority (HKEAA). In total, nine validation studies were carried out and published by David Coniam, sometimes with co-writers (Alison Sau-chu Yeung [Chap. 7 ], Zi Yan [Chaps. 10 , 12 and 1 3 ] and Peter Falvey, the co-author of this book). Second, the book provides a coherent picture, from a disparate series of valida- tion research studies, backed by useful background information on Hong Kong’s education and assessment history, the HKEAA and its work (Chap. 2 ), a pictorial representation of the OSM process taken from screenshots (Chap. 3 ) and an intro- duction to the methodological approaches and tools used in the quantitative and qualitative studies on the OSM process. Therefore, this mixture of background information and research input makes this book extremely useful for readers and other academics within other similar educational contexts. In this sense, the book also provides a useful template for authorities/jurisdictions who want to carry out thorough validation studies in preparation for the wholesale implementation of OSM. T hird, the book is a labour of love from two authors who have had a long, pro- ductive association as writers, researchers, teachers, publishers and colleagues stretching over 35 years within the Hong Kong language education context and internationally. Much of this long association started in Hong Kong where they both have an in-depth knowledge of the education and examination system and have established good working relations with the HKEAA and the Education Bureau of the Government of the Hong Kong Special Administrative Region. Such a collab- orative connection between test users and test developers and educators and minis- tries of education is rare in other education contexts. Further, much of the long Foreword vii association between the authors has a background in assessment (many books, chapters in books and article publications). David and Peter have spent 20 years on language benchmarking (with a lot of publications) and now OSM. They both have a background with examination authorities (David with the HKEAA where he was an HKEAA offi cer and long-time examination developer and Peter with Cambridge Assessment where he was an advisory consultant on secondment from his specialist work with the British Council). Such a unique relationship in assessment and in- depth understanding of the Hong Kong context is refl ected in the coherence of this programme of research. Validating Technological Innovation: The Introduction and Implementation of Onscreen Marking in Hong Kong not only brings the most recent work on OSM situated in a unique context of education but also makes many new and signifi cant contributions to the fi eld, which I believe have the potential to help test developers and test users all over the world. Faculty of Education, Queen’s University Liying Cheng Kingston, ON, Canada Reference B achman, L.F. (2000). Modern language testing at the turn of the century: Assuring that what we count counts. Language Testing , 17 (1), 1–42. Acknowledgements W e are grateful to a number of people whose efforts, scholarship and expertise made this book possible. C hristina Lee of the Hong Kong Examinations and Assessment Authority (HKEAA) provided her expertise and experience to create and write Chaps. 2 and 3 . Chapter 2 ‘The Role of the Hong Kong Examinations and Assessment Authority’ introduces the reader to the background to the development and implementation of OSM in Hong Kong and describes the processes that the innovation went through. In Chap. 3 , ‘Onscreen Marking System’, Christina, through the use of screen-grabs demonstrates for the reader what the markers physically see on the screen as they process and mark scripts and record their marks. We are also grateful to Alison Sau-chu Yeung (Chap. 7 ) and Zi Yan (Chaps. 10 , 1 2 , and 1 3 ) for their contributions to papers that they co-wrote with the authors and for their consent to publish their papers. A s we acknowledge in the footnote to Chap. 4, we are grateful to Julie Mackay of Cambridge Assessment and Sarah Phillips of RM Results for their efforts in securing us details of Cambridge examinations marked by means of OSM. In terms of the provision of relevant research information, we are grateful to Ardeshir Geranpayeh of Cambridge Assessment for his links to relevant Cambridge-based research. Finally, we would like to thank Professor Liying Cheng of Queens University, Kingston, Ontario, for writing a Foreword. She is an extremely busy academic, working in the fi eld of examinations and assessment, and we are grateful to her for her efforts at a particularly busy time in her work cycle. ix