THE IMPACT OF ASSESSMENT ON THE CONSTRUCTIVE ALIGNMENT OF A MODERN VETERINARY CURRICULUM KATE COBB Thesis submitted to the University of Nottingham for the Degree of Doctor of Philosophy July 2015 1 Abstract Constructive alignment (CA) describes an approach to education where teaching, learning and assessment are aligned to allow the learner to achieve the intended learning outcomes (ILOs) of the course. Assessment has a strong influence on learning and therefore the potential to have either a positive or negative impact on CA. The aim of the research in this thesis is to explore the effects of assessment on CA. The context for the research is the final year of study within the School of Veterinary Medicine and Science (SVMS), at the University of Nottingham. Five mixed methods studies were conducted utilising questionnaires, semi-structured interviews and focus groups. In study one the ILOs of the course were defined and subsequently used in study two as a framework for an alumni survey to determine how well prepared graduates feel for clinical practice. Studies three and four investigated the impact of assessments on learning behaviour, namely multiple-choice questions (MCQ), directly observed procedural skills (DOPS) and the script concordance test (SCT). Finally in study five the influence of the transition to practice on learning behaviour during final year was explored. Graduates felt well prepared by the SVMS curriculum for a career in practice with respect to all ILOs. However, assessments were not rated so highly when considered alone. DOPS and to some extent the SCT are considered to be authentic assessment formats and encourage a deep approach to learning. The MCQ in this context results in surface learning strategies being adopted. The imminent transition to practice has a 2 positive effect on learning behaviour, however this conflicts with the preparation required for final year examinations. Elements of the assessment strategy that have a positive and negative impact on CA are identified and discussed. Changes to the curriculum are proposed to enhance CA and ease the transition to practice. 3 For Mum and Dad 4 Acknowledgements I would like to thank the University of Nottingham for funding this PhD and my supervisors Professor Richard Hammond and Professor Sarah Freeman for their advice and support. I am especially grateful to Dr Liz Mossop not only for agreeing to supervise my research but also for her friendship and patience, thank you Liz. I would also like to thank Professor Debbie Jaarsma for her advice in the early stages of the research and Professor George Brown whose help and guidance has been greatly appreciated throughout this project. Thank you to all the practitioners, SVMS staff, students and alumni who volunteered to participate in each of the research studies. Finally I would like to say a massive thanks to my family, Mal, Steph and Jack who have all been supportive and at times extremely tolerant of me during this PhD. 5 “For every evaluative action there is an equal (or greater) (and sometimes opposite) educational reaction” Lambert Schuwirth, the “law” of educational cause and effect 6 Table of Contents Abstract ......................................................................................... 2 Acknowledgements ........................................................................ 5 List of Figures .............................................................................. 13 List of Tables ................................................................................ 15 Abbreviations: .............................................................................. 17 Chapter 1: Introduction ............................................................... 18 1.1: The history of OBE within veterinary curricula ...................................................... 21 1.2: The SVMS curriculum ........................................................................................... 24 1.3: Implications for Research ..................................................................................... 30 Chapter 2: A review of the relevant literature .............................. 31 2.1: Introduction ........................................................................................................ 31 2.2: Outcome-Based Education (OBE) .......................................................................... 31 2.2.1: An introduction to OBE: ......................................................................................... 31 2.2.2: Defining the principles of OBE ............................................................................... 32 2.2.3: OBE within the healthcare professions .................................................................. 34 2.2.4: Criticisms of OBE .................................................................................................... 35 2.2.5: Defence of OBE in the context of a veterinary curriculum .................................... 38 2.2.6: Constructing effective outcomes ........................................................................... 41 2.3: Constructive alignment ........................................................................................ 44 2.3.1: The principles of constructive alignment ............................................................... 44 2.3.2: Examples of CA in curriculum design and development ....................................... 45 7 2.3.3: To what extent does constructive alignment exist within higher education?....... 48 2.4: Outcomes assessment ......................................................................................... 50 2.5: Indirect outcomes assessment ............................................................................. 53 2.6: Direct assessment of student performance ........................................................... 56 2.6.1: Assessment of clinical competence. ...................................................................... 56 2.6.2: The role of formative and summative assessment ................................................ 67 2.6.3: Assessing clinical competence in veterinary students ........................................... 70 2.7: Learning behaviour .............................................................................................. 84 2.8: Workplace-based learning.................................................................................... 93 2.8.1: Experiential learning .............................................................................................. 94 2.8.2 Socio-cultural perspectives ..................................................................................... 95 2.8.3 Adult learning theory .............................................................................................. 96 2.8.4 Benefits of WPBL ..................................................................................................... 97 2.8.5 Challenges of WPBL................................................................................................. 98 2.9: Transition to practice ......................................................................................... 100 2.9.1: What challenges do new graduates face? ........................................................... 101 2.9.2: What makes a successful transition to practice? ................................................. 102 2.10: Summary of Chapter Two ................................................................................. 104 Chapter 3: Research design ....................................................... 106 3.1: Mixed methods research .................................................................................... 106 3.2: Methods of data collection and analysis used within this thesis .......................... 112 3.2.1: Focus Groups ........................................................................................................ 112 3.2.2: Interviews ............................................................................................................. 113 3.2.3: Surveys ................................................................................................................. 115 3.2.4: Thematic analysis ................................................................................................. 118 3.2.5: Sampling ............................................................................................................... 119 8 3.3: Ethical approval ................................................................................................. 122 3.4: Summary of Chapter 3 ....................................................................................... 122 Chapter 4: Indirect Outcomes Assessment ................................ 124 4.1: Introduction ...................................................................................................... 124 4.2: Study 1: Defining the ILOs of the SVMS graduate ................................................ 126 4.2.1: A systematic review of the literature................................................................... 127 4.2.2: The focus group ................................................................................................... 131 4.2.3: The staff survey .................................................................................................... 132 4.2.4: Results of study 1: defining the ILOs of the SVMS graduate. .............................. 140 4.3: Indirect outcomes assessment ........................................................................... 142 4.3.1: The content of the Graduate Survey ................................................................... 142 4.3.2: Graduate Survey distribution ............................................................................... 145 4.3.3: Data analysis ........................................................................................................ 146 4.3.4: Results from the graduate survey ........................................................................ 147 4.3.5: Overview and implications of chapter 4 .............................................................. 164 Chapter 5: The Educational Impact of Assessment on Final Year Students .................................................................................... 168 5.1: Study 3: The educational impact of assessment: a comparison of workplace based learning and multiple choice question formats. ......................................................... 170 5.1.1 Introduction .......................................................................................................... 170 5.1.2: Methods ............................................................................................................... 171 5.1.3: Results .................................................................................................................. 173 5.1.4: Overview and implications of study 3 .................................................................. 189 5.2: Study 4: The students’ perceptions of the Script Concordance Test and its impact on learning behaviour ................................................................................................... 193 5.2.1: Introduction ......................................................................................................... 193 9 5.2.2: Methods ............................................................................................................... 193 5.2.3: Results .................................................................................................................. 199 5.2.4: Overview and implications of study 4 .................................................................. 214 5.3: Discussion of studies in chapter 5 ....................................................................... 218 Chapter 6: What constitutes a successful transition to practice? .................................................................................................. 222 6.1: Introduction. ..................................................................................................... 222 6.2: Methods ............................................................................................................ 224 6.2.1: The student survey............................................................................................... 224 6.2.2: The employer survey ............................................................................................ 226 6.2.3: Data Analysis ........................................................................................................ 226 6.3: Results .............................................................................................................. 227 6.3.1: Participants .......................................................................................................... 227 6.3.2: Comparison of students’ and employers’ perceptions of transition to practice and employability .................................................................................................................. 228 6.3.3: Preparation for the first job in practice ............................................................... 235 6.3.4: The employers’ perception of a UK veterinary degree ........................................ 235 6.3.5: Support provided to new veterinary graduates .................................................. 236 6.3.6: Academic criteria and recruitment ...................................................................... 237 6.3.7: Non-academic criteria and recruitment .............................................................. 239 6.3.8: Attributes assessed at interview .......................................................................... 241 6.3.9: Influence of the type of practice on new graduate recruitment ......................... 243 6.4: Overview and implications of chapter 6 .............................................................. 246 Chapter 7: Discussion ................................................................ 253 7.1: What is the impact of assessment on CA within the SVMS curriculum? ............... 253 7.1.1: What are the intended learning outcomes of the veterinary medicine course? 253 10