Virtual Environment for Radiotherapy Training (VERT) Final Project Report A Project funded by the Department of Health for England and the Cancer Action Team; and Managed by the Society and College of Radiographers Project Leads Rob Appleyard - Research and Evaluation Co-ordinator and Louise Coleman - Implementation and Education Co-ordinato r Table of Contents 1.0 Purpose of the Report ...................................................................................................... 6 2.0 Introduction .................................................................................................................... 6 2.1 Background ......................................................................................................................... 6 2.2 Project Aim and Outcomes ................................................................................................. 7 2.3 The VERT Technology .......................................................................................................... 7 2.4 Project Design ..................................................................................................................... 8 2.4.1 Implementation and Education .............................................................................. 8 2.4.2 Evaluation and Research ......................................................................................... 8 3.0 Project Management ....................................................................................................... 9 3.1 Steering Group .................................................................................................................... 9 3.2 National User Group ........................................................................................................... 9 3.3 Regional User Groups ........................................................................................................10 4.0 Installation of VERT Systems .......................................................................................... 11 4.1 Installation of VERT in England .........................................................................................11 4.2 VERT in Northern Ireland, Scotland and Wales.................................................................12 4.2.1 Northern Ireland ...................................................................................................12 4.2.2 Scotland ................................................................................................................12 4.2.3 Wales ....................................................................................................................12 5.0 Implementation and Education ...................................................................................... 12 5.1 Approach to Implementation and Education ...................................................................12 5.2 Procurement and Installation Issues .................................................................................13 5.3 Training .............................................................................................................................14 5.4 Outcomes from Implementation of User Groups .............................................................14 5.4.1 National User Group and Grouping of Departments ............................................14 5.4.2 Regional User Groups ...........................................................................................14 5.4.3 Local User Groups .................................................................................................15 5.4.4 Department and Educational Institution VERT Leads ...........................................16 6.0 Evaluation ..................................................................................................................... 16 6.1 Impact on the Student ...................................................................................................16 6.1.1 First Impressions ............................................................................................16 6.1.2 Skills and Confidence .....................................................................................18 6.1.3 Staff and Student Experience ........................................................................20 6.1.4 VERT and Skills Development in Skin Apposition Techniques .......................23 6.2 The Impact of VERT on Recruitment, Retention and Attrition ......................................26 6.2.1 Recruitment ...................................................................................................26 6.2.2 Attrition .........................................................................................................26 6.2.3 Retention .......................................................................................................27 6.3 Overall Evaluation of the Implementation and Impact of VERT ....................................27 6.3.1 Use of VERT ....................................................................................................27 6.3.2 Enhancement of Knowledge and Understanding ..........................................28 6.3.3 Impact on Clinical Competence .....................................................................29 6.3.4 VERT Influence on Enjoyment of the Course .................................................29 6.3.5 Curriculum Enhancement ..............................................................................29 6.4 Barriers to Effective Use ................................................................................................30 6.4.1 Adverse Effects ..............................................................................................30 6.4.2 Staffing and Time ...........................................................................................30 6.4.3 Management of VERT Resources...................................................................31 6.4.4 Location of VERT ............................................................................................32 6.4.5 Access to Treatment Planning Data ...............................................................32 7.0 Summary of Recommendations ..................................................................................... 33 8.0 Conclusions ................................................................................................................... 35 10.0 References ................................................................................................................... 37 Appendices Appendix 1: Questionnaire Distributed to Students during the First Semester Appendix 2: Pre- and Post-experience Questionnaire Appendix 3: Staff On-line Questionnaire Appendix 4: Skin Apposition Techniques: Post-experience Questionnaire Appendix 5: Skin Apposition Techniques: Calculating Performance Score Appendix 6: Results of Skin Apposition Techniques Study Appendix 7: Final Evaluation Survey Appendix 8: Results of Final Evaluation Survey and Reports List of Tables Table 1: VERT Installations in England................................................................................................... 11 Table 2: Early VERT experience - summary of student likes and dislikes .............................................. 17 List of Figures Figure 1: Confidence before VERT experience 19 Figure 2: Confidence after VERT experience ......................................................................................... 19 Figure 3: Staff perception of students’ skills 19 Figure 4: Staff perception of students’ confidence .............................................................................. 19 1.0 Purpose of the Report The purpose of this Report is to inform the key stakeholders, radiographers and the wider radiotherapy community about the progress and outcome of evaluation of the Virtual Environment for Radiotherapy Training (VERT) Project. The Report highlights good practice and identifies some of the issues that hindered effective use of the technology during the course of the Project. The Report is not intended to be a handbook to guide users of VERT. However, it does contain examples of effective use of VERT by both clinical departments and educational institutions around the country and will be of use to those considering purchasing a VERT system or as a reference source for those who have VERT installed currently. 2.0 Introduction 2.1 Background 1 2.1.1 The National Radiotherapy Advisory Group (NRAG) Report to the UK Government in 2007 identified a potential crisis in England in relation to radiotherapy education and training and an urgent need to reduce the attrition rate of student therapy radiographers. The Report included a recommendation to introduce Hybrid Virtual Environment skills training facilities across the 10 education providers and 51 associated radiotherapy and oncology sites providing clinical training and experience. The aim would be to improve retention by enhancing the student learning experience through provision of the opportunity to develop knowledge and skills in a ‘safe’ environment while limiting the impact on already stretched clinical resources. 2.1.2 A suitable virtual reality platform was readily available – the Virtual Environment for Radiotherapy Training (VERT) system – utilising immersive visualisation technology and software 2 developed by Vertual Ltd . The VERT system provides a life-sized virtual radiotherapy treatment room and allows the user to interact with the virtual room, control the equipment and set up radiation treatments as if in the real world. 2.1.3 In response to the NRAG recommendation, the Department of Health and Cancer Action Team made £5 million available to fund: purchase of the VERT software and required hardware; refurbishment costs; and an 18-month project, led by the Society and College of Radiographers, to manage the implementation of the VERT technology and begin evaluating its impact. 2.1.4 Educational institutions providing pre-registration radiotherapy education and training in England and radiotherapy and oncology departments providing the associated student placements were invited to submit expressions of interest in participating in this supported VERT Project which ran from April 2008 to October 2009. Page 6 of 58 2.2 Project Aim and Outcomes The aim of the 18-month VERT Project was to assess the potential use of the VERT technology – the projectors, screens, immersive and seminar auditoria and the software. It was designed to achieve the following specific outcomes through an integrated approach to the introduction of VERT into educational institutions and clinical radiotherapy and oncology departments: To assess the potential impact of VERT on student recruitment and retention, especially in year one; To investigate how students learn in virtual environments; To assess the impact of VERT on student confidence; To assess the impact of VERT on student enjoyment of their courses; To assess the extent to which VERT enhances students’ knowledge and understanding of relevant radiotherapy concepts; To determine the extent to which students’ psychomotor and practical skills can be developed using VERT; To share the learning outcomes associated with the use of VERT across all the universities and departments; and To appraise the long term potential of VERT, making recommendations regarding its impact on future curriculum design and teaching, learning and assessment strategies. 2.3 The VERT Technology 2, 3, 4 2.3.1 The VERT system and its functionality have been fully described elsewhere . Basically, however, high resolution stereoscopic projection on to a large screen produces a realistic, virtual environment of a radiotherapy treatment room. It provides a life-size model of a linear accelerator with full functionality except for the production of radiation. Users wear either liquid crystal display (LCD) shutter glasses or light polarising glasses (depending on the type of system being used) so that they are immersed in a three dimensional (3-D) image projected in the space around them. The environment is a hybrid one in that a virtual linear accelerator is controlled using an actual hand- pendant. 2.3.2 It is possible to import images and radiotherapy treatment plans in DICOM format, thereby allowing a vast range of simple and complex treatment plans as well as the related anatomical data to be visualised in 3-D. 2.3.3 Different systems are provided in educational institutions and clinical areas. Educational institutions use a system called ‘Immersive VERT’. These systems are housed in a purpose-built, bespoke auditorium and employ rear projection using active stereoscopy requiring the user to wear LCD shutter glasses. A tracking system is provided that enables the image to be projected according to the user’s position relative to the projection screen thus further enhancing the degree of ‘immersion’. Radiotherapy and oncology departments use a system called ‘Seminar VERT’. These systems can be situated in seminar or meeting rooms and require no significant refurbishment to support installation. Seminar VERT features front projection using passive stereoscopy requiring the use of polarising glasses. User tracking is not provided. At the time of the Project, the cost of Page 7 of 58 Seminar VERT was approximately one quarter that of Immersive VERT, excluding room modification costs. 2.3.4 The large screen stereoscopic projection, faithful representation of the treatment room and linear accelerators, use of an actual hand pendant and, in educational institutions, use of a tracking system, all contribute to a high degree of physical and psychological ‘presence’ – the phenomenon whereby users are convinced they are part of a ‘real’ environment. 2.3.5 The VERT technology claims to offer a number of potential advantages: A cost effective alternative to training in clinical environments; Unlimited practice opportunities without risking harm to patient or equipment; Radiotherapy treatment rooms become more efficient as training demands are reduced; A realistic insight into the experience of using the equipment, but without the stress of being in a clinical setting; Enhance the understanding of those radiotherapy concepts that are often difficult to teach in a classroom and/or placement setting; Student attrition is reduced as the learning experience is enhanced. 2.4 Project Design The Project had two distinct elements, each led by a Project Co-ordinator. 2.4.1 Implementation and Education The implementation and education phase included: planning for installation of Immersive VERT systems within educational institutions and the installation of Seminar VERT systems in radiotherapy and oncology departments; training in use of the systems; incorporation of VERT experience within academic and clinical components of pre- registration radiotherapy programmes; and gathering information for dissemination and continued development based on user experience. 2.4.2 Evaluation and Research The impact of VERT was assessed from the perspective of students, academic staff and clinical radiographers to address the Project outcomes. Review of associated literature and informal data gathered through communication during the Project informed and enhanced the overall evaluation strategy which comprised both quantitative and qualitative elements including: questionnaires and tests to assess the impact of VERT on the development of student skills, confidence and performance; collection and interpretation of data in relation to the impact of VERT on recruitment and retention; reports compiled by educational institutions and clinical departments; and a final evaluation survey distributed at the end of the Project. Page 8 of 58 3.0 Project Management 3.1 Steering Group The Project was overseen by a Steering Group which determined the direction of the Project and ensured objectives were met. 3.1.1 The role of the group included: Overseeing the establishment of the Project User Group; Receiving and approving the interim and end of Project reports; Commenting on the evaluation and research elements of the Project; Ensuring that the Project considered and made recommendations on the wider use of the tool across other professions; Receiving regular progress reports from the Project Co-ordinators; Overseeing the financial probity of the Project. 3.1.2 Membership of the group consisted of the two VERT Co-ordinators and representatives from: The College of Radiographers; The Department of Health; The Cancer Action Team; The Royal College of Radiologists; The Institute of Physics and Engineering in Medicine; Higher Education Institutions; and Radiotherapy and Oncology departments. The Steering Group met 4 times during the course of the Project. 3.2 National User Group A National User Group, reporting directly to the Steering Group, was established early in the Project to provide a forum for discussing the VERT Project, its evaluation and the technology involved. 3.2.1 The role of the User Group included: facilitating a wide-reaching communication network regarding the use of the VERT technology; ensuring that the Project met the desired outcomes of the stakeholders; generating, sharing and disseminating ideas and best practice regarding the use of the VERT technology; providing the foundation for continued development after the initial supported 18 months; feeding information back to the VERT Co-ordinators as required; and reporting on the discussions to the Project Steering Group. 3.2.2 The VERT Project funded the technology for institutions in England only. However, it was considered that participation in the National User Group by colleagues in Scotland, Wales and Page 9 of 58 Northern Ireland would be beneficial and, consequently, a representative from each of these countries was invited to attend. 3.2.3 The National User Group consisted of: The Society and College of Radiographers; VERT Co-ordinator – Implementation and Education; VERT Co-ordinator - Evaluation and Research; One representative from each of the 10 English educational institutions; One representative from each group of radiotherapy and oncology departments (see 3.2.4 below); One representative from either Glasgow Caledonian University or Queen Margaret University, Scotland; One representative from Cardiff University, Wales; One representative from the University of Ulster, Northern Ireland; and Vertual Ltd. There were 4 National User Group Meetings during the course of the project. 3.2.4 Representatives from educational institutions and clinical departments were expected to cascade information about the VERT technology and generate interest in its use by both students and staff. However, because of the number of participating radiotherapy and oncology departments in England, it was impractical to include a representative from each department on the User Group. A ‘Group Representative’ system was established, therefore. Departments were allocated to one of 10 groups, based on the educational institution from which they received their main cohort of students. This was considered to be the most appropriate arrangement for the following reasons: Geographical location – the majority of the radiotherapy and oncology departments in each group were within a reasonable travelling distance from each other and the educational institution, making it easier to set up Regional User Groups; Most had already established links with each other and the educational institution, although not necessarily with the representatives who attended the National VERT User Group; and Grouping the departments with their associated educational institution was one way of encouraging communication and partnership working between them – essential for meeting the Project objective of sharing learning outcomes associated with the use of VERT across universities and departments. 3.3 Regional User Groups 3.3.1 One element of the National User Group departmental Group Representatives’ role was to report back to their regional centres. Some educational institutions and associated departments took the initiative and set up their own Regional User Groups at an early stage to facilitate information exchange and, in particular, to discuss the ways in which VERT could be used with both students and qualified staff. These groups consisted of the VERT Leads from the educational institution and each associated clinical placement site. Page 10 of 58