PRACTITIONER RESEARCH IN HIGHER EDUCATION 4 (1) Using feedback to promote Practitioner Research in Higher Education learning: student and tutor Copyright © 2010 perspectives University of Cumbria Vol 4 (1): page 30-40 Dr Jaki Lilly, Uwe Matthias Richter, Dr Berenice Rivera-Macias Anglia Ruskin University [email protected] Abstract This paper summarises a study of students’ and staff perceptions and experiences of assessment feedback practice across a post-1992 university. Phases 1 and 2 of the project gathered students’ and academic colleagues’ views on assessment feedback practice. Focus groups were then carried out with students and one discussion was video recorded for subsequent use in workshops with faculty colleagues. Students’ and staff’s thoughts on issues militating against good assessment feedback practice were gathered, commented on, analysed, and reported back to the faculties. The student focus groups provided interesting insights as to how students perceive and receive feedback which were classifi ed as being related to content, clarity and style. It was established that, with very few exceptions, issues and good practice in assessment feedback can be generalised across disciplines and, in the main, staff and students share their perceptions of what constitutes good assessment feedback. Keywords Formative feedback; summative feedback; assessment learning process. Introduction This paper summarises a study of students’ and staff perceptions and experiences of assessment feedback practice across a post-1992 University. The Assessment Feedback Project (Lilly et al., 2007; Lilly, et al., 2008) is part of the learning and teaching unit’s programme of developing learning, teaching and assessment practice in the university. It addresses the need for an informed, consistent and appropriate approach to giving students good feedback for learning (Rust, 2007:230). While the focus of this paper is on the limitations of summative assessment feedback, the reach of the project embraces the practice of feedback to both summative and formative forms of assessment. Feedback on summative assessment is defi ned as the marking of scripts and the comments and responses from staff to students’ assignments (Carless, 2006:220). In theory, it is argued that summative feedback should serve a formative function by: • comparing the students’ performance with established assessment criteria (Black and Wiliam, 1998:53–54); and • effectively communicating the steps the student may take to reduce his/her knowledge/ performance gap (Black and Wiliam, 1998:53; Hounsell, 2008:5; Sadler, 1998:79–83). 30 LILLY et al: USING FEEDBACK TO PROMOTE LEARNING: STUDENT AND TUTOR PERSPECTIVES However, as Boud notes: ‘Most comments on student work, even if students read them, occur at times that are the least propitious in terms of infl uencing subsequent student learning – such as at the end of a unit of study when they are moving on to do something different.’ (Boud, 2007:18) The results from the 2008 National Student Survey (NSS) report highlight two key issues in assessment feedback. According to the fi ndings by the Higher Education Funding Council for England (HEFCE) and NSS (2008), the timely supply of feedback to students remains to be improved. Secondly, descriptions of where and how students can develop their learning and academic skills need to be enhanced (HEFCE and NSS, 2008). Although these two concerns might characterise all modes of assessment and feedback used across the sector, they may refer to a great extent to the most common practice: feedback to summative assessment. Students need much clearer and specifi c feedback on the things they need to improve, since feedback on summative assessment aims at assisting students on their learning from the assessment (Irons, 2008:23; our emphasis). Written feedback to summative assessment does not always make enough of a contribution to students’ learning. Research shows that reasons for this include: • students’ lack of understanding about the role good feedback plays in their learning; • contradictions arising if students do not fi nd that feedback is valuable to their current and subsequent learning; and • the subsequent lack of interest in collecting their marked assignments and the feedback provided by their tutors (Carless, 2006:220; Chanock, 2000:95–98; Hounsell, 2007:102–103; Irons, 2008:23–24). In Phase 1 of the project, students’ experiences and expectations of written assessment feedback and staff opinions as to how they could improve their written assessment feedback practice were gathered. During Phase 2, these fi ndings were presented to academic colleagues and their responses to the students’ views were collated. Their thoughts on issues raised in these fi ndings were discussed, and good practice in addressing the issues was shared and recorded. Phase 2 widened the perspective from written assessment feedback to all formative and summative feedback. Phase 1: Methodology Participants and procedures During Phase 1, it was decided to review examples of good written feedback from the samples of assignments normally kept for Quality Assurance Agency for Higher Education (QAA) purposes, academic year 2005/06. Through a letter to faculties, heads of departments and programme/pathway leaders were asked to provide: • samples of good feedback practice; • copies of two samples of the associated student submitted assessment per department; • assessment information (e.g. from module guides); • assessment tasks; and • marking schemes. In the absence of a generic model of good practice, ‘good’ feedback was described as that which had been noted as such in external examiner and annual monitoring reports. 31 PRACTITIONER RESEARCH IN HIGHER EDUCATION 4 (1) The samples were reviewed to establish commonalities and differences in approach and form. The Students’ Union (SU) worked closely with us and invited student representatives from all faculties to focus groups. The focus groups were designed to capture the participants’ perceptions of criteria constituting good (written) feedback and then to review the anonymised samples against the identifi ed criteria. A total of 65 samples were received from the university’s fi ve faculties. The distribution was as follows: • Faculty A 21 • Faculty B 19 • Faculty C 4 • Faculty D 8 • Faculty E 13 While every attempt had been made to ensure that only samples of ‘good’ feedback were submitted for review, in some cases, samples of assessments were provided which had been confi rmed as good by external examiners, rather than those for which examiners had commented on the quality of the tutor’s feedback. In some cases the samples appeared to have been randomly selected. The initial review of the samples was focused entirely on the form and style used and did not attempt to establish the quality of the feedback. The review identifi ed that the form and style of feedback across the University varies considerably. Nominal Group Technique and focus groups The Nominal Group Technique (NGT) (Stewart and Shamdasani 1990:22–23) was adapted and combined with focus groups. The main objective for using NGT before focus groups was because it allows a balanced participation from each member of the group. Problems normally associated with individual contributions in focus groups were avoided (Stewart and Shamdasani, 1990:96–100); and the combination of methods proved particularly useful for getting students’ feedback (Hagyard and Keenan, 2007:1). Activity 1 Using the fi ve steps of the NGT: 1. The students independently produced written ideas – in this case their own perception of the characteristics of ‘good’ feedback - written on individual Post-it notes 2. Students then prioritised their ideas using A, B, C, D etc. 3. Each student read out their fi rst priority idea in turn, and then placed the Post-It on the fl ipchart. Each idea was discussed only to ensure there was a mutual understanding of its meaning. This was repeated until all the ideas were collected on the fl ipchart. 4. Students were given 5 points each to allocate amongst the collected ideas as they wished – 5 points to one idea, or distributed amongst a number of ideas 5. The ideas were prioritised according to the number of points they had been allocated. Finally, with the students’ consent, the focus groups were video-recorded for the purposes of data collection. 32 LILLY et al: USING FEEDBACK TO PROMOTE LEARNING: STUDENT AND TUTOR PERSPECTIVES Activity 2 Students were each asked to review two randomly selected, anonymised samples of assignment feedback provided by the faculties. They were not given copies of the assignments themselves. They were asked to use their list of characteristics of good feedback from Activity 1 as a guideline for their review, and to note any further characteristics. Phase 1: Findings Following the students’ discussions, the responses to Activity 1 were categorised as relating to the content, style or clarity of written feedback (see Table 1). The discussions demonstrated that these students place great trust in their tutors – they see them as the experts and are keen for their advice. Students mentioned that they are encouraged by enthusiastic tutors and wanted their feedback to demonstrate this. They stated that they want to feel that their work has been appreciated – even if it hasn’t passed – and they want to know that their tutor has put some work into marking it. Moreover, students want lecturers to be sensitive to the effort students make in constructing their assignments. Table 1. Students’ expectations from feedback. Content, feedback should contain: • Clear aims for improvement • Examples of how students can improve • Signposts to resources to aid improvement • Remarks on both the positive and negative aspects of the work • Remarks relevant to the work done • Clear links to the assessment criteria and learning outcomes • Comments on the quality of the work rather than the presence of elements of the assessment • An invitation to discuss the feedback Style, feedback should be: • Constructive in tone • Encouraging • Positive • Honest • Objective • Written in the third person Clarity, feedback should be: • Legible • Written in short clear sentences • Consistent between markers • Clearly aligned with the grade awarded Rae and Cochrane (2008) report on a similar research project to this one. They report on issues concerning clarity/lack of clarity as one of their arising themes, where ‘clarity’ contained the items that were defi ned here as ‘content’ (Rae and Cochrane, 2008:224–225). It is, therefore, possible to say that this parallel in fi ndings refl ects that across the sector. Students remain confused as to the potential that assessment feedback could have on their learning process (Chanock, 2000). In our study, students also showed awareness of assessment criteria in module guides, and expected their feedback to properly refl ect these. The use of assessment criteria sheets was – in general – considered useful. Nevertheless, in the students’ opinion, these should be relevant and allow for comment on the quality of the submitted assignment against the criteria, rather than just an indication of the presence of elements of the criteria. 33 PRACTITIONER RESEARCH IN HIGHER EDUCATION 4 (1) They additionally discussed the use of language – namely the tutor’s use of the fi rst or third person in written feedback. In fact, students identifi ed that since they are expected to use the third person in their assignments, they expected the tutors to use the third person in their feedback. They considered feedback written in the third person to be more objective, and to help ensure that comments were taken as relevant to the work, rather than as personal observations. This fi nding raised the most controversy when discussed with academic staff. It was concluded that the focus should be on feedback being neutral and fair rather than impersonal. Students suggested that opportunities to meet with tutors and discuss feedback are valued, particularly formative feedback on early drafts of the work with an opportunity to incorporate it into the fi nal submission. It was stated that feedback should be ‘substantial in totality’ i.e., the mix of feedback methods – on cover sheets, against criteria mark sheets, in discussion with the tutor – should address all the necessary points. Students did not feel that the length of feedback was important – too much was no better than too little. In their opinion, feedback should be concise and to the point. Finally, students indicated that they had found it diffi cult to accept constructive criticism when they fi rst started their studies, but had come to recognise its signifi cance for their learning as they progressed. They also identifi ed a need for students to be made aware of how to use and respond to constructive feedback. For instance, students who had participated in peer feedback had felt uncomfortable with it at fi rst, but ultimately found it very useful. Phase 2: Methodology Participants and procedures During the spring of 2008 workshops were held with colleagues from four faculties. To ensure a wide representation of disciplines from each faculty, sessions were arranged for up to 30 participants, with a minimum of seven. Seven sessions were held and a total of 57 members of staff attended. The sessions were planned to last three hours, with activities designed to address three primary goals: • for colleagues to have an opportunity to share good practice and learn from each other; • to collect contextualised examples of good practice, which could be returned to the faculties for the construction of localised guidelines; and • to secure examples that might inform the development of generic university-wide guidelines. The sessions included: • a brief presentation of the previous phase of the project; • collecting participants’ expectations of the workshops; • a video of a Phase 1 student focus group; • presentation of the Guidance and Feedback Loop (Hounsell, et al., 2006) a model selected as it identifi es different stages in the assessment feedback process, which can promote discussion. This model provides a vision of how to progress in relation to feedback and feedforward (Boud, 2007; Hounsell, 2007), since by developing feeding forward the process should become both sustainable, and a contribution to prepare students to learn in a holistic manner (Hounsell, 2007:103). 34 LILLY et al: USING FEEDBACK TO PROMOTE LEARNING: STUDENT AND TUTOR PERSPECTIVES The model depicts an illustration of the processes and infl uences upon students’ expectations of assessment feedback, including: (1) Students’ prior experiences of assessments; (2) Preliminary guidance; (3) Ongoing clarifi cation; (4) Feedback on performance/achievement; (5) Supplementary support; and (6) Feedforward. The Loop highlights the concerns of students about the guidance they have received before writing up an assignment, while they are writing it, and after submission (Hounsell, 2008). Furthermore, the Loop is understood as a reminder that the feedback received by students is closely linked to the guidance they are given before they start, while they are working on their assignments and after assignment submission (Hounsell, 2008:4). The structure The categories identifi ed by the students’ expectations from feedback (Table 1) were used to structure the staff workshops. The participants were organised into teams and were given the task of discussing one category and noting any examples of good practice arising during the discussions. These examples could be taken from their own teaching experience or from their knowledge of the practice of colleagues. Additionally, teams wrote down issues that they felt militated against good assessment and feedback practice. The staff discussed the gathered information at a plenary session. All plenary sessions were audio- recorded with the consent of the participants. Phase 2: Findings The activities provided data on issues that staff encounter during the stages of assessment feedback and beyond, and the plenary focused on sharing ways in which such issues can be addressed. Furthermore, it was demonstrated that academic staff have a wide range of experience and advice to share. A signifi cant fi nding was that, with very few exceptions, issues regarding assessment feedback are to a great extent common across faculties. Table 2 summarises the most signifi cant issues and suggestions for how common problems in assessment feedback practice could be addressed. This list represents the experiences of colleagues who attended the assessment feedback workshops. The data are presented according to the stages of the Loop (Hounsell et al., 2006). Table 2. Addressing issues in student feedback. The issue How can this be addressed? Students’ prior experiences of assessments Students perceive higher More work needs to be done on the information in module guides education as consumers; and student handbooks. This could include information about what education is seen as a product it means to go into higher education and expectations from students rather than as a process. and tutors. Students do not engage Introduce a facility on the virtual learning environment that would with feedback—they ask the students to discuss or evaluate the extent to which they are only focused on the understand the assessment process of each one of their modules. mark. Module leaders could analyse the results and take the appropriate action to ensure their modules’ assessment processes are understood. 35 PRACTITIONER RESEARCH IN HIGHER EDUCATION 4 (1) The issue How can this be addressed? Preliminary guidance Teaching is assessment focused. Develop features of the module guide such as learning objectives and Academic staff have little outcomes, assessment criteria, learning agreements, with students time to teach beyond what is during the fi rst sessions to generate a shared understanding of the required for the assessment. purpose of the module between the tutor and the students. Ongoing clarifi cation There is a separation between Introduce continuous forms of assessment which constructively assessment and feedback, and align assessment with learning and teaching. Examples include learning and teaching. presentations, project work and patchwork text, including peer assessment. Students have no input into Clarify – and where possible negotiate – the meaning of assessment the development of assessment criteria and provide formative feedback during the module before criteria… students are writing up their assignments. …so it is diffi cult for them to Use online activities involving peer feedback and refl ection, where understand the meaning of small groups work together to prepare their assignments. This could assessment criteria. be particularly helpful for part-time students. Group tutorials could be used to further clarify assessment criteria. The dialogue between tutors Students can self-assess their own work and hand in their comments and students is a one-way with their work for marking. Accordingly, tutors comment on and/ monologue – tutor to student. or mark the work, based on the students’ self-assessment. This starts a dialogue and decreases the sense of marking in isolation. Students begin to learn the ropes of self-assessment, and accept constructive comments in the form of feedback. Students need to develop Students present their work plans/drafts in the classroom. Peers (and insight into the quality of perhaps the tutor) review the draft before it is written up. This has their own work, to enable been found to enable experience of working as a team and promote a meaningful dialogue students’ success. between tutor and student. No face-to-face In an online-delivered module, students are asked to submit a self- interaction with the assessment with their fi nal assessment. This has proved useful with students. non-traditional students, e.g. people who have been away from institutional education for a long time. In this case, self-assessment has been built into the module. There is no training in marking The induction course could be expanded to include advice on and assessment. assessment and feedback. Also, subject, department-based staff development or team work can address specifi c marking and assessment issues for more established staff by sharing good practice. Lack of consistency between Specifi c support and guidance should be provided to (new) tutors, academic tutors as to how and to tutors who are given new modules to teach. Module leaders, much of a draft assignment who lead on a module delivery involving several staff, need to ensure they will see and comment on equality of contents, assessment and marking. Diverse forms of prior to submission. communication at different stages of the delivery process would be useful. 36 LILLY et al: USING FEEDBACK TO PROMOTE LEARNING: STUDENT AND TUTOR PERSPECTIVES The issue How can this be addressed? Feedback on performance/achievement Tutors do not have enough Allocation of realistic time allowances for marking and moderation time. built into the yearly calendar and academic timetable. Tutors have to teach big groups, After assignments have been marked, offer feedback to the entire and they don’t have time to group. Students are able to relate the generic feedback to their own feedback adequately. work, and therefore are better able to understand how their mark was arrived at, and how they could have done better. Feedback lacks clarity due to Arrangements should be made to allow feedback and marks to be unclear handwriting. given electronically/online. In some cases (e.g. online deliveries), formative feedback is already provided using email. Word-processed Feedback is provided very late feedback can address this issue, and ensure that staff have a or not at all. permanent copy of the feedback given to their students. Supplementary support Modules are not designed to Electronic feedback could include a fi eld in which students may allow time for tutors to support sign their agreement to allow subsequent tutors to follow up their students after the assessment. progress. Moreover, where possible, tutors can provide feedback on a previous module in a follow-up module. In some cases, personal development planning (PDP) has been used as a refl ective mechanism to feedforward. Feedforward The modular structure does Feedback could form part of PDP. Students should have a personal not permit the communication repository for their feedback, especially in the fi rst year. Students and of feedback across the levels, tutors could refl ect on the feedback they receive and identify generic preventing feedforward. issues and development needs. The Students’ Union has suggested that this would be more useful than discussing careers at that stage. At a later stage, the feedback would help tutors in giving career development advice. In addition to the issues listed in Table 2, the feasibility of anonymity in certain subject areas was widely debated by the workshop participants, in spite of the fact that anonymous marking is the current policy of the university. It is possible to locate the source of this debate within the double role played by the tutors who, at some point, ‘have to switch role from that of supporter of learning to assessor of achievement’ (Yorke, 2003:496). Since many teachers hold a great deal of knowledge about their students, the likelihood of anonymity being at risk is high when they have to evaluate their students’ progress, especially in subjects within the humanities, e.g. arts and design. The participants argued that sometimes it is impossible to be completely objective when marking and writing up their feedback because they could recognise the student they were assessing. Staff were aware of the benefi ts of giving feedback which could feedforward to subsequent assignments within the same module, and feedforward to future modules. Participants identifi ed a lack of understanding as to why so many students appear to make little or no use of feedback from past modules in future modules, yet it was reported that some students have asked where they could obtain copies of their previous feedback. Student engagement with the assessment process is essential to its success as emphasised by many authors (e.g. Handley et al., 2007; Hounsell et al., 2006; Nicol and Macfarlane-Dick, 2004; Sadler, 1989; Yorke, 2003). 37 PRACTITIONER RESEARCH IN HIGHER EDUCATION 4 (1) Discussion The fi ndings from both phases have strong resonance with the wider literature and research on assessment feedback. For instance, Handley et al. (2007:12–15) found that students required certain levels of psychological safety when receiving feedback on drafts. They also found that students admitted that they may take some time to develop the necessary academic skills to formulate questions to their tutors. Further, by using a scheme where feedback is given on drafts, there is no signifi cant impact on the workload of staff, and students fi nd the feedback more effective for feedforward to future activity (Handley et al., 2007:15). Academic staff identifi ed a range of issues they considered constrained their own feedback practice. These related broadly to: • time; • assessment and feedback practice; • organisation; • communication; • clarity; and • personality (Lilly et al., 2007). Within these categories, tutors identifi ed the need for guidance in designing assessment criteria and managing students’ expectations which, if addressed, would promote the development of the good feedback practice described by the students and staff in the study. There were also some ideas to support feeding-forward. For example, since the fi rst lecture of the year and the fi rst meeting with the students sets the scene for a module, it was suggested that it would be useful to talk about the students’ strengths and weaknesses during the fi rst week. This would help them to connect their past assessment feedback with the learning outcomes of the current module. Tutors and students should be encouraged to view the modules as part of a whole programme of study, instead of separate unconnected parts. Thus, both staff and students would be able to see the links across modules and elaborate on their feedback for the benefi t of students’ academic and professional development. As this project has shown, academic staff showed awareness of reasons why their feedback to students is not effective and how they could improve it (Race, 2008:4). However, this does not mean that tutors are solely responsible for the provision and effectiveness of assessment feedback. Nicol and Macfarlane- Dick (2004:1) emphasise that feedback is not exclusively produced by teachers. As this project has also identifi ed, feedback comes in various forms and may be produced by peers, tutors and from self- assessment (Race, 2008:8). Conclusion The student focus groups provided interesting insights as to how students perceive and receive feedback, which was classifi ed as being related to content, clarity and style. It was established that, with very few exceptions, issues and good practice in assessment feedback can be generalised across disciplines and, in the main, staff and students share their perceptions of what constitutes good assessment feedback. The project clarifi ed that written assignment feedback, whether formative or summative, should always be formative in nature. Students need to be able to receive good quality formative feedback to feedforward into their summative assignments (Lilly et al., 2007:10). Despite this agreement however, academic staff also identifi ed constraints on their own approach to assessment and feedback practice which indicate that the improvement of their assessment practice is not, for them, a priority. 38 LILLY et al: USING FEEDBACK TO PROMOTE LEARNING: STUDENT AND TUTOR PERSPECTIVES Therefore, in order to improve feedback it is not only assessment patterns that need to be reviewed, but also the culture of learning and teaching (Handley, 2009). A focus on increasing formative feedback (for example by the breakdown of a summative assignment into several continuous but smaller assessment tasks) challenges teachers to think creatively about how to maintain an appropriate assessment load for students, and a manageable marking load for themselves. Activities such as the use of peer feedback (Handley et al. 2007:12–15) blur the relationship between student and teacher and can be mistrusted by both. Findings like those reported here should inform and assist in the development and provision of support from wider institutional structures. For assessment feedback to be successful, it is not only research and theorisation which is required in this area (Rust, 2007; Sadler, 1989; Yorke, 2003). There should be an adequate infrastructure (that is accurate module descriptor forms, generic and, perhaps, specifi c assessment guidelines) to support the development of consistent assessment practice across disciplines. However, it is important too that tutors, as assessors, in accepting that it is as much their responsibility to clarify the assessment criteria and ensure this is understood by the students as it is the teaching of the subject (Yorke, 2003:487–488), also prioritise the improvement of their own assessment practice. References All websites accessed 3 January 2010 Black, P. and Wiliam, D. (1998) Assessment and classroom learning. Assessment in Education 5 (1): 7–74. Boud, D. (2007) Reframing assessment as if learning were important. In D. Boud, and N. Falchikov (eds.) Rethinking Assessment in Higher Education: Learning for the longer term: London: Routledge, 14–25. Carless, D. (2006) Differing perceptions in the feedback process. Studies in Higher Education 31(2): 219–233. Chanock, K. (2000) Comments on essays: Do students understand what tutors write? Teaching in Higher Education 5(1): 95–105. Hagyard, A. and Keenan, C. (2007) Nominal group technique: A brief guide. Learn Higher. www.learnhigher.ac.uk/View-document-details/737-Introduction-to-Nominal-Group-Technique.htm Handley, K. et al. (2007) When less is more: Students experience of assessment feedback, in the HEA (Higher Education Academy), 3rd Annual Conference. Harrogate. UK 3–5 July 2007. www.heacademy.ac.uk/resources/detail/when_less_is_more Handley, K. (2009) Engaging students with assessment feedback: ‘satisfying’ students or facilitating their learning? Keynote at Assessment in Higher Education Conference. Carlisle. 8th July 2009. C-SHEN, University of Cumbria: Carlisle. www.cumbria.ac.uk/Services/CDLT+/C-SHEN/C-SHEN docs/Conference8thJuly/ Handley-Carlisle8July09(2).pdf HEFCE and NSS (2008) Higher Education in Further Education Colleges in England: Outcomes of the 2008 National Student Survey. The Quality Assurance Agency for Higher Education. www.qaa.ac.uk/publications/ infobulletins/NSS08/default.asp Hounsell, D. (2007) Towards more sustainable feedback to students. In D. Boud, and N. Falchikov (eds.) Rethinking Assessment in Higher Education. Learning for the longer term. London: Routledge: 101–113. Hounsell, D. (2008) The trouble with feedback: New challenges, emerging strategies, TLA Interchange, (2): 1–9. www.tla.ed.ac.uk/documents/feedback2.pdf Hounsell, D., McCune, V., Hounsell, J. and Litjens, J. (2006) Investigating and enhancing the quality of guidance and feedback to undergraduate students, in ETL Project (Enhancing Teaching-Learning Environments in Undergraduate Courses), 3rd Biennial Northumbria/EARLI SIG Assessment Conference. Northumbria, UK, 30 August–1 September. www.etl.tla.ed.ac.uk//docs/earlinth06.pdf Irons, A. (2008) Enhancing Learning through Formative Assessment and Feedback. London: Routledge. Lilly, J., Richter, U. M. and Warnes, E. (2007) Assessment Feedback Project: Summary interim report. www.inspire.anglia.ac.uk/download.php?view.27 39