Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved Accelerating Educational Innovation in the MPH Degree Program: What is the Role of Peer Review of Teaching? Taryn Vian, SM, PhD Associate Professor Department of Global Health Boston University School of Public Health 801 Massachusetts Avenue Boston, Massachusetts, 02118, Telephone: (617)-638-5234 Email: [email protected] Paul G. Ashigbie, BPharm, MPH, DrPH(c) Research Fellow Department of Global Health Boston University School of Public Health 801 Massachusetts Avenue Boston, Massachusetts, 02118 Telephone: (617)-638-5234 Email: [email protected] ABSTRACT Purpose: The environment of public health practice is rapidly changing, creating the need to adapt graduate education and accelerate educational innovation. Formative peer review is a strategy designed to promote critical reflection on teaching and to develop faculty as teachers. Through case study methods, we explore how peer review of teaching may catalyze reflective practice and contribute to the redesign of public health education. Methods: We conducted a detailed contextual analysis of a peer review of teaching program implemented from 2011-2014 in a global health department with approximately 200 students and 26 teaching faculty. The case study used multiple data sources including peer review feedback reports, a survey of participating instructors and reviewers, and administrative data. Results: Faculty had favorable attitudes toward peer review, especially the opportunity to learn by observing others teach. Peer review reports confirmed existing good teaching practices and suggested ways to improve student engagement and develop teachers in ways which are aligned with MPH program redesign goals. Both peer reviewers and instructors reported benefiting from the process. Conclusions: Formative peer review may help to spread innovative teaching practices among faculty by providing opportunities for reflection, increasing motivation and confidence to adopt pedagogical changes, and strengthening teaching community. Increased knowledge of other teachers’ courses and social ties created through peer review can be resources in promoting MPH curriculum integration and collaboration across disciplines. Recommendations: Schools of public health should consider peer review of teaching as a pathway to promote greater student engagement and practice-based learning. In planning for peer review of teaching programs, leaders need to find ways to address time-related concerns and align the peer review process with other efforts to promote program redesign goals and strengthen teaching community. Accelerating educational innovation in the MPH degree program Page 43 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved INTRODUCTION Beh, 2014). PRT is defined as “informed colleague judgment about teaching for either The curriculum for the Masters of Public fostering improvement or making personnel Health (MPH) degree program is undergoing decisions” (Chism, 2007). Formative PRT significant changes. In 2011, a task force of the programs are designed to develop faculty Association of Schools and Programs of Public members’ self-awareness and provide teachers Health (ASPPH) studied the role of public health with information and an opportunity for critical education for the 21st century (Petersen & Weist, reflection to improve their teaching practice, 2014). The task force emphasized the need to while evaluative PRT programs seek to compare adapt education to a rapidly changing teaching to standards to make judgments environment in public health practice and to related to hiring, promotion, tenure, or salary accelerate educational innovation. The skills review (Chism, 2007; Thomas et al., 2014). PRT which students of public health need are can help teachers reflect on weaknesses and interdisciplinary, interprofessional, and systems build on teaching strengths, select teaching oriented (Frenk et al., 2010; MPH Expert Panel methods that can best meet the goals for a of the Framing the Future Task Force, 2014; course, develop more practice-based exercises Petersen & Weist, 2014). They reflect work-force and assessments, and integrate technology into competencies from the core disciplines of the classroom (Carter, 2010; Kohut, Burnap, & biostatistics, epidemiology, health policy and Yon, 2010; Thomas et al., 2014; Yon, Burnap, & management, environmental health sciences, Kohut, 2010). Faculty peers may be better and social and behavioral sciences in addition to qualified than students to assess the cross-cutting skills in communication, change appropriateness of course objectives, quality of management, informatics, political and cultural course content, whether important points are sensitivity, and leadership (Bernstein, Jonson, & being emphasized, and the aptness of examples Smith, 2000; McKee & Tew, 2013). The new used in class (Courneya, Pratt, & Collins, 2008; focus on applied learning and integration was d'Eon, Overgaard, & Harding, 2000; Nelson, expected to require major changes in teaching 1998). The Harvard T.H. Chan School of Public (Frenk, Hunter, & Lapp, 2015). Health includes peer review of teaching as a component of their new educational agenda, Research supports the assumption that the suggesting that it can be a vehicle to adapt the quality of teaching affects student learning, and values of collaborative, team-based research to continuous development of faculty as teachers is the teaching enterprise (Frenk et al., 2015). Yet, critically important to successful education little is known about the application of peer redesign (Bernstein et al., 2000; Clayson, 2009; review of teaching in the context of MPH McKee & Tew, 2013; MPH Expert Panel of the education. Implementation of PRT programs in Framing the Future Task Force, 2014). To other disciplines has proven challenging achieve the goals proposed by the ASPPH task because the goals and format must be adapted force, course design and teaching must change to institutional context, including the in ways that promote student engagement and department’s mission, values, student practice-based learning to build competencies in demographics, resources, and faculty disciplines public health, with more time devoted to (Chism, 2007; Quinlan & Akerlind, 2000). Other applying skills such as data analysis or critical obstacles or barriers to successful program reasoning, and practicing communications and operation include fear on the part of the leadership (Brame, 2013; Frenk et al., 2015; instructor being reviewed, uncertainty about Fulton, 2013; Herreid & Schiller, 2013). Yet to what to review, resistance to innovation that date, most of the literature on redesigning the involves participants doing more work, and MPH has focused on curriculum choices (what implementation factors such as how reviewers to teach) rather than teaching methods and are chosen, power differentials, and the amount professional development of teaching faculty of guidance or flexibility built into the process (how to teach). (Costello, Pateman, Pusey, & Longshaw, 2001; Courneya et al., 2008; Frenk et al., 2010; Kell & Peer review of teaching (PRT) is one method Annetts, 2012; Thomas et al., 2014). We of promoting professional development of examine issues encountered in using PRT to teachers, encouraging reflective practice, and help faculty in one department reflect on motivating faculty to teach from a different teaching practice, and explore what role PRT perspective (Thomas, Chie, Abraham, Raj, & might play in accelerating educational innovation Accelerating educational innovation in the MPH degree program Page 44 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved in MPH programs across departments and application of public health skills with the schools of public health. instructor as mentor and coach. Five task forces were created at the school level to promote the PURPOSE programmatic changes: core curriculum, concentrations, practicum, culminating The purpose of this study is to explore how a experience, and teaching methods and global health department at a major school of assessment. The teaching methods and public health implemented PRT in conjunction assessment task force called for competency- with MPH curriculum redesign, in an effort to based learning to promote skills which would improve teaching which would reinforce improve students’ job-readiness, and highlighted competency-based learning. The study the need to develop faculty and provide them documented the views and perceptions of with support for pedagogical changes. faculty about the PRT process, the types of feedback and observations gleaned through The primary objective of GH’s peer review of PRT, and the challenges in carrying out the teaching policy was to improve the quality of program. The insight generated from this study teaching. PRT was intended to affirm the may help envision the role of PRT in promoting importance of teaching, encourage innovation innovation in MPH education, and inform the and improvement in teaching methods, and design and implementation of PRT across encourage sharing of teaching techniques. It departments and in other public health was seen as a form of collaborative analysis that institutions. would help individual faculty to develop as teachers while reinforcing their commitment to Background on the Peer Review of Teaching teaching. Excellence in teaching would in turn Policy and Process improve the competencies of MPH graduates, The Department of Global Health (GH) especially in light of the move toward admits 90-110 new MPH students per year. Full- competency-based education. A second time students generally can complete objective of PRT was to enable GH faculty to coursework in three semesters including a understand the courses taught by other faculty practicum experience. The Department adopted members so that faculty could become better a PRT policy in December 2010. While the advisors, more successfully matching students number of faculty in the department varies over with course offerings that fit their needs and time, the department generally has 24-26 interests. Finally, the department hoped that teaching faculty in a given year, and 6 research PRT would help to create a stronger faculty. Teaching faculty are expected to teach professional community in order to resist the at least 8 credits as well as doing research, centrifugal tendencies which are inevitable as while research faculty are primarily engaged in faculty members pursue individual career goals research and are not required to teach, though and funding opportunities (d'Eon et al., 2000). they may offer to teach a course in exchange for salary coverage. As the School of Public Health PRT process does not have tenure, all faculty are non- The process was designed so that teams of tenured. two faculty members observe each course assigned for review. In fall and spring Implementation of peer review of teaching semesters, the Chair or his/her designee coincided with a school-wide redesign of the develops a list of courses to be reviewed. Every MPH curriculum started in 2011. The goal of this course is not reviewed every semester, and redesign was to provide graduates with the reviews are not conducted in summer to ease knowledge and skill sets needed for the burden on faculty. Typically a course will be employment, and to assure that the program reviewed every 2-3 times it is offered. The Chair was adapted to the changing context in which or designee assigns faculty teams to review the public health is practiced. As many students courses. Factors considered in selecting were coming straight from undergraduate reviewers are the background and strengths of studies without much work experience, the reviewers and how they may be helpful to the redesign aimed to increase applied learning instructor, and the reviewers’ ability to benefit experiences to simulate work settings. Ideally from the review by gaining ideas for their own students would be spending less class time courses. Over time, review teams are exposed listening to lectures and more time practicing the to many different courses. Given the inter- Accelerating educational innovation in the MPH degree program Page 45 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved disciplinary nature of global health, reviewers faculty. The time period examined was January often do not have content expertise in the topic 2011 to May 2014 (7 semesters). A combination of the course being reviewed. This is not seen of data collection methods is well suited to as a problem, as the goal of PRT is to improve understand the relationship between and share teaching methods, and to provide organizational behavior and its specific context opportunities for collaborative reflection on (Hartley, 1994). Administrative data sources teaching. Peer review teams are encouraged to included the school’s registrar office records meet with the instructor before observing the (courses offered by the department within this class. During this meeting they discuss specific period, and the instructors who taught the aspects of the course the instructor may want to courses) and records from the department focus on, and review the course goals, syllabus, curriculum coordinator’s office (courses and teaching and assessment methods. This is assigned for review each semester, assigned meant to provide context for the observation and reviewers, number of reviews which actually to help the reviewers see how the session to be took place, and availability of review memos). In observed fits into the instructor’s overall plan. addition, we used the peer review memos to Before conducting the observation, reviewers determine the number of sessions observed for may also examine other materials related to the each course, and to help determine which course including the course web site, e-portfolio faculty members assigned to the review actually site, or past student course ratings. participated. We also reviewed the content of all written peer review memos submitted during the Review teams schedule a time to observe the study time period to elicit themes. class. The team is expected to attend one (generally 3-hour) class session and to review To complement the administrative data and the materials for the class. Given travel peer review memo information, we conducted an schedules for GH faculty, sometimes reviewers anonymous survey of GH faculty using split up and review different sessions of the Qualtrics® online software. The survey was sent course. Following the review, the review team is to 26 full-time teaching faculty with regular expected to prepare a brief memorandum appointments. The survey was sent in January summarizing impressions and suggestions. This 2014, and the recall period was 2013. Faculty document is shared with the instructor, and the answered questions about the time spent in the reviewers and instructor meet to amplify or peer review process, knowledge, and attitudes clarify the comments. The memorandum is then or opinions about the program. The knowledge, submitted to the faculty chair who reviews to attitude, and opinion questions were evaluated identify and prioritize resources to support on a 5 point Likert-like scale (1=strongly agree, 5 changes in the course or skill development for = strongly disagree). Knowledge of the policy the instructor. The review does not become part was probed with two questions: “The department of the faculty member’s personnel file, nor is it uses peer review feedback in the faculty used for annual performance review. Peer performance review and planning process,” and review tends to be more effective when the “The department uses peer review feedback in process is flexible and owned by the the curriculum revision process.” We asked participating faculty (Thomas et al., 2014). To three questions about usefulness of peer review, encourage ownership, guide questions and an including “PRT helps me learn new methods of optional rubric are provided as possible tools, course design, teaching, or assessment,” “PRT but the review teams can decide which tools increases my knowledge of course offerings,” they want to use. The optional guide questions and “PRT helps me to get to know other and rubric focus on aspects of course instructors better.” Finally, we asked opinion organization, clarity of content, interaction in the questions to probe whether the reviewer thought classroom, lecture style, discussion the instructor had found the feedback helpful, management, use of media, and exercises or and whether the instructor found the reviewer’s projects to promote applied learning. feedback helpful. We did not ask any demographic information in order to protect METHODS confidentiality. We applied case study research methods to provide a rich description of how the PRT program is operated and was perceived by Accelerating educational innovation in the MPH degree program Page 46 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved Analysis RESULTS Administrative data were analyzed in an Administrative Data Excel spreadsheet. We calculated total and Twenty-six faculty participated in the PRT average per semester for: courses offered; program as a reviewer for at least one semester courses assigned for review; reviews which with 11 faculty participating as reviewers 3 or actually took place; reviews for which a written more times between 2011 and 2014. Each memo was available. We also calculated the faculty member reviewed 2.5 courses on average number of reviews per course; average average. The GH department offered 19 courses number of reviews per teaching faculty member; and scheduled 9 peer reviews per semester average number of sessions observed per (47%) on average. review; and proportion of faculty assigned to review who actually participated. We Over the 7 semesters, 28 unique courses summarized average length of memos, and the were offered and all were scheduled for at least proportion of memos which used an evaluation one review. Core courses with sections taught rubric versus narrative style. by different instructors were often scheduled for review multiple times. Of the 62 reviews The content of peer review memos was scheduled, written memos were available for analyzed using a modified grounded theory only 37 (60%), and verbal feedback was given approach (Corbin & Strauss, 2008). Our for an additional two reviews (3%). It is assumed analytical strategy involved constant that the remaining 23 scheduled reviews (37%) comparison, i.e. looking at observations and probably did not take place due to scheduling recommendations contained in each memo, and problems or travel. This means that of the 28 comparing it with observations and comments in courses scheduled for review, 4 (14%) were other memos for similarities and differences. never reviewed, 13 (46%) were reviewed once, Segments of text which were found to be 9 (32%) were reviewed twice, and 2 (7%) were conceptually similar were grouped together reviewed three or more times. On average each under higher-level descriptive concepts (Corbin unique course was actually reviewed 1.3 times & Strauss, 2008). The lead author, who is an with written results. experienced researcher trained in qualitative methods, read each memo and entered sections Some courses scheduled for team review of text into an Excel spreadsheet. These were actually reviewed by only one faculty segments of text were then analyzed and member. On average, 85% of faculty who were assigned codes to indicate different descriptive assigned to a reviewed course contributed to a themes (Miles & Huberman, 1994). review memo, and review teams observed an average of 1.3 sessions per course. Review Survey data were analyzed in Excel. Data memos were generally 1-3 pages long. Eight were stratified by reviewers and instructors memos (22%) incorporated an optional rubric where necessary, and described using which rated class organization, lecture style, proportions and averages. exercises, student engagement, etc. as “excellent”, “good”, or “needs improvement,” Institutional Review Board Approval while the remaining review memos used The study was reviewed by the university’s narrative style. All review memos included a Institutional Review Board and found to be summary of observations and suggestions. exempt because it does not meet the definition of human subject research. Review Memo Content Reviewers provided comments on topics Informed Consent including the syllabus, choice of teaching The survey used for the study was methods, classroom management, lecture skills, anonymous and responding to the survey was and discussion management. Feedback optional. We used de-identified data for this depended on whether the instructor had specific study and therefore we could not obtain goals which he or she wanted the reviewers to informed consent of study participants. consider; for example, one instructor asked for reviewers’ recommendations for how students could become more engaged during class sessions while another instructor was interested Accelerating educational innovation in the MPH degree program Page 47 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved in better ways to evaluate group projects. of the overall course goals. They also suggested Feedback on content was offered if the reviewer additional readings or perspectives connecting had related expertise, such as an infectious other disciplines: for example, a professor with a disease doctor reviewing a course on humanities degree, after reviewing a course controversies in global disease control and session on vaccine refusal taught by a medical eradication. Recurrent themes in the peer review doctor, ruminated that John Stuart Mill would feedback included: active learning, making ultimately come down on the side of compulsory connections between course objectives and vaccination without exclusions because not activities, and teaching techniques. vaccinating compromises the health and liberty of others. The reviewer supplied a lengthy quote Active learning from Mill’s book On Liberty to support this Reviewers gave praise and constructive argument. criticism to instructors to enhance active learning and student engagement, as shown in these Frequently, reviewers observed ways in quotes: “The session was a great example of the which the instructor could make explicit the "inside out" organization of teaching where the connections between sessions, summarize key students are actively engaged in learning. Work points, or anticipate what is coming next as in outside the session (watching video lectures, these quotes: “It might be good to introduce the preparing their own group's consulting readings for the day by saying how or why they assignment, reading another group's report) were chosen, or what the students are meant to made it possible for this session to be so active. get out of them. Are they linked in any way; an During class, groups of students evaluated overarching theme?” “I don't usually subscribe another group's report on a drug management to the philosophy that one needs to hit students issue, and prepared and gave a 10-minute over the head with the course objectives, but it presentation with the findings of their might have helped [if you would] place the evaluation.” “[In this session,] student groups discussion in the overall context of the course had designed active-learning strategies to and give objectives for the lecture.” convey nutritional messages. Each group had just 5 minutes. One group did a charades game, Teaching styles and techniques another used a polling software to survey class Reviewers observed ways in which the participants through text messaging, a third instructor set up the classroom to enable active group had written a song with nutrition discussions, intervened to prevent domination messages in the lyrics…the students seemed by vocal students, and used techniques such as happy and engaged.” Active learning sometimes randomly picking students to lead discussion as involved individual activities such as students a way to motivate students to be more prepared filling out a worksheet or team activities and therefore more able to actively participate. (presentations, group projects) inside or outside Reviewers mentioned ways to widen the classroom. One reviewer acknowledged a participation, suggesting techniques such as faculty member’s innovation in incorporating coaching student discussion leaders and tweeting into the classroom while also preparing discussion questions in advance. questioning how to evaluate the impact of tweeting on educational outcomes. Other Reviewers gave feedback on systematic reviewers suggested strategies such as pair- organization of lectures, confidence and shares, “clickers” (audience response system), command of material, the instructor’s voice and “minute papers” (a classroom assessment enthusiasm, and the preparation of slides. technique to get rapid feedback on a main idea Reviewers liked when instructors integrated introduced in the session), and structured examples and experience into lectures, as shown in these quotes: “One of the most discussions to further engage students. effective elements was when the instructor Connections and context turned the lecture from conceptual to the specific Another theme of reviewer comments was by integrating data from recent scientific helping learners see connections and publications which were thematically linked to understand the context in which course the student projects.” “The instructor spoke from concepts can be applied in professional practice. experience, probably the greatest strength of the Reviewers praised instructors for using class. It really addresses, ‘what is it like to work techniques which put the session in the context in the field?’” Many reviewers made suggestions Accelerating educational innovation in the MPH degree program Page 48 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved about how to design better presentations, use About three-quarters of instructors believed questions more effectively, and wait an that peer review was important for faculty adequate amount of time for students to answer. development and 44% of instructors reported they had already made changes in course Other themes design or teaching based on peer review Reviewers often commented on themes ripe feedback. Faculty shared their positive for deeper discussion among faculty, such as perceptions, as in these quotes: “I think this whether instructors should ‘cold call’ on quiet process helps knit the department together.” “I students, the appropriate role of teaching highly value peer review. I’ve made changes assistants in lecturing, whether to allow re- based on comments from previous reviews.” “I submissions of work for a higher grade, how to enjoyed having my colleagues in the classroom. assure individual accountability in group I always ask them to participate and that adds a projects, and what to do about students who are nice new voice to the session.” Other faculty web surfing or emailing in class. While reviewers voiced doubts about the usefulness: “[The acknowledged the complexity of these issues usefulness] depends on the feedback provided. and there is likely no one right way, they would Teaching styles vary and not all often mention their own preferences as these recommendations are helpful.” “The reviewer reviewers did on the topic of grading: “I wonder if came—but so far has not responded to request students will be happy doing so much [group] for feedback.” Fifty-three percent of instructors work and getting few individual grades, but that thought peer review of their course was may just be my prejudice against group projects. worthwhile or very worthwhile. I didn't like them as a student and did not find them useful when I tried them as a teacher.” “If Attitudes toward peer review suggest that the we are really interested in the intellectual growth reviewers gain as much if not more from the of our students …then we would want to peer review process as the instructor whose encourage them to redo work, continue to course is being reviewed (Table 1). Almost practice, and if that produces a better product, three-quarters of reviewers said that conducting reward them accordingly.” peer review helped them to learn new design, teaching and assessment methods. Reviewers Survey Data also said they gained knowledge of the range of We received responses from 19 of 26 GH course offerings in their department and got to faculty surveyed (73% response rate). Fifteen know other faculty better through the peer (79%) of the respondents had participated in review process, as these quotes suggest: “I PRT in 2013. Where reviews were not enjoy doing the reviews. I especially like meeting completed, faculty explained reasons including with the instructors ahead of time and finding out scheduling problems (being asked to review a what they like or are worried about in their course that met when the reviewer was courses. I like to see how other instructors teaching), international travel, and “running out structure their courses.” “I value the opportunity of time” or forgetting. Reviewers reported to learn from my colleagues and offer feedback. spending a median of 4 hours while instructors While it can be hard to fit in our busy schedules, said they spent 0.5 hours on the peer review it’s always worth it.” (instructors did not count the time being observed, as they would have been teaching Some respondents didn’t know how their anyway). feedback had been perceived or suspected that the feedback they gave to instructors was not Most of the faculty surveyed said that when welcome. In addition, reviewers were not aware they had reviewed a course, they had met with of how the department was using the peer the instructor, reviewed the syllabus, observed a review feedback: only 27% believed the session, and shared written feedback with the department uses the peer review memos in the instructor afterwards (Figure 1). Less common curriculum revision process (a stated goal of the peer review activities were talking to students to peer review policy) while the same percent get their feedback and reviewing the course web believed that the department used the peer site or past course evaluations which are review feedback in the faculty performance available to all faculty and students online. In review and planning process (the policy explicitly one case, a pair of reviewers had conducted an states that the peer review process is not online survey to get student feedback. evaluative). Accelerating educational innovation in the MPH degree program Page 49 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved CONCLUSIONS As has been noted in other studies, the PRT process provided opportunities for faculty Through this case study, we documented and development not only for the instructor but for analyzed the implementation of a PRT program the reviewer as well (Kohut et al., 2010; meant to further MPH curriculum re-design goals Wilkerson & Irby, 1998). In one study, reviewers to emphasize competency-based learning. We perceived the process more useful for their own found that the peer review process was teaching than did faculty being reviewed; generally, though not uniformly, well received by however, the PRT process in that study was faculty. The proportion of instructors who said being used for evaluative purposes to make they had already made changes in course tenure and promotion decisions, a different design or teaching as a result of PRT (44%) was context (Kohut et al., 2010). Observing someone similar to that found in other studies; for else is a way to self-critique and learn new example, about one-third of participants in a practice (Chism, 2007), and peer review helps PRT program at the University of Nebraska faculty interact and learn from one another, reported making significant changes to some providing collegial support (d'Eon et al., 2000). components of teaching as a result of the The GH PRT program shows that this was an program (Bernstein et al., 2000). The content of important outcome: 78% of reviewers and 72% review memos was consistent with the stated of instructors noted that the peer review process goals of the policy, i.e. to encourage innovation helped them get to know other faculty as and improvement in active teaching and learning colleagues. The increased knowledge of other methods, help faculty develop as teachers and teachers’ courses and the social ties created advisors, and to create a strong community. through PRT are resources which can be used in developing MPH curriculum integration and One theme emphasized by reviewers was making connections between coursework and the need to include more practice-based public health practice. Over 80% of reviewers examples in lectures. This suggests that public noted that PRT helped them understand the health faculty recognize that practice experience Department’s courses, which helps them is highly valued by graduate students, most of become better advisors to students. In addition, whom are being trained for practice rather than the GH department has added “teaching research careers or academia. Integrating innovation” as a standing agenda item at examples and stories from applied practice monthly faculty meetings. This additional forum helps students to see the relevance and for communicating about teaching aligns with possible applications of theoretical concepts. the PRT goals and further reinforces the While not, in itself, active learning, this may help teaching community. It has helped faculty make the material more stimulating and identify in-house experts for particular interesting which is a quality of good teaching innovations, and has helped to emphasize the (Thomas et al., 2014). Reviewers’ specific value placed on educational innovation. criticism of lecture styles also seemed intended to help instructors increase engagement with In addition to providing feedback to individual students at their level of understanding teachers, the PRT process highlighted more (Wilkerson & Irby, 1998). general issues or questions which leaders may want to address in the context of school-wide A large number of comments from reviewers MPH redesign. Topics included the value of related to how instructors could strengthen the competency-based grading (i.e. allowing connection between the content of the course students to re-submit work until they achieve and the situations in which students might be competency, and only grading the final product), expected to use or apply the content as public how to engage introverted or shy students and health professionals. In active-learning courses, students from diverse cultural backgrounds, and this is a challenge because the key points are how to minimize technology distraction while emerging in real-time from the class. Faculty also meeting students where they are, i.e. members who are innovating with active incorporating social media and other new learning may not yet be fully competent at aiding technology into the classroom. It would be students’ sense-making when the activity helpful to provide more opportunities for the concludes (Ketelaar, Beijaard, den Brok, & faculty as a whole to discuss such observations Boshuizen, 2013). PRT provides an opportunity and issues. PRT helps to promote active, to coach faculty in this difficult skill. practice-based learning because it implicitly Accelerating educational innovation in the MPH degree program Page 50 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved recognizes that teaching is not merely a purpose of PRT and taking action to reduce technical enterprise, but a social practice (d'Eon power imbalances in pairing of reviewers and et al., 2000). By allowing the reviewer and the instructors is even more important to ensure instructor to define the process, valuing the acceptance of the process. process of reviewing as well as receiving feedback, and providing multiple opportunities to Another factor associated with successful engage in reflection, PRT helps facilitate the PRT is whether faculty have established norms adoption of new social practices around of shared inquiry and prior experience working in learning, such as project-based learning. teams (Thomas et al., 2014). Research has shown that in disciplines where faculty naturally Limitations engage in individual research with sole This study has several limitations. Although authorship, PRT may be less successful we had a high response rate to the online (Thomas et al., 2014). Our case study was survey, not all faculty responded and we cannot conducted in a department which was already know how the perceptions and experience of used to working in interdisciplinary teams for non-respondents might have differed. The research projects, which may have made it thematic analysis drew on written peer review easier to apply a team approach to reflect on memos, which were only provided for 60% of teaching. In situations where team work among scheduled reviews. Finally, our analysis of a faculty is not the norm, leaders may need to single case study may not reflect the experience spend more time in forming teams and of other schools or departments. However, the developing teamwork and reflective practice framework that resulted may help guide the skills to ensure PRT works well. design of PRT programs in other settings, as discussed below. Policy Design Leadership is essential to motivate faculty RECOMMENDATIONS especially in public health schools with soft funding where faculty are juggling the need to Based on our review of PRT literature and cover their salary by generating research dollars this case study, we have created a framework and may see teaching as less essential for which captures our experience and may guide survival. Time and burden on faculty are other schools of public health considering consistently raised as issues affecting PRT adopting PRT as part of the redesign of MPH implementation success (Kell & Annetts, 2012; programs (Table 2). The framework illustrates Siddiqui, Jonas-Dwyer, & Carr, 2007; Thomas et pre-disposing factors, policy design, and al., 2014). While the model of peer interaction program implementation features, and how and consultation adopted in this case study was together they may lead to the intended not very intensive--requiring about a half-day of outcomes of PRT. each participating faculty member’s time per semester and minimal administrative support-- Pre-disposing Factors the program experienced compliance problems: Positive perceptions of PRT are correlated 40% of review teams either did not submit with collegiality and good relations among written memos or did not conduct the assigned reviewers and observers (Byrne, Brown, & review, and several comments from faculty Challen, 2010). Power dynamics and fear can mentioned the time burden. Leadership is have an important influence on PRT needed to signal that committing time to the implementation and outcomes (Byrne et al., process is valued and rewarded. 2010; Kell & Annetts, 2012; Kohut et al., 2010). Our study was conducted in a non-tenure Leadership is also needed to monitor and situation, which may have reduced power evaluate how PRT initiatives contribute to the imbalances as we did not need to consider the outcomes of student learning and professional social relationships among tenured and non- development. Closer management with more tenured faculty when assigning reviews or frequent reminders and assistance in scheduling sharing review feedback. The absence of tenure could also help increase post-observation may have made it easier for faculty to see the written feedback from reviewers and facilitate review process as developmental rather than as deeper, more meaningful discussion of an evaluation tool (Peel, 2005). In a tenure observations. Leadership must also make sure system, being very clear about the formative that the structure for PRT is focused and Accelerating educational innovation in the MPH degree program Page 51 Journal of Health Education Teaching, 2015; 6(1): 43-56 Copyright: www.jhetonline.com All rights reserved periodically reinvigorated to avoid stagnation that reviewers were influenced by their own and complacency (Byrne et al., 2010; Kell & perspectives on teaching, e.g. an aversion to Annetts, 2012; Kohut et al., 2010). For example, group projects. In addition, our finding that some to re-invigorate a moribund PRT program, one reviewers were uncertain whether their advice university introduced year-long peer was useful suggests that it might have been development team projects, where 2-5 teaching helpful to discuss criteria for good teaching to staff worked together on a project to improve develop a common understanding. This need practice such as effective use of tutorial support, not reduce the flexibility afforded the faculty in planning and developing a new course, or the tools and process used for review. In fact, adapting pedagogy to use interactive initial cycles of PRT could be used to construct whiteboards (Byrne et al., 2010). Kell and evidence of qualities of effective teaching (Drew Annetts (2012) found changing teams was & Klopper, 2014). helpful so conversations didn’t stagnate and reflection continued to be constructive. Program Implementation A key question in PRT program The developmental focus of the PRT implementation is who is qualified to be a peer program must be clearly communicated in order reviewer (Thomas et al., 2014). One study to reduce fear and foster development of determined that feedback received from subject reflective practice (d'Eon et al., 2000). Even specialists and non-specialist assessors was though PRT in this case study was formative equally reliable and valid (Hanson, 1993). Our and not evaluative, there were misconceptions investigation provides further evidence that among faculty. For example, about a third of reviewers with different disciplinary backgrounds faculty believed the department uses the peer can still provide useful formative feedback and review feedback in the faculty performance ideas to stimulate critical reflection. While PRT review process, which is not the case. Ensuring can help assess relevancy of course content, faculty are clear on the purpose of the PRT may the purpose of formative PRT is much more increase engagement with the process. about encouraging reflective practice-related dialogue, sharing ideas about specific pedagogic In addition, faculty ownership of the process, innovations, and sharing constructive feedback through opportunities to choose tools and that can lead to new understanding and approaches and to influence the way reviews improved practice (Kell & Annetts, 2012; are carried out, increases positive perceptions of Thomas et al., 2014). In order to achieve these PRT and may enable a more critical and goals, the program needs to assign reviewers to collegial examination of teaching practice maximize exposure of individual faculty to (Thomas et al., 2014; Toth & McKey, 2010). We different styles and approaches to teaching. fostered faculty ownership through opportunities to choose rubrics, reporting style, specific Training of peer reviewers may also be session or sessions to attend, and the types of beneficial. Although the GH PRT did not provide other information to consider (e.g. student specific training to peer reviewers, some studies evaluations, web sites). have shown that trained reviewers may be more accurate observers and insightful of their own A factor which has impeded PRT in some capabilities, thus establishing trust (Carolan & studies is uncertainty about what should be Wang, 2012; Drew & Klopper, 2014; Kohut et reviewed (Thomas et al., 2014). Some PRT al., 2010; Thomas et al., 2014). Training should processes use external criteria, while others recognize that peer review is not just a way to base the assessment on goals set by the identify mechanical “tips” to improve teaching, reviewer and instructor (Siddiqui et al., 2007). but rather it is a service activity to foster The GH department’s PRT policy did not define questioning of beliefs, assumptions, and habitual “good teaching” criteria in advance other than practices around teaching (d'Eon et al., 2000; suggesting that improved teaching should Thomas et al., 2014). promote competencies that MPH graduates need in the work place. Review teams were Program implementation requires special allowed to decide which aspects of teaching to attention to create administrative tools to keep assess, and how to interpret and apply the goal track of reviewer assignments and reports, in of competency-based learning. As in other order to assure equal coverage of courses, and studies (Courneya et al., 2008), we observed diversity and breadth of experience for Accelerating educational innovation in the MPH degree program Page 52