International Journal of Teaching and Learning in Higher Education 2011, Volume 23, Number 1, 114-118 http://www.isetl.org/ijtlhe/ ISSN 1812-9129 Investigating Peer Review as an Intentional Learning Strategy to Foster Collaborative Knowledge-Building in Students of Instructional Design Jennifer M. Brill Charles B. Hodges Virginia Tech Georgia Southern University Peer review has been advocated for as an intentional strategy to support the knowledge and skill attainment of adult learners preparing for professional practice, including those students preparing for instructional design and technology practice. The purposes of this article are to discuss the practical application of peer review as an instructional strategy by articulating its use in both face-to- face and online Instructional Design courses and to formulate directions for future research on the use of peer review in instructional practice. Findings from a literature review of student-to-student peer review and the authors’ experiences with the use of peer review in Instructional Design courses are used to foster a discussion that interweaves both important scholarly and practical elements. Citing Mills and Cottell (1998), Bangert (2001) standards developed by the Association for Educational observed that several professional organizations, Communications and Technology (AECT, 2001) and including those affiliated with such diverse professions the National Council for Accreditation of Teacher as accounting and teaching, endorse “instructional Education concerning formative evaluation. Peer strategies, that promote active learning, complex review supports the concept of practice and its ongoing problem solving, experiential approaches, group work, development as taking place within a situated and and innovative uses of technology” (p. 77). Current authentic context that supports a community of shared national standards for educational technology goals, artifacts, and interactions (Brown, Collins, & demonstrate that these aims are relevant for students of Duguid, 1989; Wenger, 1998), as well as the instructional design and technology. Specifically, the constructivist notion of shared knowledge-building National Educational Technology Standards and through experience (von Glasersfeld, 1995). Peer Performance Indicators for Students (ISTE, 2007) review as an instructional strategy also aligns with the promote communication and collaboration, critical priorities identified by research on professional groups, thinking, problem solving, decision-making, and digital group work, and group learning by fostering citizenship, the last of which includes a positive attitude interpersonal skills in the marketing classroom toward using technology that supports collaboration. (Chapman & van Auken, 2001) as well as shared One instructional strategy in alignment with such creativity and reflection in the geographical information professional standards is student-to-student peer review systems classroom (Livingstone & Lynch, 2000). In of course-related work. For the purposes of this fact, peer scaffolding is identified not only as a viable discussion peer review refers to "the structuring of a alternative to instructor-scaffolded activities (Lai & process to allow peers to review each other's Law, 2006), but also as a vital element to the professional processes and/or products with the goal of collaborative group learning experience (Dalgarno, improving such processes or products” (Woolf & 2001; Towns, Kreke, & Fields, 2000). Quinn, 2001, p. 22). Peer review is a learning strategy The purposes of this article are to discuss the situated at the highest level of Bloom’s taxonomy of practical application of peer review as an instructional learning in the cognitive domain (Bloom, Krathwohl, & strategy in both face-to-face and online Instructional Masia, 1956). Therefore, it is an attractive goal for Design courses, and to formulate directions for research educators, particularly those facilitating the learning of on the use of peer review. Findings from a literature adults preparing for professional careers requiring the review of student-to-student peer review and the analytic and evaluative skills associated with problem- authors’ experiences with the use of peer review in solving, and certainly in fields such as teaching and Instructional Design courses will be used to initiate and instructional design and technology. Peer review is well foster the discussion. aligned with the concept of formative evaluation (Dick, Carey, & Carey, 2009) and, therefore, fits especially Conceptual Context well within the context of an Instructional Design (ID) course, where formative evaluation is an important A review of the literature on peer review as an concept and skill for students to master. Incorporating instructional strategy offers the following insights. peer review of course-related project work in an First, findings reveal that peer review benefits students Instructional Design course reinforces the accreditation by helping them to: identify good practice and be more Brill and Hodges Peer Review as a Learning Strategy 115 critical (Davies, 2000; Harris, 2006), strengthen self- experience at a large research university in the regulation behaviors in order to provide constructive southeastern United States. feedback on peer assignments (Ku & Lohr, 2003), The Principles of Instructional Design course is a Ozogul, Olina, & Sullivan, 2008), develop critical required course for all Instructional Design and thinking skills (Li & Steckelberg, 2004), articulate Technology (IDT) majors. Residential masters and design decisions in a professional context (Casey, doctoral students complete this foundational ID course Branvold, & Cargille, 1996), and comprehend the face-to-face in the fall of their first year in the program. problem-solving and formative nature of professional The course enrollment averages fifteen students, many practice, including instructional design practice (Woolf of whom are international students, most of whom do & Quinn, 2001). Second, peer review benefits not have formal instructional design training or instructors in that it may reduce the time required to experience, and a few of whom come from other evaluate complex assignments (Bangert, 2001; Ozogul, disciplines such as educational psychology, engineering et al., 2008), thus potentially providing more time to education, and agricultural education. The course meets offer higher level consultative guidance. Third, best face-to-face for three hours per week for fifteen weeks. practices in peer review suggest that instructors should Students are introduced to the concept of peer provide clear criteria for peer feedback to avoid review as a practical means for engaging in superficial feedback (Ku & Lohr, 2003), train students instructional design work during the second class on evaluation processes (Ozogul & Sullivan, 2009), and meeting. The instructor discusses the concept, presents use blind review to reduce bias (Li & Steckelberg, a generic process for completing peer review, and 2004; Ozogul, et al., 2008). Fourth, challenges to peer solicits input on common “rules of engagement” when review include: fostering a work context that feels safe it comes to providing feedback. After reading about and and familiar enough for peers to become and remain discussing instructional goals, they complete a brief in- engaged in productive ways and providing enough class assignment during which each student drafts an guidance and structure for peers to maintain their focus instructional goal related to an identified ID project and on desired processes and outputs (Woolf & Quinn, provides it to a fellow student for review. Each student 2001). provides written feedback according to the criteria The next section will provide two examples from given for sound instructional goals and debriefs his/her the current practice of two IDT instructors at different partner that same night in class. higher education institutions. The first example details This first peer review assignment is meant to be the use of peer review in a face-to-face instructional simple, structured, and monitored by the instructor in design course. The second scenario describes its use in order for students to experience low-threat practice with an online instructional design course. Although the peer review as well as have an opportunity to get to professional context for each course is the same, peer know one another better. Students are debriefed about review, as reflected in the literature review of its the peer review experience and reminded that they will benefits, is relevant to other professional disciplines use peer review in varied forms throughout the rest of requiring group problem solving. In fact, Topping the semester. Students are somewhat shy about (1998) analyzed the use of peer assessment in group providing feedback to one another during this first peer work across such varied disciplines as math modeling, review assignment. The assignment’s simplicity, business administration, speech communications, structure, and rules of engagement appear to ease this psychology, microclimatology, and engineering design. anxiety. The face-to-face setting is advantageous in that Further, in their meta-analysis of peer assessment in the instructor can closely monitor students’ reactions higher education, Falchikov and Goldfinch (2000) and experiences, providing guidance and found that “peer assessment can be successful in any encouragement as needed. discipline area and at any level” (p. 317). By week three, students are grouped into teams of three to four and assigned one real-world instructional Peer Review in a Face-to-Face design project to work on for the rest of the semester. Instructional Design Course The goal for each team is to develop an instructional unit that meets the identified needs of the project. Each The first author has been using peer review in an week, students work through an iterative process instructional design and other IDT and teacher whereby they read about a new ID core concept, e.g. education courses since 2002. Over that time, the peer learner analysis, content analysis, etc., outside of class review strategy has evolved in scope, structure, and and receive instruction on that concept in class to draw depth based on student responses to the process, out critical elements, explore examples, and practice literature findings, as well as instructor reflections on application of the concept. Then, in their teams, they practice. This section will describe the nature of peer draft the relevant portion of the instructional design for review as of the Fall 2009 instructional design course their team project, submitting it for peer review by Brill and Hodges Peer Review as a Learning Strategy 116 members of another team the following week during 2009. The class is part of an online M.Ed. program in class. The structure of these weekly formative reviews is Instructional Technology offered through a regional less formal, although concept-relevant rubrics are comprehensive university in the southeastern United provided as an additional means of support for States. The students were enrolled in a course titled knowledge and skill development. The challenge for the Instructional Design. Each section had an enrollment of instructor is to encourage students to refer back to their 25 students, and the students were distributed widely support materials in conducting these reviews, as well as across a large state in the southeastern United States. to mitigate any conflicts that may arise. Students respond Most of the students were practicing K-12 educators. positively to these reviews, noting that they often benefit The course was offered in a completely asynchronous from perspectives outside of the team as well as see format. things in the work of other teams that they can bring back As part of the class, a learner analysis paper was to their teams to improve the work to-date. assigned during the fourth week of the 15-week A final, more formal and extensive, graded peer semester. Students in the course were required to review occurs over weeks 12 and 13 of the semester. complete a detailed learner analysis and were provided By this time, each project team has a complete draft of with assignment details and the scoring rubric. The their instructional unit that has been subject to the students were given 13 days to complete the weekly formative evaluations. At week 12, the team assignment. Part of the assignment included submits one full copy of their unit, via a project web participation in a blind peer review process, which site, to three to four individual peer reviewers chosen consisted of two steps, prior to submitting the paper to by the instructor. Each peer reviewer is provided with the instructor for evaluation. Individuals posted their instructions and rubrics for completing the review and learner analysis papers (step 1) and provided feedback has one week’s time to complete the review outside of to one other student's posting (step 2). class and provide electronic copy back to the authoring Eight days were scheduled for the students to team and the instructor. By this time, students are write their papers and post them for review. The comfortable with one another, with peer review, and students posted their papers to an anonymous with the nature of the projects. They comment regularly discussion forum in the course management system. that this more extensive peer review is one of the most Students were asked to include a pseudonym in the valuable assignments in the course, forcing them to re- subject line of their posting, and to communicate the engage with core principles and concepts explored pseudonym to the instructor using email. Two days during the semester at a deeper level in order to provide were allotted for the review element of the peer useful ID feedback to another team on a project that review process. Students were instructed to select one they understand themselves has become “near and paper to read and on which to provide feedback in the dear” to the team. During the week 13 class period, the discussion forum. The identity of the reviewers was instructor debriefs students on their experiences with not available to the students receiving feedback. The this assignment, asking them to reflect on what the instructor suggested that the scoring rubric for the authoring team members gained from the review in assignment be used to structure the feedback. terms of improving their instructional designs and what Additionally, students were directed to be "critical and the peer reviewers gained in terms of ID knowledge and constructive, but polite." skill development. At the conclusion of the peer reviewing experience, The intent going forward is to continue peer review students were instructed to revise their papers based on in this course and conduct research to investigate the the peer reviewer feedback and to include a section at role of peer review in a face-to-face instructional design the end of the paper explaining the changes initiated by course. Anticipated outcomes of the research include the peer reviewing process. Three days were scheduled reporting impact on student learning and providing after the review period for revisions and final guidelines for the effective application of peer review in submission of the assignment. the development of instructional design and other The practice described here is part of an emerging professionals-in-training who must engage in group research program aimed at investigating the use of peer problem construction, collaboration, and resolution as review in online Instructional Design courses. The part of professional practice. general focus of the research program is to develop a set of empirically grounded best practices for using peer Peer Review in an Online review in online instructional design courses. Feedback Instructional Design Course from this pilot project indicates that students appreciated the process and the opportunity to learn The second author used a structured peer reviewing from their classmates using peer review. The instructor process for an assignment in his two sections of an was not overburdened with logistical or technical online Instructional Design course during the fall of matters facilitating the process. A next step is to study Brill and Hodges Peer Review as a Learning Strategy 117 whether or not the peer reviewing process improves the Bloom, B. S., Krathwohl, D. R., & Masia, B. B. (1956). quality of the work submitted by the students. Taxonomy of educational objectives: The classification of educational goals. New York, NY: Concluding Remarks and Directions for Research D. McKay. Brill, J. M., Bishop, M. J., & Walker, A. (2006). An Peer review as an instructional strategy for investigation into the competencies required of an developing instructional design and technology effective project manager: A web-based Delphi professionals has the potential not only to support study. Educational Technology Research and professional standards but also to address ongoing Development, 54(2), 115-140. concerns regarding the inadequate preparation of Brown, J. S., Collins, A., & Duguid, P. (1989). Situated instructional design and technology professionals. Peer cognition and the culture of learning. Educational review can support the need for instructional design Researcher, 18(1), 32-42. students to understand real-world instructional design Casey, C., Branvold, D., & Cargille, B. (1996). A practice as non-linear, complex, and demanding cross- model for peer review in instructional design. functional collaborative problem-solving and Performance Improvement Quarterly, 9(3), 32-51. management skills (Brill, Bishop, & Walker, 2006; Chapman, K. J., & van Auken, S. (2001). Creating Casey, et al., 1996; Woolf & Quinn, 2001). Similar positive group project experiences: An examination concerns articulated by other disciplines can be of the role of the instructor on students' perceptions addressed with peer review as well (Maleki, 2009; of group projects. Journal of Marketing Education, Queeney, 1996). 22(2), 117-127. Our review of the literature draws out some Dalgarno, B. (2001). Interpretations of constructivism important benefits of peer review as a promising higher and consequences for computer assisted learning. education pedagogy, particularly for those adult British Journal of Educational Technology, 32(2), students being mentored into a new profession that 183-194. demands collaborative problem posing, reflection, and Davies, P. (2000). Computerized peer assessment. resolution. Peer review has been shown to promote the Innovations in Education and Training recognition of good practice as well as critical and International. 37(4), 346-355. constructive collaborative dialogue. The cases Dick, W., Carey, L., & Carey, J. O. (2009). The presented here suggest that peer review can been systematic design of instruction (7th ed.). Upper integrated into the higher education classroom Saddle River, NJ: Pearson. effectively and can benefit from intentional literature- Falchikov, N., & Goldfinch, J. (2000). Student peer based strategies such as clear feedback criteria and assessment in higher education: A meta-analysis blind review, but they only do so anecdotally. Thus, our comparing peer and teacher marks. Review of next steps are to conduct empirical research in both Educational Research, 70(3), 287-322. face-to-face and online settings to investigate learning Harris, M. J. (2006). Three steps to teaching abstract outcomes and instructional strategies. Our research and critique writing. International Journal of plans respond to the advocacy of scholars and Teaching and Learning in Higher Education, practitioners for more research and models to better 17(2), 136-146. understand peer review as an intentional learning ISTE (2007). National Educational Technology strategy for adult learners (Casey, et al., 1996; Standards and Performance Indicators for Students Falchikov & Goldfinch, 2000; Woolf & Quinn, 2001). Retrieved September 18, 2009, from Ongoing scholarship among higher education http://www.iste.org/Content/NavigationMenu/NET professionals offers an important venue for dialogue S/ForStudents/2007Standards/NETS_for_Students about peer review as an opportunity for advancing _2007_Standards.pdf instructional practice, research, and better professional Ku, H.-Y., & Lohr, L. L. (2003). A case study of preparation for real-world practice. Chinese students' attitudes toward their first online learning experience. Educational Technology References Research and Development, 51(3), 95-102. Lai, M., & Law, N. (2006). Peer scaffolding of AECT. (2001). Association for Educational knowledge building through collaborative groups Communications and Technology Retrieved from with differential learning experiences. Journal of http://www.aect.org/standards/initstand.html Educational Computing Research, 35(2), 123-144. Bangert, A. W. (2001). Peer assessment: A win-win Li, L., & Steckelberg, A. L. (2004). Peer assessment instructional strategy for both students and support system (PASS). TechTrends, 49(4), 80-84. teachers. Journal of Cooperation & Collaboration Livingstone, D., & Lynch, K. (2000). Group project in College Teaching, 10(2), 77-84. work and student-centered active learning: Two Brill and Hodges Peer Review as a Learning Strategy 118 different perspectives. Studies in Higher Woolf, N. H., & Quinn, J. (2001). Evaluating peer Education, 25(3), 325-345. review in an introductory instructional design Maleki, R. A. (2009). Business and industry project- course. Performance Improvement Quarterly, based capstone courses: Selecting projects and 14(3), 20-42. assessing learning outcomes. Industry and Higher ____________________________ Education. 23(2), 91-102. Millis, B. J., & Cottell, P. G., Jr. (1998). Cooperative JENNIFER M. BRILL is an assistant professor in the learning for higher education faculty. Phoenix, Learning Sciences and Technologies department at AZ: Orxy Press. Virginia Tech. Jen earned a Ph.D. in Instructional Ozogul, G., Olina, Z., & Sullivan, H. (2008). Teacher, Technology from the University of Georgia, an M.Ed. self and peer evaluation of lesson plans written by in Instructional Systems from Pennsylvania State preservice teachers. Educational Technology University, and a B.S. in biology from Boston College. Research and Development, 56(2), 181-201. She has worked as an instructional design and Ozogul, G., & Sullivan, H. (2009). Student performance technologist since 1986. Her research performance and attitudes under formative interests include systemic change in professional work evaluation by teacher, self and peer evaluators. communities, instructional design, and learning and Educational Technology Research and performance improvement. Development, 57(3), 393-410. Queeney, D. S. (1996). Redefining competency from a CHARLES B. HODGES has nearly 20 years of systems perspective for the 21st century. experience with technology-based instruction. He Continuing Higher Education Review. 61, 3-11. earned a Ph.D. in Instructional Design and Technology Topping, K. (1998). Peer assessment between students from Virginia Tech and mathematics degrees from in colleges and universities. Review of Educational Fairmont State College (B.S.) and West Virginia Research, 68(3), 249-276. University (M.S.). His professional interests are self- Towns, M. H., Kreke, K., & Fields, A. (2000). An efficacy and self-regulation in online learning action research project: Students perspectives on environments, instructional software evaluation, and the small-group learning in chemistry. Journal of preparation of instructional designers. He is an assistant Chemical Education, 77(1), 111-117. professor of Instructional Technology at Georgia von Glasersfeld, E. (1995). Radical constructivism: A Southern University where he teaches in the online way of knowing and learning. Washington, DC: Instructional Technology Program. Formerly, he was Falmer Press. the manager of Virginia Tech's Advanced Technology Wenger, E. (1998). Communities of practice: Learning, Learning Center, commonly known as the Math meaning, and identity. New York, NY: Cambridge Emporium. University Press.