University of Washington Office of Educational Assessment Civil and Environmental Engineering (CEE) 220: Anytime Anywhere Final Evaluation Report Bayta L. Maring and Joanna Burgess August 2005 TABLE OF CONTENTS Executive Summary....................................................................................................................................3 Background.................................................................................................................................................7 Methods........................................................................................................................................................8 Results.......................................................................................................................................................10 Student Performance..............................................................................................................................10 Student Satisfaction................................................................................................................................13 Specific Course Innovations...................................................................................................................16 Online Moodle System.......................................................................................................................16 Online Reading Quizzes.....................................................................................................................17 Online Homework Grading.................................................................................................................19 Extra Out-of-Class Help.....................................................................................................................20 Alternate Lab Time.............................................................................................................................23 Suggestions for Improvement.................................................................................................................25 Additional Ideas for the Future................................................................................................................27 Evaluator comments and Recommendations........................................................................................28 Resource Re-Allocation..........................................................................................................................29 Student Experience................................................................................................................................31 Adaptability.............................................................................................................................................32 Appendix A: Detailed Findings from Mid-term and final surveys.......................................................34 Appendix B: Detailed Findings from the Student Interviews..............................................................48 Appendix C: Detailed Findings from the Course Evaluations.............................................................60 CEE Anytime Anywhere Final Evaluation Report 2 EXECUTIVE SUMMARY This report describes evaluation findings from a redesign of Civil and Environmental Engineering (CEE) 220 Mechanics of Materials. The goals for this project were to re-allocate course resources, including lab space and instructor/TA time without any decline in student performance or satisfaction. To this end, the Winter 2005 version of CEE 220 included the following innovations: • Use of an integrated online course system, called Moodle, that centralized and organized class materials, assignments, and activities • Online pre-lecture quizzes, implemented through the Moodle system, that were designed to prepare students for lecture above and beyond the reading itself • Homework assignments that were completed and immediately graded online, also via the Moodle system • Extra out-of-class help, including a weekly session at the Center for Learning and Undergraduate Enrichment (CLUE), specific tutoring times for CEE 220 at the Engineering Advising and Students Center (EASC), and an online discussion board (also a part of the Moodle system) • An option to attend weekly labs outside of the regularly scheduled sessions To assess the effectiveness of these changes in regards to the project goals, the Office of Educational Assessment was hired to conduct an evaluation of the new CEE 220 course. The following evaluation activities were conducted as part of this evaluation: • Student surveys administered at the middle and end of the term • Interviews with 12 CEE 220 students • Interviews with the instructor and TAs for the course • Comparison of student performance data (grades and exam scores) and standard UW course evaluation data between CEE 220 in Winter 2005 and Winter 2004, when it was taught by the same instructor Summary of Findings Student Performance • In terms of overall grades, there were no significant differences between CEE 220 in Winter 2005 and 2004, although an analysis of exam scores revealed that students in 2005 scored significantly worse than 2004 students on the second mid-term and final. However, because the exams were somewhat different across years, it is unclear whether they were appropriate assessments of student learning. • According to their own reflections, the instructor and TAs suggested that students’ understanding of the material was deeper in this class than they had seen in the same or similar classes. Student Satisfaction • Overall, student response to the new version of CEE 220 was overwhelmingly positive, and was, on the whole, more positive than the same course taught by the same instructor in Winter 2004. CEE Anytime Anywhere Final Evaluation Report 3 • The instructor’s commitment to the course and concern for student learning was highlighted by a large number of students as something that made the course unique and a positive experience. • Compared to students in CEE 220 in 2004, students in Winter 2005 generally reported devoting more hours to this course and that a larger proportion of these hours was valuable. Moodle System in General • For the most part, students responded positively to the integrated online course content, although there was a small number of individuals who had generally negative reactions to Moodle and the course in general. • The TAs and instructor were very pleased with the time they saved by using the Moodle system to handle administrative aspects of the course such as assignment grading and grade recording. Online Reading Quizzes • According to self-report, the majority of students (68.8%) completed all of the pre-lecture reading quizzes and all of them completed at least three-quarters of the quizzes. • Data indicated that at the beginning of the term, students completed the quizzes after they had done the reading, but by the end of the term, they would look at the quiz before or while they were reading. • Students saw three primary benefits of the reading quizzes: (1) highlighting important points in the reading; (2) motivating them to complete the reading; and (3) showing them what they do not understand from the reading. • Although students, in general, did not strongly suggest that the pre-lecture quizzes enhanced their learning, the TAs and instructor believed they did. The instructor suggested that the quizzes might have helped students learn, though repetition and exposure, without them realizing it. • Student satisfaction with the reading quizzes was extremely varied, with some students considering them an important part of their learning experience and others considering them a nuisance. Online Homework Grading • There was consensus among students, TAs and the instructor that the primary benefit of the online homework grading was that students received immediate feedback about their responses, making the assignments a learning tool as opposed to an evaluation. • Students appreciated the opportunity to attempt the homework multiple times, which also reinforced the idea of homework as a learning tool. • Online grading was a welcome change to the TAs because it freed up time to spend on the more rewarding activities of teaching and working with students. Extra Out-of-Class Help • In general, students made use of the out-of-class help to assist them in completing their homework assignments. • In addition to helping students with their homework, the CLUE sessions benefited both students and instructor by allowing them to interact more informally than in lecture. CEE Anytime Anywhere Final Evaluation Report 4 • Tutoring at the EASC was appreciated by most of the students who attended, though all of these students thought the room should be larger and that adding another TA or two would be helpful. • The discussion board was active for this class and students appreciated it as a 24-hour source for help. The instructor and TAs were minimally involved in the discussion board and were impressed at the extent to which students helped each other. Alternate Lab Times • Only one group of students opted to have their lab session at an alternate time (9:30am instead of 8:30am). • Although many students appreciated the opportunity for flexibility, almost half reported that the time they had originally scheduled “was perfect.” • Students who would have preferred another time but did not reschedule either could not find a mutually agreeable time with their group or had committed to other activities that made it difficult to change the lab time. Suggestions for Improvement Across all data sources, participants proposed a wide variety of suggestions for improvement. Below is a brief summary of those mentioned most frequently; a more thorough list is provided in the main report. • Overall, the strongest suggestion was to obtain a faster server for the Moodle system, which has already been done. • Another strong suggestion was to increase the amount of practical examples and applications in both lecture and lab. • To improve the reading quizzes, the strongest student suggestions were: (1) make the due dates consistent; (2) provide additional hints for incorrect answers; and (3) save answers for each re- attempt (note that the instructor thought that compelling students to remember answers for each re- attempt enhanced learning). • In regards to the online homework assignments, students made the following suggestions: (1) provide additional hints, as in the Tycho system used by the Physics department; (2) make it easier to re-submit questions; and (3) do not require submission of an additional paper copy of the homework. • Ideas for encouraging students to attend alternate lab times included: (1) offering additional lab times; (2) organizing groups according to availability; and (3) establishing a full, unstructured day for lab sessions. Evaluator Comments and Recommendations Based on an interpretation of the data, the evaluator drew several conclusions and made a few specific recommendations. Below is a brief description of these more expansive thoughts. • The Moodle system afforded a number of opportunities for re-allocating resources. In particular, online homework grading, discussion board, and online grade book saved a considerable amount of TA and instructor time. CEE Anytime Anywhere Final Evaluation Report 5 • These time saving aspects of Moodle (homework grading, discussion board, grade book) might be the strongest “selling points” for future instructors. • If Moodle is used more widely, consideration should be made for students who do not have easy access to a computer or the internet. • One option for the alternate lab space re-allocation would be either a dedicated “lab day” or a completely independent lab assignment, so that student groups could complete the sessions on their own with available online assistance. • To encourage students to learn from each other, lab groups could be encouraged to also be study groups or teams, though this might require a linear grading structure and a reward structure to motivate students to work in teams. • The Program in Course Redesign, funded by the Pew Charitable trust might provide valuable ideas and insight into how to re-allocate resources in this course. • Holding a CLUE session, perhaps in lieu of an instructor’s office hour, is a valuable and “low cost” (in terms of instructor time) way to provide additional assistance to students and potentially increase capacity of the course while still giving students a valuable learning experience. • If the reading quizzes are implemented in future courses, certain changes, such as consistent due dates, should be made to these assignments to maintain student satisfaction with the course. • It is unclear, at this point, how these innovations affected student learning. More controlled follow-up studies, might provide more accurate information about any longer-lasting effects on how well students absorbed and retained the material from CEE 220. • In general, the primary challenge for this course redesign will be adaptability. The instructor for this version of CEE 220 was uniquely committed to the course and concerned about student learning, which are qualities that cannot be disseminated. • Acknowledging that changes will happen slowly and beginning with small cost (in terms of time) changes would be advisable. • If TAs are relied upon to carry out some of these changes, these responsibilities should be acknowledged as part of their time commitment to the course. • Providing support for new Moodle users, perhaps by assigning a person in the department or college as a “Moodle expert,” might facilitate the adoption of this potentially resource-saving system. CEE Anytime Anywhere Final Evaluation Report 6 BACKGROUND Civil and Environmental Engineering (CEE) 220 Mechanics of Materials is an introductory engineering course required for undergraduate degrees in five of the ten departments comprising the University of Washington’s College of Engineering. In Winter 2005, a redesign of this course was implemented as part of the “CEE Anytime Anywhere” project. The purpose of the redesign was to re-allocate course resources, including lab space, instructor time, classroom time, and graduate teaching assistants’ time, without a loss in students’ performance or satisfaction. Towards this end, the following changes were made to the course: • Use of online Moodle system. Moodle is open source software designed to provide an easy-to-use interface for both students and instructors for organizing and delivering course content in an online environment. For CEE 220, the Moodle system included announcements from the instructor, general information for the class (e.g., syllabus, requirements), listing of all assignments, lecture notes and handouts, quiz keys, a student discussion forum, an online grade estimator, and other resources. Providing this information online was intended, in part, to assist the instructor in communicating with students about course materials. There were two changes implemented on the Moodle system that were of particular interest in this project, pre-lecture quizzing and online homework grading, which are described below. • Pre-lecture reading quizzes. These brief quizzes were implemented through the Moodle system and contained several questions (some multiple choice, some short answer) about the reading assignment for the upcoming lecture. Students received a small amount of course credit for completing these assignments in a timely manner. The goal of these assignments was to increase the amount of out-of-class learning students did in preparation for lecture, thus increasing the value and/or impact of in-class time. • Online homework grading. All homework assignments for CEE 220 in Winter 2005 were submitted online. Students also handed in paper copies of their assignments for the purpose of validation and as a back-up in case of online system failure. The paper assignments were not graded. The purpose of this innovation was to conserve TA time so that more time could be dedicated to direct interactions with students. • Increased availability of out-of-class help. One unique aspect of this course was a special session held on Monday evenings at the Center for Learning and Undergraduate Enrichment (CLUE). These informal review sessions were led by the instructor and primarily covered students’ questions from the upcoming week’s homework assignments. In addition to the CLUE session, the instructor led an out- of-class review session before each exam and the Engineering Advising and Student Center (EASC) held tutorial sessions on Tuesdays and Thursdays specifically for CEE 220. The purpose of this increased out-of-class help was to encourage students to work together in groups on course material and questions outside class, thereby increasing the value of in-class time. • Option of alternate lab times. In an effort to redistribute lab space resources, the experimental version of CEE 220 included an option for participants to attend a lab session outside of the regularly scheduled lab times. The original project proposal included a provision for undergraduates to be paid CEE Anytime Anywhere Final Evaluation Report 7 assistants during this lab time, with TA’s also available for help. This option was presented three weeks into the quarter, after participants had already formed lab groups based upon their birthdays. Only one group opted to have their lab session one hour later than scheduled (9:30am instead of 8:30am), which overlapped with an existing lab when TA’s were present. Hence, no undergraduate helpers were needed. The University of Washington’s Office of Educational Assessment (OEA) was asked to conduct an evaluation of the revised version of CEE 220 with a focus on (a) documenting impact on student performance and student satisfaction; (b) exploring both students’ and instructors’ experiences with each of the specific innovations; and (c) gathering suggestions for further improvement. The following methods were used to collect relevant information: • Student Surveys: A brief online survey that focused on the course innovations was administered in the middle of the term and a similar paper survey was administered during one of the final class sessions. • Student Interviews: In Spring 2005, twelve of the 88 students enrolled in the course participated in interviews with the evaluator. • Instructor Interviews and Diary: The two teaching assistants and the instructor were interviewed in Spring 2005. The instructor also kept a diary of his experience in the course, which provided additional data. • Student Performance Data: Exam scores and final grades from the experimental CEE 220 taught in Winter 2005 and the traditional version of the course taught in Winter 2004 by the same instructor were analyzed and compared. • Course Evaluation Data: Standard UW course evaluation data from CEE 220 in Winter 2005 and Winter 2004 were analyzed and compared, including both numerical ratings and students’ comment sheets. METHODS Below is a summary of the participants and procedures for each of the methods used in this evaluation project. Student Surveys Mid-Term Survey An email was sent to all 88 students in CEE 220 during the sixth week of the term inviting them to participate in the online survey. The instructor also made announcements in class about the survey, emailed students encouraging them to participate, and added a link to the survey in one of the online reading quizzes. CEE Anytime Anywhere Final Evaluation Report 8 A total of 42 students (46.7%) completed the online survey administered in the middle of the term. No demographic information was collected from participants. Results form the mid-term survey were summarized in a brief report that was submitted to the instructor and project lead in February 2005. All of the questions from this survey are included in Appendix A. Final Survey A pencil-and-paper survey was administered to all students who attended the penultimate class session. The survey contained many of the same questions as the online survey administered in the middle of the term. The instrument was distributed and collected simultaneously with the administration of the standard UW course evaluations. The survey was completed by 48 students (54.5%); no demographic information was collected. All of the questions from this survey are included in Appendix A. Student Interviews Early in the quarter after they had completed CEE 220 (Spring 2005), all 88 students who had taken the course were sent an email message inviting them to participate in a 90-minute focus group. Refreshments were offered as an incentive. The instructor sent one email encouraging students to participate. Due to lack of response, a subsequent email was sent to the same list of students, inviting them to take part, instead, in individual interviews at a time convenient for them. The incentive for participation was a 128GB flash drive for the first ten interviewees. A total of twelve students (three female and nine male) participated in this aspect of the evaluation (two of them consenting to participate without incentive). Ten individuals were interviewed in person at various locations on the UW campus (primarily in the Husky Union Building); three of these participants were interviewed together as a group. An additional student was interviewed by phone. During all of these interviews, the evaluator took type-written or hand-written notes. An audio-recording was made of all of the in-person interviews for the purpose of accuracy of quotes and clarification. One additional participant responded to the interview questions via email. All of the questions from the interview protocol are included in Appendix C. Instructor Interviews and Diary Two teaching assistants and the instructor were interviewed in person, with the two TAs interviewed as a group. Sessions were audio-recorded for accuracy and clarification. In addition, the instructor kept a class diary, that included weekly entries tracking topics covered and noting some observations about classroom sessions, out-of-class help sessions, and classroom innovations. Student Performance Data Exam and final grade data from CEE 220 in Winter 2005 and Winter 2004 were obtained from the instructor, who had taught both courses. Students who missed two or more exams were not included in the analysis resulting samples of 83 students for 2004 and 85 students for the experimental class in 2005. CEE Anytime Anywhere Final Evaluation Report 9 Course Evaluation Data Data from two standard UW course evaluation forms were used in evaluation of this project. The first was a scannable form with 31 numerically rated items about overall class satisfaction, class structure and organization, quantity and quality of content learned, and required workload/effort for the course. In addition, the back of the form included 11 questions (again, all numerical ratings) adapted from the Accreditation Board of Engineering and Technology (ABET)’s program outcomes for baccalaureate level programs. The second form included one forced-choice question and four open-ended questions. Evaluation data from these forms for the Winter 2005 CEE and the Winter 2004 CEE 220 courses were analyzed and compared. In 2004, 48 (57.1%) students returned the scannable form, and 42 (50%) submitted open-ended comment sheets. In 2005, 46 students (52.3%) returned scannable forms and 48 (54.5%) returned open-ended comment sheets. RESULTS This section includes a synthesis of all the data used in this project, organized according to evaluation topic: (a) student performance; (b) student satisfaction; (c) specific course innovations; (d) suggestions for improvement; and (e) additional ideas for the future. Please see the appendices for a more detailed, comprehensive and descriptive analysis of the student surveys (Appendix A), student interviews (Appendix B), and course evaluation data (Appendix C). Student Performance One of the goals of this project was to experiment with innovations that might re-allocate resources while not affecting student performance. . As described above, several students were not included in grade analyses because they had missed two or more exams. Two of 85 students enrolled in the 2004 and five the original 90 enrollees in 2005 fit this criterion. This difference in drop rate was not statistically significant.1 Figure 1 shows differences in grades across the three exams in 2004 and 2005.2 Note that the exams student took in 2004 and 2005 were similar, but not identical. In fact, the final exam in 2005 was worth 100 points while in 2004 it was worth 80. 1 According to a chi-square test of association, p > .05. 2 Note that these scores do NOT include “0” scores for students who did not complete an exam. Similarly, the grades shown in Figure 2 do not include grades from students who missed two or more exams. CEE Anytime Anywhere Final Evaluation Report 10