Evaluation of the Appalachian Regional Commission Oak Ridge National Laboratory Summer Institute for Math/Science/Technology VMarch 2006 Evaluation of the Appalachian Regional Commission Oak Ridge National Laboratory Summer Institute for Math/Science/Technology March 2006 Evaluation of the Evaluation of the Appalachian Regional Commission Appalachian Regional Commission Oak Ridge National Laboratory OaSkum Rmeird Ingsteit uNte faort Mioanth/aScli eLncae/Tbecohrnoalotgoyry Summer Institute for March 2006 Math/Science/Technology March 2006 Prepared by Linda S. Simkin, M.Phil. Valerie Futch, B.S. Prepared for Appalachian Regional Commission 1666 Connecticut Avenue, NW Suite 700 Washington DC 20009 Acknowledgements Many people contributed to this evaluation and report. We would like to begin by thanking Catherine Saldutti, president of EduChange in New York City and valued expert consultant for this project. Ms. Saldutti provided knowledgeable guidance about STEM programs for students and teachers as well as about the current state-of-the-art in evaluating these programs. Alexandra Weinbaum, vice president and director of AED’s Center for School and Community Services also provided expert guidance throughout the evaluation and valuable insights about national efforts to improve college-access and success for underrepresented populations. In addition, we would like to thank members of AED’s staff for the technical expertise they provided on this evaluation. Eliana Orellana and Wanda Dallas provided assistance with database development and data entry, and Adria Gallup-Black conducted the statistical analyses. Elayne Archer, who was responsible for editing this report, brought to it both her deep knowledge of educational assessment and her keen attention to language. We would also like to acknowledge the ARC- ORNL participants who took the time to complete surveys and provide thoughtful feedback in interviews at the time of the year when holidays and end-of-semester assignments competed for their attention. Finally, we thank Jeffrey Schwartz for his ongoing and knowledgeable assistance. The site visit to the ARC-ORNL Summer Institute in 2005 that he arranged for the AED evaluator was especially valuable. The background he provided, as well as meetings with staff of the ORNL and the Oak Ridge Institute for Science and Education he arranged, informed the evaluation. We appreciated being able to have informal conversations with the 2005 participants and were impressed by the palpable enthusiasm of students, teachers, chaperones and mentors. Linda Simkin, M.Phil. Valerie Futch, B.S. Academy for Educational Development New York City March 2006 Highlights This report describes some of the key immediate and long-term outcomes achieved by the Appalachian Regional Commission (ARC)–Oak Ridge National Laboratory (ORNL) Summer Institute for Math/Science/Technology for student and teacher participants. This two-week summer program provides high school students and teachers from the Appalachian region the opportunity to work with mentor scientists from ORNL on inquiry-based, applied projects in science, math, and computer technology. The institute culminates with group, student and teacher, presentations about their projects. Other planned activities are designed to promote teamwork, expose students to college opportunities, and promote pride in the cultural richness and historical importance of the Appalachian region. The goals of the Summer Institute, in operation since 1990, are to1) encourage more high school students to continue their studies beyond high school, 2) encourage more students to pursue careers in the projected shortage areas of science, technology, engineering, and math (STEM), and 3) raise the level of math, science, and technology instruction in high schools throughout the region to facilitate the first two goals. The findings in this report are the result of an evaluation conducted by staff of the Academy for Educational Development, in which data were collected from eight cohorts of participants attending the Summer Institute between 1997 and 2004. The evaluation took place in 2005-06 and is based on surveys from 89 students and 67 teachers, as well as interviews with a small number of participants. The evaluation has several limitations, including absence of a comparison group that would have made it possible to attribute outcomes solely to participation at the institute. Also, this evaluation was able to collect data only after participation in the institute, whereas with a carefully constructed pre-post comparison, the evaluation could have measured changes in student/teacher knowledge, attitudes, and intentions. Key findings for students and teachers follow. Students • Male and female students from each of the 13 states in the region participated in the Summer Institutes, and 31% of students came from counties designated as economically distressed by ARC. • Seventy-four percent of students said they felt more confident in their STEM abilities as a result of the institute. • Participation in the Summer Institute influenced 24% of students to take more science classes and 22% to take more math classes when they returned to high school. Somewhat more than half the students reported that their experience at ORNL reinforced prior decisions about the science and math courses they had already chosen to take (56% and 52%, respectively). • All the student-survey respondents reported that, even prior to attending the Summer Institute, they planned to attend college. Even so, the students reported many ways that the institute reinforced their intentions to go to college and reduced some of the barriers. More than 50% of students reported that the program positively influenced their intentions to go to college. • Fully 96% of student participants who had graduated from high school at the time of the survey had continued their formal education. Of the 83 respondents who provided data, a great majority (79%) went to four-year institutions; 15 % went to community colleges or technical centers; 2% to military academies; and 3% did not continue their education beyond high school. • Of the 23 students who attended the institute in summer 1997 and 1998, all reported having attained higher education—26% had some college, 39% had earned a bachelor’s degree, and 35% went on to graduate work. • After the Summer Institute, 36 students went on to earn a total of 46 degrees from higher education institutions. Overall, 54% of degrees earned were in STEM—38% of associate degrees, 52% of bachelor degrees, and 86% of graduate degrees. • In total, 51 student-survey respondents indicated that they were currently pursuing post- high school education; 82% were pursuing degrees in STEM fields, including core subjects, health, and computer sciences. • Fifty-five percent of participants who were employed full-time reported that their jobs required proficiency in one of the STEM fields “to a great extent.” • The highlight of the experience named by students most often was interacting with people, especially their peers, from different regions of the country and meeting the ORNL scientists. • Many students reported that the Summer Institute experience helped them gain more mature perspectives socially and academically, an observation that was corroborated by teachers. According to many students, it was particularly significant to become aware that “there were others like me.” • Fifty-nine percent of students currently reside in the Appalachian region, and 53% anticipated residing in the region five years hence. There was no difference in intention to remain in the region among those who had majored in STEM or were currently majoring in STEM. Teachers • Teachers from each of the 13 states in the Appalachian region participated; 47% of teacher participants were from ARC-designated distressed counties. • Many teachers reported that they incorporated activities and approaches learned at the Summer Institute into their classrooms. When they returned to their classrooms, 77% drew on their experience at the institute for explanations and examples, 52% did so for classroom demonstrations, and 50% incorporated new knowledge in their lab experiments. • Eighty-six percent of teachers said that, following the institute, they encouraged students to continue their education in science, math or technology. • Sixty-nine percent reported talking to students in their classrooms about research applications in the STEM areas. • Seventy-five percent reported that they shared their experience and things they had learned with other teachers through informal conversations. Four teachers who gave formal presentations to colleagues estimated reaching a total of 95 teachers. • Fifty-seven percent of teachers said the institute encouraged them to seek more professional development in STEM. • Of the benefits of attending the Summer Institute named by teachers, they most often described the opportunity of meeting and networking with other teachers from the region. • Many interviewed teacher participants desired a more effective professional development component that would provide them with lesson plans and resources to better integrate their ORNL experience into their own classrooms. • Ninety-one percent of teachers currently reside in the Appalachian region, and 93% said they planned to reside in the region five years hence. Based on the findings, the evaluation report offers recommendations to help the institute fulfill its purpose more deliberately and effectively. The first recommendations listed below address the program overall, and then address specific student and teacher components of the program. Overall Program • Clarify recruitment criteria and process for students and teachers and make these criteria known to participants as well as the groups that recruit them. • Increase the number of youth from underrepresented groups in STEM. • Create connections with college-access programs in the region to expand the applicant pool and support student success over the long term. • Continue to conduct a mixed-method evaluation using a design that incorporates data collection before, and at two points following, participation in the Summer Institute. This will necessitate better student tracking. Student Components • Match student interests with projects or, if this is not possible, provide students with the opportunity to sample many projects. • Provide opportunities for students to meet or work with graduate students and young professionals. • Foster peer communication during and after the institute, both within and among different cohorts. • Create connections between the Summer Institute, sending school, and parents to support participants’ progress when they return home. Teacher Components • Give more attention to curricular issues by providing more structured time for teachers to discuss both specific curricular implications as well as ways to continue to promote interest in STEM education and careers among their students. • Foster networking among teachers from multiple years of the Summer Institute. • Make explicit expectations that teachers share their experience with other teachers in their schools and districts. Table of Contents 1. Introduction...............................................................................................................................1 1.1 Background...........................................................................................................................1 1.2 Description of the Appalachian Region................................................................................2 1.3 Description of the Summer Institute.....................................................................................3 2. Evaluation of Other High School STEM Enrichment Programs.........................................5 2.1 Evaluation Approaches.........................................................................................................5 2.2 STEM Evaluation Results.....................................................................................................6 2.3 Resources..............................................................................................................................8 3. Evaluation Methodology..........................................................................................................9 3.1 Approach and limitations......................................................................................................9 3.2 Student and Teacher Surveys..............................................................................................10 3.3 Student and Teacher Interviews..........................................................................................10 3.4 Study Population.................................................................................................................10 3.5 Response Bias.....................................................................................................................11 4. Student Findings.....................................................................................................................14 4.1 Description of Student Respondents...................................................................................14 4.2 Overall Assessment of the Summer Institute......................................................................14 4.3 Influence of Peer Support...................................................................................................16 4.4 Influence of Mentors, Chaperones, and Teachers...............................................................17 4.5 Changes in Attitudes about School and Subjects Studied..................................................17 4.6 Pursuit of STEM Education and Employment....................................................................19 4.7 College Aspirations and Planning.......................................................................................21 4.8 College Enrollment.............................................................................................................23 4.9 Workforce Participation......................................................................................................26 4.10 Student Recommendations................................................................................................28 5. Teacher Findings.....................................................................................................................30 5.1 Teacher Respondents..........................................................................................................30 5.2 Overall Assessment.............................................................................................................32 5.3 Impact on Teaching.............................................................................................................33 5.4 Barriers to Bringing ORNL Experiences to Classrooms....................................................36 5.5 Pursuit of Professional Development..................................................................................37 5.6 Impact on Other Teachers...................................................................................................37 5.7 Appalachian Region Employment......................................................................................37 5.8 Teacher Recommendations.................................................................................................38 6. Recruitment.............................................................................................................................40 6.1 Teacher Recruitment...........................................................................................................41 6.2 Student Recruitment............................................................................................................42 7. Review of Selected Pre-College STEM Programs................................................................44 8. Conclusions and Recommendations......................................................................................49 8.1 Recommendations about Program Objectives and Recruitment........................................49 8.2 Student-Specific Recommendations...................................................................................52 8.3 Teacher-Specific Recommendations..................................................................................54 8.4 Recommendations for Future Evaluation...........................................................................55 Appendices......................................................................................................................................i Tables Table 1—Percentage of students who became science majors post-program in selected STEM programs...................................................................................................7 Table 2—Comparison of respondents with all attendees and with attendees who were located but who did not respond to the survey–students.....................................12 Table 3—Comparison of respondents with all attendees and with attendees who were located but who did not respond to the survey–teachers......................................13 Table 4—Selected characteristics of student survey respondents................................................15 Table 5—Parents’ highest level of formal education...................................................................15 Table 6—Students’ ratings of the extent to which the Summer Institute influenced awareness, confidence and interest in STEM.....................................................19 Table 7—Students’ ratings of the extent to which various aspects of the Summer Institute positively influenced their thinking about going to college.....................22 Table 8—Mean scores for students’ ratings of the extent to which they received encouragement to attend college from the following:....................................23 Table 9—Students’ highest educational attainment.....................................................................24 Table 10—Major fields of study...................................................................................................24 Table 11—Degrees pursued or earned in STEM and other fields of study..................................25 Table 12—Workforce status of student participants....................................................................26 Table 13—Students’ opinions about STEM proficiency requirements of current jobs and future career...............................................................................................27 Table 14—Students who currently live in the Appalachian Region (AR) and those who plan on living there in 5 years, by cohort.....................................................27 Table 15—Selected Characteristics of Teacher Survey Respondents..........................................31 Table 16—Educational attainment of teachers.............................................................................32 Table 17—Research experience prior to attending the Summer Institute....................................32 Table 18—Teachers who reported incorporating Summer Institute experiences in their classrooms................................................................................................................34 Table 19—Teachers who gave high ratings to selected outcomes...............................................36 Table 20—Percentage of teachers who encountered various obstacles........................................36 Table 21—Current teaching status and place of employment......................................................38