Abstract Title Page Not included in page count. Title: Evaluating Math Recovery: Investigating Tutor Learning Author(s): Sarah Elizabeth Green and Thomas Smith 2010 SREE Conference Abstract Template Abstract Body Limit 5 pages single spaced. Background/context: Description of prior research, its intellectual context and its policy context. The success of Math Recovery, a constructivist, adaptive one-on-one tutoring intervention depends heavily on the skill and knowledge of the teachers who are selected to deliver it. MR is adaptive in that a key component of the program is that the tutor is expected to adjust instruction to the current level of a student’s thinking at any given point in time. This makes the practice of conducting the tutoring far more demanding than many more scripted or prescribed interventions. In the course of Math Recovery training, tutors are asked to think about mathematics and student learning in ways that are not typical in many US classrooms. Therefore, they are being asked to learn a new of way of teaching mathematics to students. The complex learning demands on new tutors are similar to the learning demands on classroom teachers attempting to implement ambitious reform mathematics instruction. Teachers learn from various sources in addition to their formal professional development, such as learning in and from practice (Franke & Kazemi, 2001). The literature has shown that teachers’ learning in practice often results from gaining access to students’ thinking and reasoning and having to reorganize their own understanding as a result (Franke, Carpenter, Fennema, Ansell, & Behrend, 1998). Since a key component of MR is uncovering, diagnosing, and building upon students’ thinking the potential for this type of generative learning in practice is substantial. Therefore, while it may seem that selecting tutors at the outset of the adoption of MR for their knowledge and skills would be important, the extent to which tutors can learn from enacting MR and gain the necessary knowledge as a result is still unknown. This analysis begins a line of investigation that attempts to understand the relationships between teachers’ knowledge and teachers’ learning in practice. One potential area where tutors may grow in their knowledge as a result of MR is their mathematical knowledge for teaching (MKT). While MKT may generally be important for good mathematics teachers (Ball, Lubienski, & Mewborn, 2001; Ball, Thames, & Phelps, 2008) it is particularly central to MR. MKT is knowing math in a specialized way that is particular to the profession of teaching. For instance, while average people competent in mathematics need to know how to add multi-digit numbers, teachers also need to know why the conventions for addition work, what are typical ways that students approach these kinds of problems, common errors they make, non-conventional methods that will work, which methods will be best built on later in their mathematical learning, etc. In MR, tutors are consistently assessing students’ current methods for solving problems and determining how to build on their current understanding, which implicates their MKT as an important aspect of the knowledge needed for good MR tutors. Researchers at the University of Michigan have developed an assessment of this type of math knowledge, the Learning Mathematics for Teaching (LMT) assessment (Hill, Schilling, & Ball, 2004). In this analysis we use tutors’ performance on the LMT as a way to understand how this kind of knowledge changes as a result of being a MR tutor. Another important area of knowledge for MR tutors is their knowledge of the MR Learning and Instructional Frameworks in Number (LFIN and IFIN, respectively). MR training is dedicated in 2010 SREE Conference Abstract Template 1 large part to tutors learning to understand and use these frameworks as a part of their practice as MR tutors. The frameworks lay out developmental trajectories for students in early number learning and suggest instructional activities to support students at various points along those trajectories. A tutor’s ability to understand and use the frameworks is key to their implementation of the tutoring program. Again, it may be that as tutors gain experience with different students and their thinking their understanding and skill in using the frameworks becomes more complete. Anticipating and understanding students’ thinking is central to both MKT and understanding the MR frameworks. This may mean that tutors who have more MKT can learn to understand and enact MR tutoring more quickly and effectively. This analysis uses tutors scores on a test of their knowledge of the frameworks (TKA) to investigate this aspect of changing knowledge as a result of gaining experience as a MR tutor and the relationship to MKT. There are two typical ways to enhance the skill and knowledge of teachers: hiring and professional development. While selecting teachers based on skills and knowledge seems logical, this may not always be practical for districts and schools hoping to adopt a new program. Absent direction from researchers or program developers districts may not know what qualities and skills are necessary and if they do know they may not be able to locate that type of expertise in their local context. Additionally, some of the knowledge and skill needed to deliver an intervention such as Math Recovery is specialized to the intervention itself. In this paper we investigate these possibilities, as well as a third, that tutors learn from the practice of tutoring itself. The findings from this analysis have potential implications for policy and implementation of MR as well as for future studies of this and like interventions. Purpose / objective / research question / focus of study: Description of what the research focused on and why. The goal of the overall study was to evaluate the potential of Math Recovery (MR), a pullout, one-to-one tutoring program, to increase mathematics achievement among low-performing first graders, thereby closing the school-entry achievement gap by enabling them to achieve at the level of their higher-performing peers in the regular mathematics classroom. An additional purpose of the study is to inform the design of future effectiveness and scale-up studies, should they be warranted, as well as policy decisions regarding adopting the program and selecting tutors. In order to achieve these purposes, we used two measures of tutor knowledge, the LMT and a developer created test covering the understanding and use of the MR frameworks. This analysis answers the following research questions regarding tutor knowledge: 1. Were there initial differences in the MKT of the tutors by site? 2. Were there differences in the uptake of MR training by site? Did tutors’ knowledge of the frameworks differ by site at the start of the experiment? 3. Do initial differences in tutor knowledge (both MKT and knowledge of MR frameworks) persist as tutors gain experience with MR and learn through practice? 2010 SREE Conference Abstract Template 2 4. How do differences in tutor knowledge and tutor learning relate to student outcomes?* Setting: Description of where the research took place. The two-year evaluation of Math Recovery was conducted in 20 elementary schools (five urban, ten suburban and five rural), representing five districts in two states. Each was a ‘fresh site’ in that the program was implemented for the first time for the purposes of the study. The newness of MR to the 18 tutors (two tutors worked at more than one school) makes investigating the change in knowledge over time particularly relevant for suggesting potential policy implications in new adoptions of the program. Each state was a different “site” in the context of this analysis in that they received different initial training experiences and had different ongoing support for their learning of the program. . Population / Participants / Subjects: Description of participants in the study: who (or what) how many, key features (or characteristics). Students were selected for participation at the start of first grade based on their performance on MR’s screening interview and follow-up assessment interview. The screening is designed to select the lowest achieving first graders (25th percentile and below) in terms of math achievement. The number of students eligible for tutoring ranged from 17 to 36 across the 20 schools. The number of study participants before attrition totaled 517 in Year 1 and 510 in Year 2, of which 172 received tutoring in Year 1 and 171 received tutoring in Year 2. The participating districts hired 18 teachers to receive training and participate as MR tutors—all of whom had at least two years of classroom teaching experience. Sixteen of the tutors received half-time teaching releases to serve one school each; two of the tutors served two schools each and thus were full time tutors. Intervention / Program / Practice: Description of the intervention, program or practice, including details of administration and duration. MR consists of three components: 1) student identification and assessment, 2) one-to-one tutoring, and 3) tutor training. In the first component of the program, the tutor conducts an extensive video-recorded assessment interview with each child identified as eligible for the program. The tutor analyzes these video-recordings to develop a detailed profile of each child’s knowledge of the central aspects of arithmetic using the MR Learning Framework, which provides information about student responses in terms of levels of sophistication The second component of the program, one-to-one tutoring, is diagnostic in nature and focuses instruction at the current limits of each child’s arithmetical reasoning. Each selected child receives 4-5 one-to-one tutoring sessions of 30 minutes each week for approximately 11 weeks. The tutor’s selection of tasks for sessions with a particular child is initially informed by the * This question will be addressed in the full paper, though it is not included in the remainder of this proposal. 2010 SREE Conference Abstract Template 3 assessment interview and then by ongoing assessments based on the student’s responses to prior instructional tasks. The Learning Framework that the tutor uses to analyze student performance is linked to the MR Instructional Framework that describes a range of instructional tasks organized by the level of sophistication of the students' reasoning together with detailed guidance for the tutor. The third component of the MR program, tutor training, involves 60 hours of instruction provided by an MR leader. The goal of this training is to support tutors’ in learning new practices for clinical assessment and intervention teaching in which they use the LFIN and the IFIN to adjust instruction based on cognitive evaluations of student responses. Training was conducted at each site by MR expert trainers for five days during the summer and with five additional days of follow up throughout the first two months of tutoring. Additionally, the tutors had support for site coordinators who met with them on a monthly basis. Research Design: Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic essay, randomized field trial). The larger evaluation study was a randomized field trial. In each year (2007-08 and 2008-09 academic years), 17 to 36 students deemed eligible (based on an initial MR screening) from each of the 20 schools were randomly assigned to one of three tutoring cohorts or to the “wait list” for MR. The cohorts, consisting of three students each, were staggered across different start dates (i.e., Cohort A—September, B—December, C—March). In both years students on the randomly ordered waiting list were selected to join an MR tutoring cohort if an assigned participant left the school or were deemed “ineligible” due to a special education placement. The number of study participants totaled 517 in Year 1 and 510 in Year 2, of which 172 received tutoring in Year 1 and 171 received tutoring in Year 2. To study teacher knowledge and learning, we assessed teachers at three time points with instruments discussed previously, the LMT and the TKA, to determine whether there were differences in the uptake of training across sites. Teachers at the two sites were trained somewhat differently as reported by the site coordinators and MR experts. Some of the differences in training were attributed to initial perceived differences in teachers’ MKT. Therefore, while this study does not have a control group, the differences between the groups provide variation for us to attempt to understand some of the conditions of implementing tutoring that effect tutor learning. It is our hop to uncover important factors of tutor learning to study more rigorously in future studies. Data Collection and Analysis: Description of the methods for collecting and analyzing data. The tutors were assessed using the LMT assessment to measure their MKT and the TKA to asses their knowledge of the MR frameworks. The assessments were given at three time points: end of MR training, end of year 1, and end of year 2. These assessments were scored and double- entered into a central database for further analysis. 2010 SREE Conference Abstract Template 4 This analysis uses one-way analysis of variance (ANOVA) where the predictor is training site membership. There were two main training sites. Therefore, in order to answer the questions that are a focus of this investigation, ANOVA is used to test for a difference between means of the groups on both the TKA and the LMT at time 1 and at time 3. In the full paper, we include hierarchical linear models (Raudenbush and Bryk, 2002) to model growth of tutor knowledge and also use this analysis method to link student outcomes (see proposal 1 for a full description) to learning rates of tutors during tutoring. Findings / Results: Description of main findings with specific details. Initially, there is a significant difference between groups at time 1 on the LMT (F=7.81, p = 0.013) and on the TKA (F=15.18, p=0.0013). This indicates that the tutors in site A were initially more knowledgeable in their MKT and also learned more about the MR frameworks from the initial MR training. This confirms the reports of the site coordinators and MR experts who conducted the training. However, at the end of the study there are no longer significant differences between these groups on either measure (LMT: F=3.55, p=0.08 & TKA: F= 1.36, p=0.26). Plotting the means of the groups on both measures over the three time points shows that both groups increased their mean scores on these measures over the three time points and therefore the lack of difference between groups at the end of the study is not due to tutors at site A decreasing in knowledge, but rather a steeper increase in knowledge at site B. In the full paper growth models are used to discuss this pattern of knowledge growth further. Conclusions: Description of conclusions and recommendations based on findings and overall study. These results have two implications for policy and for future studies of this intervention. First, tutors who had higher MKT at the outset also had higher scores on the TKA (r = 0.5, p= 0.03). Since, these tutors are all new to MR, this suggests tutors with more math knowledge for teaching may learn more from the initial MR training, potentially making them better choices for tutoring early on. Second, the initial differences did not persist between groups after two years of tutoring experience which suggests that tutors can and do grow in their understanding of the MR frameworks and also in their math knowledge for teaching through their MR tutoring practice. This is likely related to repeated attempts to understand students’ thinking. Students’ thinking and solution method is a key aspect of MKT and also an important part of using the MR frameworks with understanding. Understanding the exact mechanism for how tutors learn from the practice of tutoring students is an issue for research. An implication for policy and adoption of MR is that while initially tutors may struggle in their knowledge of MR, time and experience with the program will likely increase their knowledge of the program over time. 2010 SREE Conference Abstract Template 5 Appendices Not included in page count. Appendix A. References References are to be in APA version 6 format. Ball, D. L., Lubienski, S. T., & Mewborn, D. S. (2001). Research on teaching mathematics: The unsolved problem of teachers' mathematical knowledge. Handbook of research on teaching, 4, 433-456. Ball, D. L., Thames, M. H., & Phelps, G. (2008). Content knowledge for teaching: What makes it special? Journal of Teacher Education, 59(5), 389-407. Franke, M. L., Carpenter, T., Fennema, E., Ansell, E., & Behrend, J. (1998). Understanding teachers' self-sustaining, generative change in the context of professional development. Teaching and Teacher Education, 14(1), 67-80. Franke, M., & Kazemi, E. (2001). Teaching as learning within a community of practice: Characterizing generative growth. In T. Wood, B. C. Nelson & J. Warfield (Eds.), Beyond classical pedagogy in elementary matheamtics: The facilitative nature of teaching (pp. 47- 74). Mahwah, NJ: Erlbaum. Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers' mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11-30. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage Publications. 2010 SREE Conference Abstract Template A-1 Appendix B. Tables and Figures Not included in page count. 2010 SREE Conference Abstract Template