Abstract Title Page Not included in page count. Title: Exploring optimal conditions of instructional guidance in an Algebra tutor Authors and Affiliations: Hee Seung Lee [email protected] Carnegie Mellon University John R. Anderson [email protected] Carnegie Mellon University Susan R. Berman [email protected] Carnegie Learning, Inc. Jennifer Ferris-Glick [email protected] Carnegie Mellon University Ambarish Joshi [email protected] Carnegie Learning, Inc. Tristan Nixon [email protected] Carnegie Learning, Inc. Steve Ritter [email protected] Carnegie Learning, Inc. SREE Fall 2013 Conference Abstract Template Abstract Body Limit 4 pages single-spaced. Background / Context: In designing learning environments that support student learning, we face many instructional design decisions. These include when and how to provide examples, verbal explanations, feedback, and other scaffolding features. In this paper, we investigate instructional guidance as it relates to Cognitive Tutor, an intelligent tutoring system that provides students with an interactive learning environment. Cognitive tutors for high school mathematics have been successful in raising students’ test scores (e.g., Koedinger et al., 1997; Morgan & Ritter, 2002; Pane et al., 2013) and the Cognitive Tutor Algebra curriculum is being used in over 3,000 schools nationwide. A large number of studies have reported that provision of worked-examples facilitates learning (e.g., Atkinson et al., 2000). For example, Salden et al. (2010) implemented a faded worked example (Renkl et al., 2004) to a Geometry Cognitive tutor and reported increased learning outcomes. Although there is strong evidence that learning is facilitated by provision of worked examples, effects of providing verbal explanations along with examples are mixed at best (Lee & Anderson, 2013). However, in our prior study using a laboratory version of the algebra tutor (Lee et al., submitted) we found that the effect of providing verbal explanation depended on the transparency of the problem structure. Students were able to successfully master a problem-solving skill without explicit verbal direction as long as the problem structure was apparent enough that they could understand solution steps of the examples. We speculated that students in the no-explanation condition could learn to solve problems from non-verbal scaffolding methods and constrain their search space from immediate feedback about the correctness of their solutions. The immediate feedback helped reduce floundering and discover a problem solving rule from a student’s own solutions. Corbett and Anderson (1995), however, provided only limited support for the effectiveness of immediate feedback over delayed feedback. Using the Lisp tutor, they found immediate feedback led to faster, but not better learning than the delayed feedback condition although both conditions led to better learning than the no-feedback condition. Purpose / Objective / Research Question / Focus of Study: This paper reports early results from a large scale experiment evaluating different instructional conditions implemented in the Cognitive Tutor Algebra. This study explores the degree to which we can teach without verbal explanations and tests whether the results obtained in our laboratory will generalize to real classrooms. We developed two versions of the Algebra Tutor which differed on whether students were given explanations when they asked for hints and made errors. To enable students to learn without explanations, various modifications were made to the tutor, guided by three principles that followed from our prior study. The principles were 1) show intermediate cognitive steps that are not obvious to students via a worked-out example, 2) communicate problem structure by highlighting critical features of the problem, and 3) avoid excessive floundering by reducing off-path activity. We tested the effectiveness of the modified tutor by comparing to the effectiveness of the standard tutor (i.e., Carnegie Tutor Algebra 2012 release). Population / Participants / Subjects: High school students from a large rural and urban school district in West Virginia participated in SREE Fall 2013 Conference Abstract Template A-1 this study. The student population came from eight schools and consists of 89% white, 7% African-American, 1% Asian and others. We randomly assigned 22 classes (481 students) to the explanatory hint condition, 16 classes (282 students) to the non-explanatory hint condition, and 23 classes (429 students) to the control condition, resulting in a total of 61 classes (1192 students, 28 teachers participating). All students in a class were in the same condition to prevent contamination between students. Also, teachers who had multiple classes were assigned to multiple conditions. Intervention / Program / Practice: The Cognitive Tutor provides students with a sequence of topics (“units”). Students progress through the units at their own pace, dependent upon the time needed to master the material. Students are judged to have mastered the material if they are able to consistently, correctly complete components of the problem and their associated skills, without first making errors or asking for a hint (Corbett & Anderson, 1995). In this study, the experimental units covered linear equation solving and constituted 5 non-contiguous units in the 40-unit Algebra sequence. Students started working through the tutor in sequence with the curriculum from Fall of 2012, so the study lasted for the entire 2012/13 school year. All students worked with the same standard Carnegie tutor (2012 release) for most of the units except for the experimental units. These experimental units included “One-step linear equations”, “Two-step linear equations”, “Linear equations with similar terms”, and “Linear equations and the distributive property”. Research Design: This study contrasted two modified tutors with the standard tutor as a control condition. (Please insert Table 1 here) Table 1 summarizes changes included in the two modified versions of the tutor and corresponding features of the standard tutor. (Please insert Figure 1 here) The experimental versions of the tutor consisted of explanatory and non-explanatory conditions, depending on the type of hint messages provided to students. In the explanatory condition, students were given direct instruction about how to perform a certain step along with explanation on why that problem could be solved in that way. In the non-explanatory condition, students were left to discover a problem solution step while a hint message simply gave a direction on the tutor interface. For instance, for an equation of -9 + 2w = 3, the explanatory hint was “To eliminate -9, add 9 to both sides of the equation because -9 + 9 = 0. In the transform menu, select “add to both sides” and enter 9” whereas the non-explanatory hint was “Select an item from the transform menu and enter a number.” These hint messages were available on students’ request and when students failed to perform a correct step in a minute. In the case of wrong attempts, identical text prompt appeared in both conditions to instruct students to undo the step and ask for a hint. Figure 1 shows an example of the problem-solving environment in the modified Algebra Tutor. In both conditions of the modified tutor, largely four major changes were implemented to the standard tutor. First, students were given a side-by-side worked example so that they can make analogies from example to problem. These examples were randomly selected from a predefined pool of problems to prevent superficial analogy that is sometimes observed when an example is too similar to a problem. The side-by-side worked example faded when the student reached 75% average skill mastery for that unit. Second, several non-verbal scaffolding devices were included to highlight problem structure that is not often obvious to students. These include a color-coding of variable side of the equation and alignment of equations on the equal (=) signs. Students often show difficulty understanding the relation between the transformed equations. SREE Fall 2013 Conference Abstract Template A-2 Visual alignment was included to help students see these relations. Third, we prevented off-path activity by asking students to undo their just performed step when the performed step is incorrect. Also, when students fail to perform a correct step for 1 minute, we took it as an evidence of floundering and the tutor automatically provided a hint message that corresponds to the step without requesting a hint. Fourth, a hint was provided in a single message rather than in a progressive manner. The standard tutor uses multiple levels of hints, with each giving progressively more specific advice. However, this kind of progressive hint sometimes causes poor help seeking behavior. For example, students frequently use bottom-out hints to obtain answers without reading prior hint messages that explain the underlying principle of the answer (Baker et al., 2004). Data Collection and Analysis: At time of writing this report, we had a total of 400 students who completed the first 13 units in the Algebra curriculum. We included these completed students (122 explanatory, 103 non- explanatory, and 175 control condition) for data analysis. To examine the effect of instructional conditions, we focused on the three measures. These include the number of hints per problem, the number of errors per problem, and the proportion of mastered skills per section. Among the 13 units, only units 4, 5, 9 and 13 differed between the experimental conditions. Thus, we treated the first three units (1-3) as pre-units, 4-5 units as early experimental units, 6-8 units as early non-experimental units, 9 and 13 units as later experimental units, and 10-12 units as later non- experimental units. Initial data screening confirmed that there was no difference among the three conditions on any of the three measures in the pre-units. A one-way analysis of covariance was conducted to determine the effect of instructional conditions on learning when controlling for the performance of the pre-units. Findings / Results: (Please insert Figure 2 here) Figure 2 shows (a) adjusted mean number of hints per problem, (b) adjusted mean number of errors per problem, and (c) adjusted mean proportion of mastered skills for the three conditions on the early experimental units, early non-experimental units, later experimental units, and later non-experimental units. Regarding hints, there was a significant effect of instructional conditions in the early experimental units, F(2, 396) = 17.58, p < .001, η 2 p = 082, the early non-experimental units, F(2, 396) = 4.43, p = .012, η 2 = 022, and the later p experimental units, F(2, 396) = 26.05, p < .001, η 2 = .116. Students from the both modified p tutors used more hints than the control condition in both early experimental units and later experimental units, ps < .001, but there was no difference between the explanatory and non- explanatory condition, ps > .50. Regarding errors, there was not a reliable mean difference among the three conditions in any of the units. Regarding the proportion of mastered skills, there was reliable effects of instruction in the early experimental units, F(2, 396) = 3.29, p = .038, η 2 p = .016, and in the early non-experimental units, F(2, 396) = 4.30, p = .014, η 2 =.021, but not in p the later experimental and non-experimental units (Fs < 1). These effects were because explanatory students mastered fewer skills than the other groups of students. In the early experimental units, the explanatory students (M = .96, SD = 0.05) mastered fewer skills than the non-explanatory students (M = .98, SD = 0.05), p = .034. In the early non-experimental units, the explanatory students (M = .89, SD = 0.07) mastered fewer skills than the control condition (M = .91, SD = 0.07), p = .011. SREE Fall 2013 Conference Abstract Template A-3 Conclusions: To summarize the results, we did not find any benefits of adding verbal explanations. The explanatory and non-explanatory students did not differ in terms of number of hints or errors in any of the experimental and non-experimental units. In fact, the non-explanatory students showed higher proportion of mastered skills than the explanatory students in the early experimental units (Figure 2c). The results suggest that students were able to learn without verbal direction in certain instructional environments. We attribute the success of the non- explanatory group to two main factors. First, a side-by-side example was available while students were working on a problem and this helped students to generate a correct problem solution step on their own. Second, prevention of off-path activity seemed to reduce unnecessary floundering and guide students to correct solution paths. Correctly solved problems become an example for later problems. However, when students show excessive floundering, they may not be able to remember how they solved a problem (Lewis & Anderson, 1985). Regarding the contrast between the modified and standard tutors, the modified tutor group requested more hints than the control group in both the early and later experimental units (Figure 2a). We believe this greater hint-seeking behavior in the modified tutor is a consequence of blocking off-path attempts. In case of wrong attempts, the modified tutor (both explanatory and non-explanatory) asked students to undo the just performed step and use a hint. This seemed to encourage students to request help whenever they make an error and in turn increase the help seeking behavior. In the standard tutor, students can continue after performing a non-optimal (but mathematically acceptable) transformation to the equation. More interestingly, there were small but significant downstream consequences of the modified tutor on the early non- experimental units (but not on the later non-experimental units). As shown in the proportion of mastered skills (Figure 2c), the students who used the modified tutor in the experimental units mastered fewer skills than those who used the standard tutor in the experimental units although both groups used an identical standard tutor in the non-experimental units. Also, the modified tutor group requested more hints than the control group. There are a number of possible explanations for these downstream effects. First, students with the modified tutor may have faced problems re-adapting to the standard tutor when they came to the non-experimental units. In the non-experimental units, they were not able to take advantage of various scaffolding features (e.g., side-by-side example) that were available in the experimental units. Due to the lack of a side-by-side example, they might have requested more hints, and in turn gained less learning outcome. Second, reliance on hints on the experimental units might have had a persistent effect on the following non-experimental units among the modified tutor students. If they learned that their hint requests helped them move forward faster, their help seeking behavior might have been reinforced and persisted in later units as well. Third, removing off-path activity on the experimental units might have decreased freedom for exploration and in turn decreased the attitude of “I can do it on my own.” The modified tutor did not allow off-path activity; thus, students did not have an opportunity to observe what happened to the equation when a wrong transformation was performed. This feature may have decreased the feeling of self-control and increased the attitude of relying on the direction. In this research we tried to enable students to learn without explicit explanations. We realized this objective by providing worked examples and preventing off-path activity to reduce unnecessary floundering. Explanatory verbal explanations appear to be not necessary when students can learn from examples. SREE Fall 2013 Conference Abstract Template A-4 Appendices Not included in page count. Appendix A. References References are to be in APA version 6 format. Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70, 181-214. Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-task behavior in the Cognitive Tutor classroom: When students “game the system.” Proceedings of ACM CHI 2004: Computer-Human Interaction (pp. 383-390). Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43. Lee, H. S., & Anderson, J. R. (2013). Student learning: What has instruction got to do with it? Annual Review of Psychology, 64, 445-469. Lee, H. S., Betts, S., & Anderson, J. R. (submitted). Revealing hidden problem structure by verbal instruction versus non-verbal scaffolding methods. Lewis, M. W., & Anderson, J. R. (1985). Discrimination of operator schemata in problem solving: Procedural learning from examples. Cognitive Psychology, 17, 26-65 Morgan, P., & Ritter, S. (2002). An experimental study of the effects of Cognitive Tutor® Algebra I on student knowledge and attitude. (Available from Carnegie Learning, Inc., 1200 Penn Avenue, Suite 150, Pittsburgh, PA 15222). http://www.carnegielearning.com Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2013). Effectiveness of Cognitive Tutor Algebra I at scale. Santa Monica, CA: RAND Corporation. http://www.rand.org/pubs/working_papers/WR984. Renkl, A., Atkinson, R. K., & Große, C. S. (2004). How fading worked solution steps works - a cognitive load perspective. Instructional Science, 32, 59-82. Salden, R. J. C. M., Aleven, V., Schwonke, R., & Renkl, A. (2010). The expertise reversal effect and worked examples in tutored problem solving. Instructional Science, 38, 289-307. SREE Fall 2013 Conference Abstract Template A-5 Appendix B. Tables and Figures Not included in page count. Table 1. Summary of characteristics in the three versions of the Carnegie Tutor Algebra. Explanatory Non-explanatory Control condition condition condition (standard tutor) “how” and “why” Information on tutor “how” and “why” Hint - Content information interface information Hint - Presentation Non-progressive single message Progressive Hint - Voluntariness Auto-appearance after 1 minute delay None Side-by-side Presents an example next to the problem None worked example Disappears when average skill mastery = 75% - Color-coding Other non-verbal - Alignment between equations None scaffolding features - Highlighting changed parts between transformations Error flagging Prevents off-path activity, requires an UNDO for Error using an orange the incorrectly performed step. color SREE Fall 2013 Conference Abstract Template B-1 Figure 1. An example of a problem-solving environment of the modified tutor. Side-by-side worked example is presented on the left side of the screen while students are working on a problem. The variable side of the equation is color-coded (green background) and equations are aligned with the equal (“=”) sign. When students make an error, the tutor requires them to undo their performed step. SREE Fall 2013 Conference Abstract Template B-2 (a)     5   Explanatory   f   4.5   er  o 4   Non-­‐explanatory   b Control   m 3.5   u n n  ts   3   an mehi 2.5   d   2   e t s 1.5   u dj A 1   0.5   0   Early  exp  units   Early  non-­‐exp  units   Later  exp  units   Later  non-­‐exp  units     (b)     12   Explanatory   f   Non-­‐explanatory   o r   10   Control   e b m 8   u n  nrs   ao 6   meerr d   4   e t s u dj 2   A 0   Early  exp  units   Early  non-­‐exp  units   Later  exp  units   Later  non-­‐exp  units     (c)   Explanatory   of   Non-­‐explanatory   on   1.0   Control   ti r o op 0.9   prs   ean  skill 0.8   m d   e 0.7   t s u dj A 0.6   0.5   Early  exp  units   Early  non-­‐exp  units   Later  exp  units   Later  non-­‐exp  units     Figure 2. (a) adjusted mean number of hints per problem, (b) adjusted mean number of errors per problem, and (c) adjusted mean proportion of mastered skills for the three conditions on the early experimental units, the early non-experimental units, the later experimental units and the later non-experimental units. SREE Fall 2013 Conference Abstract Template B-3