Implementing an adaptive intelligent tutoring system as an instructional supplement Andrea Phillips1, John F. Pane1, Rebecca Reumann-Moore2 & Oluwatosin Shenbanjo2 RAND Corporation1 & Research for Action2 Andrea Phillips, M.Ed. Corresponding author Email: [email protected] Phone: 412-683-2300 X 6094 Fax: 412-623-2800 John F. Pane, Ph.D. Email: [email protected] Rebecca Reumann-Moore, Ph.D. Email: [email protected] Oluwatosin Shenbanjo, M.S. Email: [email protected] Citation: Phillips, A., Pane, J. F., Reumann-Moore, R., & Shenbanjo, O. (2020). Implementing an adaptive intelligent tutoring system as an instructional supplement. Educational Technology Research and Development, 1-29. The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A1400221 to RAND Corporation. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. Implementing an adaptive intelligent tutoring system as an instructional supplement Blended learning is an instructional model that combines teacher instruction with online or digital learning. It can also help to enable more personalized approaches by freeing some of the teacher’s time, which otherwise would have been used to provide whole-class instruction, so they can focus on individual students while other students are using technology (Pane et al., 2017). Some empirical evidence suggests that technology-based curricula can help personalize students’ learning experiences and facilitate the development of mathematical skills (Koedinger et al., 2000; Ritter et al., 2007; Schacter, 1999; Wenglinsky, 1998). A meta- analysis conducted by Means et al. (2010) estimated that interventions combining online and face-to-face instruction in a blended-learning approach appeared to produce more positive effects than either online or face-to-face instruction alone. Subsequently, more evidence has been emerging from rigorous studies on learning systems operating in classroom settings, and the evidence of efficacy has been mixed (Morgan & Ritter, 2002; Pane et al., 2010; 2014). These studies highlighted some of the challenges implementing the systems in classrooms, which may contribute to a lack of consistent positive effects on student learning. Literature on blended learning stresses the importance of (1) intentional integration of the two modalities to create an integrated learning experience and (2) effective professional development on how to divide the instructional role between teacher and technology (Bailey et al., 2013; Bowyer & Chambers, 2017; Ferdig & Kennedy, 2014; Patrick et al., 2013; Watson et al., 2013). Yet, robust research on blended learning implementation is sparse, with some authors calling on future research to develop better ways to measure implementation. Both Karam et al. (2016) and Snodgrass, Rangel et al. (2015) identified the need to investigate dosage (how much of the intervention is needed to have an effect) and fidelity (whether the intervention is implemented as intended) as important aspects of blended learning implementation. Karam et al. (2016) extended this by suggesting that, irrespective of implementation fidelity, it is important to understand how the intervention changes instructional practices. Bowyer and Chambers (2017) and Baily et al. (2013) echo the importance of examining teaching, learning, and instructional design. Others discuss the importance of learning analytics (data collected by the software) to gain a deeper insight into the implementation of digital innovations and their effects on teaching and learning. Learning analytics can enable evaluators to examine how a user interacts with various components and weigh this along with dosage to gain insight into student engagement with the technology and the impact of specific components (Snodgrass Rangel et al., 2015). The present study builds on many of these recommendations to help enrich the field’s understanding of blended learning implementation. Present study This article studies the implementation of ALEKS (Assessment and LEarning in Knowledge Spaces) in a blended-learning model. ALEKS is an intelligent tutoring system for mathematics designed to be integrated with existing curricula. This is the first rigorous experimental evaluation of ALEKS in K-12 in-school settings, and the first to gather extensive implementation data. We selected ALEKS for this study because it was in widespread use in high school classrooms and had shown promise for improving student achievement but was not yet subjected to a rigorous test of efficacy for that use. Prior research on ALEKS generally relied on non-causal methods or examined its use in other contexts. Encouraging effects have been reported for K-12 students in afterschool programs; for post-secondary students; and for adult learners (Ahlgren & Harper, 2009; Baxter & Thibodeau, 2011; Carpenter & Hanna, 2006; Craig et al., 2014; Hu et al., 2012; Hu et al., 2013; Huang et al., 2016). Craig et al. (2013) examined the relationship between student mindset and engagement with ALEKS. A study by the developers found that the system showed promise in improving algebra readiness (ALEKS Corporation, 2011) but did not focus on implementation. Two middle school studies by Sullins et al. (2013) found positive correlations between usage and student achievement, although it is unclear how online and traditional learning were blended in that implementation. Our study examined the use of ALEKS as a supplement to an existing high school algebra curriculum, as part of a randomized controlled trial studying its efficacy for improving achievement. Approximately 2,500 students were randomly assigned to use ALEKS as a supplement to the district’s algebra curriculum, or a control group that just used the algebra curriculum. The details of the study design and results of the outcomes study will be reported in a separate publication. Briefly, the study found no significant effect on an end-of-course algebra exam. The focus of this article is to address gaps in the research literature on implementing intelligent, adaptive learning systems in a blended learning environment (Bailey et al., 2013; Bowyer & Chambers, 2017; Karam et al., 2016; Snodgrass, Rangel et al., 2015). As such, it focuses on the classrooms in the experiment that were assigned to use ALEKS. The implementation study is guided by the following research questions: (1) What models did teachers use to integrate ALEKS into instruction? (2) To what extent did implementation adhere to the core aspects of the ALEKS design? and (3) To what extent did blended use of ALEKS enable personalized instruction? Basic overview of the ALEKS experience for students and teachers Upon first entry into the ALEKS system, the student takes a diagnostic (or placement) assessment that seeks to uncover what the student already knows or does not know. The result of this assessment is displayed to students. Students view the algebra topics already mastered and those not yet mastered. ALEKS curates a customized path of “ready to learn” algebra topics for the student because they have mastered the necessary prerequisites. The student can choose any topic from among the ready-to-learn topics for the ALEKS course. The ALEKS algebra course includes more than 350 topics, some of which reflect prerequisite algebra topics. This mechanism thus implements a mastery-based approach to progression through the software, while still giving students some choices of what to work on. Theoretically, this ready-to-learn, mastery-based strategy could be beneficial compared to typical classroom instruction where all students cover material at the same pace regardless of whether they understand key prerequisites. Using this strategy, students can fill in gaps of understanding, work a zone of proximal development (Vygotsky, 1930–1934/1978), experience success, and build a better foundation for learning more difficult topics. This can potentially lead to more robust learning. However, if students spend a substantial amount of time filling in prerequisite gaps, their learning, even if substantial, might not align with the algebra curriculum—thus posing a tension known as the mastery versus coverage dilemma (Slavin, 1987). Implementing ALEKS as a supplement may ease the tension somewhat because teachers can cover the required course content and standards while ALEKS provides students with personalized instruction to support increased mastery. ALEKS personalizes student experiences using an algorithm based on the Knowledge Space Theory. The main instructional activity within ALEKS is solving problems in a constructed-response environment that uses realistic input tools and avoids multiple-choice questions to help ensure that the student demonstrates mastery. The student receives immediate feedback and has access to step-by-step explanations (essentially, worked examples) of how to solve problems. Mathematical terms and concepts that appear in the problems and explanations are hyperlinked to a central glossary. After learning a topic, the student returns to an updated ALEKS pie to choose the next topic to learn. Periodic progress assessments are used to confirm retention, and if the student no longer demonstrates mastery of a previously-completed topic it is returned to the “need to learn” portion of the pie. The student continues to participate in this process of learning and assessment until they reach their learning goals. The assumed learning goal in ALEKS is 100 % completion of the course, but instructors can set intermediate goals for their students. For teachers, ALEKS provides a learning management system to support class administration, instruction, customizing course content, and progress monitoring. Various reports can be used as part of a data-driven, decision-making process to guide their work, both during whole-class instruction and for small group or individualized instruction while other students are using the software. For example, ALEKS can identify topics that require additional focus for the whole class or identify groups of students who are all ready to learn the same topics. Teachers can work with these groups or assign them to work collaboratively using ALEKS-generated worksheets. Teachers can also use the software to create homework assignments and quizzes. Methods Study Design and Participants The study was implemented in a large, urban school district in the mid-Atlantic region, where levels of mathematics proficiency measured by the 2014 state algebra I assessment were well below state averages, with performance gaps by race/ethnicity. Nine high schools participated in the study for two consecutive school years, with different cohorts of algebra students each year, but, the same teachers to the extent possible given teacher turnover. The two-year implementation allowed for possible improvement in implementation after teachers gained experience with ALEKS. The study actually took place over three years because some schools entered the study later. Table 1 summarizes characteristics of the participating schools. Five offered open admission to students coming from middle schools within a feeder region; students from outside the feeder region could apply for admission. The other four schools required all students to apply, with admission criteria related to attendance, punctuality, behavior, grades, and standardized test scores. The vast majority of students enrolled in study classes were non-white and about half of the students in study classes were female (Table 1). Most students (66%) were economically disadvantaged. Fourteen percent of students had an Individualized Education Plan and very few students were limited English proficient. The vast majority of students enrolled in study Table 1 Characteristics of participating students by study school School Participating School Number of Non-white (%) Female (%) Economic Disability (%) Limited English School Years Admissions Students Disadvantage Proficient (%) Criteria (%) S01 2014-15 Yes 208 99 56 77 16 2 2015-16 S02 2014-15 209 100 36 71 36 2 2015-16 S03 2014-15 487 85 43 62 7 9 2015-16 S04 2014-15 291 100 45 71 14 1 2015-16 S05 2014-15 Yes 243 89 58 53 5 3 2015-16 S06 2014-15 Yes 214 99 33 71 23 2 2015-16 S07 2015-16 354 75 44 66 9 10 2016-17 S08 2015-16 Yes 280 80 59 51 8 2 2016-17 S09 2015-16 208 99 56 80 24 16 2016-17 Overall 2494 90 47 66 14 1 classes were enrolled in 9th or 10th grade. On the of 8th grade state mathematics assessment— the assessment used as a pretest—the average study student scored at about the 23rd percentile statewide. With one exception, all participating teachers were veteran, with more than 5 years experience teaching mathematics. Of the 38 teachers, three had limited exposure to ALEKS prior to the study. No teachers implemented personalized learning previously; a few teachers occasionally used software or applications to supplement the district curriculum. The implementation model in this study involved using ALEKS as a blended learning supplement to those existing district materials, including a scope and sequence plan. Teachers received the initial training from ALEKS staff—employees of the software developer focused on training and supporting implementation—prior to implementing the software. Training provided an overview of the software features for students and teachers, and covered guidelines on implementation, including specific information on how long and how often students should be working in ALEKS. It also provided instruction in how to read ALEKS reports to find key indicators of strengths, weaknesses, and progress, and best practices. In the 2015–16 and 2016– 17 school years, teachers received monthly in-person visits from ALEKS staff. Data collection The research team conducted three site visits per year to all classes in the study. The visits collected information on instruction, student engagement and ALEKS implementation. The first site visit occurred in the fall, approximately one month after schools deemed course enrollment stable. The third site visit generally occurred in the weeks before the state assessment, and the second visit was about halfway between the other two. Each site visit included observations of instruction and teacher interviews. We designed classroom observation and teacher interview protocols, informed by an implementation checklist used by ALEKS staff for their implementation support visits, material covered during teacher training sessions, and protocols used in previous studies (e.g. Augustine et al., 2016; Karam et al., 2016). The checklist covers several points the company believes are important for the software to work effectively in a blended environment, such as ensuring the algebra curriculum is appropriate given the student’s preparation, use by students of notebooks, and use by teachers of reporting features to monitor progress and make instructional decisions. At teacher trainings, ALEKS staff recommended each student use the software for at least two hours per week, or at least 60 hours over the school year. Teachers were encouraged to consider a variety of models for how they might integrate ALEKS into their course to accomplish usage and blended learning guidelines. An online repository contributed to by teachers who had used ALEKS previously also offered some possibilities. Observation domains included time on task, student engagement, indicators of general instructional quality, and indicators of quality for implementation of ALEKS. We reviewed the instrument with ALEKS staff and piloted it in classes that did not participate in the study. During the pilot, we refined the protocol and established inter-rater agreement of 100% on each item. When two raters did not initially agree, we discussed the item and arrived at consensus. Raters continued to observe classes until achieving 100% agreement in an observation. After each observation of a study class, one rater reviewed the running record of instruction and ratings for agreement. The interview protocol covered curriculum, planning, instructional practices, and support received for implementing the algebra I curriculum and ALEKS. Interviews were semi-structured and conducted in-person. ALEKS provided student-level logs of software usage. In this paper we summarize usage time (in hours) for students enrolled in ALEKS classes, and classroom level medians of usage. Other, more complex analyses of the log data are planned for other publications. Analytic approach Given the flexibility offered to teachers in how to use ALEKS, as well as lack of prior research on ALEKS implementation, we developed a framework for analyzing implementation (Dane &