Reporting from the Test Kitchen: Blended Learning at New Leaders Rebecca Sutherland & Colin Porter Acknowledgments This study was a collaborative project involving program participants in five engagements from more than 10 different districts and charter schools, as well as individuals from teams across the New Leaders organization. Rebecca Sutherland led the project, with enormous and invaluable contributions from Colin Porter. Rebecca Sutherland and Colin Porter undertook data collection and analysis, with Colin Porter leading in developing the qualitative coding scheme and Rebecca Sutherland in conducting the quantitative code analyses. Rebecca Sutherland wrote the report with substantial input throughout from Colin Porter; Rebecca Sutherland and Colin Porter jointly wrote the case studies. The researchers would like to thank the New Leaders staff members who provided crucial input and guidance throughout the project — including Mark Bailey, Jennifer Baker, Janisia Bennet, Kavish Burney, Denise Corbett, Maria Esponda-Medina, Michael Falletti, Danielle Goddard-Ellis, Hannah Grantz, Jodut Hashmi, Emily Hicks-Rotella, LaWanda Hill, Felecia Irick, Hannah Irvin, Laura Kadetsky, Amy Koenigs, Carol Johnson, Krista Lewis-Johnson, Tangela Johnson, Tori McRoberts, Ceci Morales, Ed Morris, Rayanne Mroczek, Scott Neil, Brenda Neuman-Sheldon, Alexandra Pallas, Gabe Scheck, Kimberly Washington, and Margaret Young. Alexandra Broin, Devon Dillon, Jean Desravines, Michael Falletti, Danielle Goddard-Ellis, Jackie Gran, LaWanda Hill, Hannah Irvin, John Jenkins, Tangela Johnson, Brenda Neuman-Sheldon, Kimberly Washington, Gabe Scheck, and Margaret Young provided helpful feedback on drafts of this report. Virginia Patterson led report design. Above all, New Leaders is deeply grateful to the educators who participated in this study, wel- comed the researchers into their schools, made time during their busy days for interviews, and generously shared their experiences and insights. Their hard work and commitment to their students’ learning, and their dedication to their school communities are enduring inspirations. This report was made possible by a generous grant from the Bill & Melinda Gates Foundation. © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 2 Chapter 1 Blended Learning: Choosing a Recipe for Success In the 2017–18 school year, New Leaders studied a series of innovative programs for public K–12 school leaders and faculty that incorporated independent online instructional components with in-person professional development sessions, job-embedded practice, and personalized coaching. New Leaders positioned itself to leverage the use of technology in delivering high-quality adult learning to a wide span of geographic regions, and in accom- modating the scheduling needs of busy education professionals who are unable to attend frequent in-person sessions. In adopting a blended learning paradigm for professional development, New Leaders paved the way for nimbler, more integrated, and, ultimately, more effective partnerships. For the past 17 years New Leaders has provided crucial, high-quality professional development that fosters leadership skills in working educators. Multiple independent studies have confirmed the effectiveness of New Leaders’ training for improving student achievement and school climate.1,2 New Leaders is committed to using data to continuously improve its programming and partnerships. With a longstanding history of sharing lessons with the field to advance effective leadership practices, the organization sought to document and examine the development and implementation of its new blended learning programs. The goals were to learn what worked and what didn’t — and to make specific improvements that could yield stronger results for participants and their students in the future. Two decades of peer-reviewed studies of in-person, online, and blended instruction have consistently demonstrated that quality trumps delivery method. High-quality blended programs outperform low-quality online or in-person programs, and vice versa.3 1  Gates, S. M., Hamilton, L. S., Martorell, P., Burkhauser, S., Heaton, P., Pierson, A., . . . Gu, K. (2014). Preparing principals to raise student achievement: Implementation and effects of the New Leaders program in ten districts — Appendix. Santa Monica, CA: RAND Corporation. 2  Gates, S.M., Baird, M.D., Doss, C.J., Hamilton, L.S., Opper, I.M., Master, B.K., …Zaber, M.A. (2019). Preparing school leaders for success: Evaluation of New Leaders’ Aspiring Principals Program, 2012-2017. Santa Monica, CA: RAND Corporation. 3  Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: U.S. Department of Education. Retrieved from http://www.ed.gov/about/offices/list/opepd/ ppss/reports.html © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 3 Although no published research currently addresses the delivery of blended leadership development programs to teams of in-service teachers and administrators,4 examples from higher education and adult learning suggest that well-supported blended learning can be effective in this context. Drawing on blended and adult learning theories, this report presents the findings of New Leaders’ school year 2017–18 mixed-methods investigation into its blended learning programs. A comparative analysis of programs in districts and charter networks in five different regions across the United States reveals the opportunities and challenges of these programs. Developing and implementing strong blended adult learning engagements has much in common with adapting tried-and-true recipes and creating new ones to use with new kitchen tools: Successful experimentation depends on knowledge of the ingredients and tools, and on an explicit understanding of the desired outcome. Just as there are many ways to make a delicious lasagna, there are many ways to build and deliver effective blended adult instruction — provided that the essentials are in place and that the cooks know what to do with them. This investigation of New Leaders’ blended engagements in school year 2017–18 reveals what those essentials are and offers suggestions for tailoring the necessary components of blended adult learning to the local context. Methodology This study addressed the following questions: 1. What are the features of effective blended learning models for adult professional development? 2. What lessons for the field of adult professional development do New Leaders’ blended program models offer? 3. What are the successes, opportunities, and challenges that New Leaders has encountered in developing and delivering blended programs? To address these questions, New Leaders conducted a series of program observations and interviews with blended-program participants. Two New Leaders researchers spoke with participants in five New Leaders’ blended learning programs during the 2017–18 school year. Interviews with program participants were conducted in-person and were recorded.* District names have been removed to retain confidentiality. Coded interview transcripts captured the catalysts and dampeners (things that either help or hinder) of program completion. In participant interviews, the team identified five thematic categories that closely align with the literature on blended learning (see Appendix B). The team applied the coding scheme to the interview transcripts, and double-coded a subset to ensure the reliability of the analysis.** This study also includes: interview data from New Leaders staff who were directly involved in scoping, designing, and implementing blended learning engagements during school year 2017–18; 2017–18 participant survey data routinely collected by New Leaders; and user meta- data from Canvas, a learning management system that provides insight into how participants used the online resources New Leaders made available to them. 4  Voci, E., & Young, K. (2001). Blended learning working in a leadership development programme. Industrial and Commercial Training, 33(5), 157–161. * Interview participants were compensated for their time with DonorsChoose gift cards. ** (Cohen’s kappa = .72) © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 4 Chapter 2 Prepping for Blended Learning: Organizational Alignment, Resource Allocation, Staff Buy-In, and Key Ingredients Many institutions see blended learning as a panacea that promises less overhead and better learning outcomes.5 However, research shows that high-quality blended learning must begin with an organization’s commitment both to garner the staff’s support for this complex endeavor and to provide adequate start-up resources — technological resources, staff time, and financial support.6,7 Whether for school-age students or adult learners, blended learning programs may initially be more expensive than traditional in-person programs because of essential investments in technology and other areas of support.8,9 Orga- nizations that implement blended instruction must first precisely map out the necessary resources — in technology, instructional staff, and support staff — and must make sure that organizational structures are in place to support the transition. I. Organizational Alignment: Using a New Tool in a New Context — New Leaders’ Flexible Engagements Some of the blended engagements in this study were new partnerships in school year 2017–18, whereby New Leaders collaborated closely with districts on targeted programming to address their most pressing priorities. New Leaders and its partners worked together to select “à la carte” from a menu of research-based instructional content drawn from established New Leaders programs. These “flexible” engagements could differ from established New Leaders programs in several ways: duration (e.g., four or six months, instead of a whole school year); content (e.g., mixing content from different established New Leaders programs); or designated participants (e.g., principals and their leadership teams alongside principal supervisors). 5  Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. San Francisco: John Wiley & Sons. 6  Moskal, P., Dziuban, C., & Hartman, J. (2013). Blended learning: A dangerous idea? The Internet and Higher Education, 18, 15–23. 7  Stacey, E., & Gerbic, P. (2008). Success factors for blended learning. Hello! Where are you in the landscape of educational technology? Proceedings ASCILITE Melbourne 2008, 964–968. 8  Means, B. et al. (2009). 9  Brown, J., & Kurzweil, M. (2017). Instructional Quality, Student Outcomes, and Institutional Finances. Washington, DC: American Council on Education. Retrieved from https://www.acenet.edu/news-room/Pages/Instructional-Quality-Student-Outcomes-and-Institutional-Fi- nances.aspx © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 5 As it embarked on delivering blended content in new ways during school year 2017–18, New Leaders was also finding its feet with new, flexible program models. Some staff members felt underprepared to navigate new engagements that were both blended and flexible. II. Adequate Resource Allocation: The Costs of Blended Learning Building blended instructional programs can be expensive. The prevailing assumption that blended programming is cheaper to produce and deliver is widespread; when unexamined, this belief can undermine efforts to set up high-quality blended programming.10 If budgets are set without accounting for the extra staff hours needed for training and for providing ongoing technical support to staff and participants, essential resources may not be in place from the start; furthermore, staff may miss or de-prioritize those needs once the program is underway (especially if they are competing with other urgent needs for finite financial or time resources). Researchers struggle to calculate the relative costs of online, blended, and in-person programs because of vast differences among programs, institutional accounting practices, the cost formulas used for comparison, and other issues. For example, when one widely reported study showed that blended learning at the University of Central Florida reduced overall expenditures by nearly 40 percent, critics pointed out that this figure omitted many start-up costs and ongoing supports involved in the virtual program components, because the calculations focused primarily on instructor time and facilities costs.11 New Leaders developed virtual content starting in 2014, with generous support from a host of foundations.12 The content initially served as a standalone, virtual program; it later evolved into discrete modules for blended engagements. Our experience underscores the start-up costs associated with creating content on the front end, and then providing adequate ongoing support for instructors and students. III. Buy-In from Staff and Cross-Team Coordination While recognizing that blended instructional models have great “Initially, I spent a lot of time with the potential, research shows that teaching fully or partially online has platform. When I’m in the room, it’s required more of instructors’ time than teaching in person — as about my facilitation style and being much as three times more.13 These issues of skill and time converge able to read the room, faces, etc. When as instructors often find themselves providing front-line technical support to students, a time-consuming task for which they may not be you’re online, you have to anticipate prepared. But when adequately supported, blended instructors often and solve for technology problems realize greater student satisfaction than in traditional course settings.14 before the program actually starts.” Among other decisions, designers of blended adult professional NEW LEADERS PROGRAM IMPLEMENTATION STAFF development programming must determine how to create a thriving in-person and online community, engage participants, build in flexibility, and facilitate participant learning.15 10  Bakia, M., Shear, L., Toyama, Y., & Lasseter, A. (2012). Understanding the Implications of Online Learning for Educational Productivity. Washington, DC: U.S. Department of Education, Office of Educational Technology. 11  Bakia et al. (2012). 12  Generous support for New Leaders’ development of online content was provided by Charles and Helen Schwab, the Wasserman Foundation, and Anonymous. 13  Ocak, M. A. (2010). Blend or not to blend: A study investigating faculty members’ perceptions of blended teaching. World Journal on Educational Technology, 2(3), 196–210. 14  Caravias, V. (2018). Teachers’ conceptions and approaches to blended learning: A literature review. In Oncioiu, I. (Ed.), Driving Innovation and Business Success in the Digital Economy, pp. 228–251. Hershey, PA: IGI Global. 15  Boelens, R., De Wever, B., & Voet, M. (2017). Four key challenges to the design of blended learning: A systematic literature review. Educational Research Review, 22, 1–18. © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 6 New Leaders staff are keenly aware of how highly participants value opportunities to network with their peers and interact personally with their instructors: “[Participants] seem to be most interested in the videos, exemplars of assignments, things that they can read through and watch at their own pace to prepare for the in-person sessions, so that they can really use the session time to connect with their peers that they otherwise don’t usually see.” NEW LEADERS PROGRAM IMPLEMENTATION STAFF When creating their own blended courses, many online instructors have discovered that a high-quality educational experience requires more than retrofitting an existing in-person course with online discussion boards and YouTube links. The design and delivery of online blended courses — for children or adults — require a dramatically different technological and pedagogical skill set than the design and delivery of courses taught in-person demand.16 The best of both worlds, as blended learning is sometimes characterized, is often achieved when a team builds new courses from scratch. An effective blended learning program design is not simply a ratio of time spent in-person versus online.17 Surveys of adult learners reveal that they are eager for more experimental and integrative approaches to blended instruction, even as instructors and institutions favor more conservative, traditional-looking programs that are perceived to be less likely to fail in engaging students and imparting learning.18 Perhaps unsurprisingly, surveys show that teachers are more willing than other professionals to experiment with different ways of learning that use online resources, and have provided useful feedback to course designers to shape educational programs for adult learners.19 “I heard…that something new was coming down the pipe to build leadership teams, and that we should be on the lookout for it, and it might be something that we should consider as principals to help move the work inside the schools. So, once it did come down, I jumped on board immediately… [and at the end of that program] said, ‘Hey, if you guys are having it for ‘17 or ‘18, my school needs to be a part of it.’” ”V,” PARTICIPATING PRINCIPAL “It was kind of amazing — even though [our principal] wasn’t with us for that first video watching, we came away with the same ideas. We can see our way of work, and then we can also be very reflective on our individual styles, our individual way of work. The last [video] we did, we all sat and did it together and, again, we were able to talk [about]...our strengths as well as our weaknesses and know how to be better as a team. [The videos] helped us develop a way of work that supports the bigger picture.” ”N,” PARTICIPATING TEACHER 16  Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7(2), 95–105. 17  Graham C. (2013). Emerging practice and research in blended learning. In Handbook of Distance Education (3rd edition), Michael Grahame Moore (Ed.), p. 334. New York: Routledge. 18  Mirriahi, N., & Alonzo, D. (2015). Shedding light on students’ technology preferences: Implications for academic development. Journal of University Teaching & Learning Practice, 12(1), 6. 19  Koutsodimou, K., & Jimoyiannis, A. (2015). MOOCs for teacher professional development: Investigating views and perceptions of the participants. In Proceedings of the 8th International Conference of Education, Research and Innovation, 18–20 November. © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 7 IV. Five Key Ingredients of Blended Learning for Adult Audiences Our research identified five prevalent themes in blended participants’ accounts of their experiences with New Leaders’ programs: Accountability, Prior Experience, Motivation, Self-Regulation, and Relevance. These themes are not new; decades of research in adult learning previously revealed the importance of similar principles when teaching adults. In general, best practices in adult blended learning mirror strong in-person learning experiences — albeit redefined, tweaked, and combined in interesting ways to accommodate participants’ use of technology in the learning process. The five themes outlined below describe aspects of participants’ experiences in New Leaders’ programs that are distinct but often interdependent. For example, a participant who values the relevance of the program’s content to her work might recognize that in order to maximize her learning she will need to deploy self-regulation strategies to manage her time and engagement effectively. Likewise, participants who are motivated by the opportunities to engage with their peers in rich discussions of content assigned as pre-work can find group sessions to be an effective accountability mechanism, which in turn encourages them to use self-regulation strategies to complete assignments in advance. 1. Accountability “Now we are being held accountable for [our learning in the program] because whatever work we bring back [to our teams], we created it, where before, we’d read this and report out on it. That’s completely different from actually going and seeing how is that going to look.” “C,” PARTICIPATING TEACHER One tenet in teaching adult learners is to allow them to assume responsibility for their own learning. Of special significance in adult professional development is how individuals see their efforts contributing to the benefit of the group’s goals. In online and blended learning environments, this type of cooperation can be cultivated by learners as they share goals, resources, roles, tasks, and accomplishments.20 When participants in New Leaders’ blended programs felt that their “The session plans assume that the supervisors or districts engaged with and supported New Leaders participants are already familiar with programming, they described feeling accountable for the work they the basic concepts that are being had done in the program. covered, via the pre-work. Everyone In contrast, in one engagement none of the interviewed participants in this cohort did the pre-work. mentioned being accountable to their supervisors for how they That’s the real potential, maximizing implemented what they’d learned during the New Leaders summer the opportunities for the in-person workshops. District administrators gave building leaders autonomy for sessions, with informed role-play how they led their teams, and most participants did not use the online resources during the school year, in sharp contrast to how participants or other activities.” behaved in the other four regions. NEW LEADERS PROGRAM IMPLEMENTATION STAFF MEMBER New Leaders staff involved in blended engagements recognize the importance of participant accountability, and that participants will get out of the program only what they put into it. 20  Heba, E. D., & Nouby, A. (2008). Effectiveness of a blended e-learning cooperative approach in an Egyptian teacher education programme. Computers & Education, 51(3), 988–1006. © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 8 2. Prior Experience With Online Learning Studies of blended learners show a positive relationship between “And in the audience we have some previous experience with online tools and beliefs about their people who are technologically own abilities, or self-efficacy, in learning outcomes.21 Yet, for savvy. We should decrease stuff on adult learners, age is widely perceived as negatively influencing hard paper and increase our use of success in online and blended learning, due to course designers’ beliefs about technological illiteracy among older adults.22 Some technology. But some people prefer surveys show that younger professionals with greater exposure old-fashioned paper-and-pencil. to online education report less skepticism of blended and online So we need to be malleable.” learning than their older peers, whereas others suggest that a gap in technological experience and mindset is closing among students NEW LEADERS PROGRAM IMPLEMENTATION of different ages.23 Conversely, when given a choice of instructional STAFF MEMBER modes, older adults are more likely than their younger peers to select online learning for its flexibility. Older adult learners are also more likely to seek help and engage with peers online, behaviors that are associated with positive outcomes in these environments.24 Interviews conducted by New Leaders researchers indicated that most blended partici- pants were experienced online learners. For example, many participants mentioned prior experience with online learning in coursework for advanced degrees and professional certifications, which helped them know how to plan for adequate time and space to engage with the online instructional content. In one engagement, participating principals and some of their staff members had prior experience with New Leaders programming; they mentioned that their experience with the New Leaders model, vocabulary, and tools made it easier to engage with the blended program because they knew what to expect and how to implement what they were learning into their practices. 3. Self-Regulation Adult students in online classes who report better self-regulation (e.g., setting goals, monitoring their understanding, and adapting their thinking) realize greater academic achievement.25 Meanwhile, students who earn failing grades in online courses cite a lack of study time and absenteeism as primary causes.26 Communicating a blended program’s daily, weekly, and monthly time requirements up front in a clear fashion allows participants to plan accordingly and to thoughtfully schedule when they will engage with online content and undertake independent assignments. Effective blended programs will facilitate participants’ planning with regular communication about the program’s timeline, and with well-placed reminders about upcoming deadlines. The established structure and expectations of the Emerging Leaders program allowed many participants to plan and schedule their work. 21  Shea, P and Bidjerano, T (2010). Learning presence: Towards a theory of self-efficacy, self-regulation, and the development of a communities of inquiry in online and blended learning environments. Computers & Education, 55, 1721-1731. 22  Wolfson, N. E., Cavanagh, T. M., & Kraiger, K. (2014). Older adults and technology-based instruction: Optimizing learning outcomes and transfer. Academy of Management Learning & Education, 13(1), 26–44. 23  Betts, L. R., Hill, R., & Gardner, S. E. (2017). “There’s not enough knowledge out there”: Examining older adults’ perceptions of digital technology use and digital inclusion classes. Journal of Applied Gerontology, October 2017. 24  Deschacht, N., & Goeman, K. (2015). The effect of blended learning on course persistence and performance of adult learners: A difference-in-differences analysis. Computers & Education, 87, 83–89. 25  Moos, D. C., & Ringdal, A. (2012). Self-regulated learning in the classroom: A literature review on the teacher’s role. Education Research International, 2012, 1–15. 26  Mansur, A. B. F., & Yusof, N. (2017). Comparative classification of student’s academic failure through social network mining and hierarchical clustering. Computer Engineering and Applications, 6(2), 79–86. © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 9 “I learned early on that I have to put in my calendar a week in advance before pre-work is due to make sure that I get it done. So it’s definitely something you have to learn, that you can’t open up on Friday and think you’re going to get everything finished.” “D,” PARTICIPATING TEACHER Nevertheless, finding the time to engage with online instructional content and complete assignments was a challenge for some participants in New Leaders’ blended programming: “I knew I had to do the assignments, [but we] didn’t know it was an ongoing thing. Because of that, it was kind of like, ‘Oh, by the way, you’re going to a week-long thing in the summer.’ And so, we did that. Then it was like, ‘Oh, and you’ve got these books to read and these assignments.’ That has been challenging and really stressful.” “P,” FLEXIBLE-PROGRAM PARTICIPANT 4. Motivation While self-regulation creates the conditions for blended learners “I think almost every conversation to successfully complete a course, motivation provides the impetus I started with [my coach] was, to engage in these behaviors.27 Comparisons of student survey data ‘Remember, I have no idea what and course grades show that motivation corresponds with perfor- I’m doing.’ And she was always very mance in blended learning courses.28,29,30 The field of adult learning theory often takes for granted that adult learners are self-motivated, understanding about it, and she but the participation in flexible engagements organized by districts didn’t let me make excuses. Even or other supervisory bodies is often a mandatory part of a given role, though I really wanted to, she didn’t even if it is presented to participants as optional. In-service teachers let me say, ‘Well, that’s fine.’ She and administrators may believe in the importance of ongoing profes- definitely pushed me, and I was sional development, but it may not motivate them over long periods if the training does not carry over into actual practice.31 appreciative of that, and being very compassionate and understanding Across all engagements, program participants often mentioned of how I felt, and especially feeling their motivations for engaging with New Leaders’ blended content: stretched so thinly at times.” personal growth, professional growth, and achievement of team goals for student outcomes. The virtual and in-person coaching that “L,” PARTICIPATING TEACHER participants received was often cited as a very influential motivator. “Before the training, you know, I just saw things from the perspective of a teacher of what’s happening in the classroom and why is admin doing this or why are they doing that over there. Now that I’m admin and went through the training and seeing everything for myself, I see all the different levels and how all the different pieces make a school work.” “M,” PARTICIPATING ASSISTANT PRINCIPAL 27  Baird, D. E., & Fisher, M. (2005). Neomillennial user experience design strategies: Utilizing social networking media to support “always on” learning styles. Journal of Educational Technology Systems, 34(1), 5–32. 28  Garrison and Kanuka (2004). 29  Lim, D.H. & Morris, D.L. (2009), Learner and Instructional Factors Influencing Learning Outcomes within a Blended Learning Environment. Educational Technology & Society, 12 (4), 282–293. 30  So, H.J. & Brush, T.A. (2008). Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment: Relationships and critical factors. Computers & Education, 51 (2008) 318–336 31  Baker, J., Chaseling, M., Boyd, W., & Shipway, B. (2018). Teachers’ response to a new mandatory professional development process: Does it make a difference? Professional Development in Education, 44(4), 570–582. © 2019 New Leaders Inc. All rights reserved. | BLENDED LEARNING AT NEW LEADERS 10