understanding the “how” of quality improvement Lessons from the Rhode Island Program Quality Intervention by Elizabeth Devaney, Charles Smith, and Kenneth Wong Over the past 10 years, afterschool and youth develop- and work their way up to management positions with- out receiving training or education in how to lead an ment programming has moved from providing childcare organization. They may not see themselves as instruc- for working parents to being an integral component of ELIZABETH DEVANEY is an independent consultant working pri- the learning day, supporting the academic, social, and marily with expanded learning, afterschool, and youth development organizations. Before starting her consulting practice, she was dep- emotional development of young people (C. S. Mott uty director and quality improvement director of the Providence After School Alliance in Providence, Rhode Island, and project director at the Foundation, 2007; Durlak & Weissberg, 2007). An im- Collaborative for Academic, Social, and Emotional Learning (CASEL). She was the recipient of a W. T. Grant Foundation Distinguished Fel- portant part of that transition has been a growing em- lows award in 2009. CHARLES SMITH, Ph.D., is executive director of the David P. Weikart phasis on improving program quality. Many communi- Center for Youth Program Quality, a division of the Forum for Youth ties around the country have begun to create site-level Investment (FYI), and the vice president of research for FYI. He was the principal investigator for the recently completed Youth Program continuous improvement models (Wilson-Ahlstrom & Quality Intervention study, a randomized trial designed to raise the Yohalem, 2008; Yohalem & Wilson-Ahlstrom, 2009). quality of instruction in out-of-school time by building managers’ Aligned performance measures help program adminis- continuous improvement skills. KENNETH WONG, Ph.D., is the Walter and Leonore Annenberg Chair trators evaluate the quality of young people’s experience for Education Policy and the chair of the Education Department at and give them a framework for improvement. Brown University. While holding joint appointment with the Education Many of these quality interventions target the lead- Department and the Annenberg Institute for School Reform, he directs ers of afterschool organizations rather than simply di- the master’s program in urban education policy. A political scientist by recting attention to the teaching staff. Afterschool pro- training, he has conducted extensive research in urban school reform, gram managers often start their careers as front-line staff state finance and educational policies, and federal education policy. tional leaders and may not have training in how to Assessment (HighScope, 2005) and a locally developed change the direction and design of their organization or administrative checklist. The RIPQA was piloted and how to develop the people who work for them. rolled out statewide in 2006. Since then, PASA has part- The literature on school leadership and climate nered with the Rhode Island After School Plus Alliance change highlights why a leader-focused approach makes and the 21st Century Community Learning Centers ini- sense. Researchers have demonstrated that improvements tiative at the state Department of Education to create an in school leadership can lead to improved teaching capac- improvement system—the Rhode Island Program ity and therefore to improved student achievement. In Quality Intervention (RIPQI)—with the assessment tool their meta-analysis of 70 studies of principal leadership, at its center. Close to 100 organizations across the state Waters, Marzano, and McNulty (2003) identified 21 sep- are engaging in the process, including all 65 of the 21st arate responsibilities of school instructional leaders, from Century Community Learning Centers. The RIPQI in- fostering a culture of shared beliefs to establishing order cludes the following activities: to providing resources and professional development. • Training in the use of the RIPQA Improvements in a leader’s ability to perform these re- • Observation of individual program offerings at the sponsibilities were linked to im- point of service (where youth proved student achievement. Other Researchers have and adults interact) by teams of reviews of the research have simi- demonstrated that impartial external advisors and larly found that school leaders have internal staff improvements in school a responsibility to set direction, de- • Assessment of management prac- leadership can lead to velop people, and redesign the or- tices including staffing and pro- ganization in order to achieve im- improved teaching fessional development supports, proved student outcomes capacity and therefore to family and community engage- (Leithwood, Seashore Louis, ment, and administrative practices improved student Anderson, & Wahlstrom, 2004). • Quality improvement planning achievement. Our study looks at how par- with the support of a trained ticipation in a continuous quality quality advisor improvement initiative produces • Five hours of on-site technical higher-quality practice in Rhode Island’s afterschool assistance connected to the quality improvement plan community by fostering change in program management • Participation in optional training aligned with the practices. Among other findings, we discovered that RIPQA quality improvement begins with program managers, who then lead the process of change. Each participating organization is paired with an ex- pert “quality advisor” or coach for up to 25 hours every The Rhode Island Program Quality Intervention other year to complete the RIPQI process. First, a team of The Youth Program Quality Intervention (YPQI), devel- program staff uses the administrative checklist (RIPQA oped by the David P. Weikart Center for Youth Program Form B) to rate the organization on various administra- Quality, is one intervention focused on program managers tive practices. The advisor helps the team to arrive at con- that is being used in communities across the country sensus about strengths and areas for growth and to de- (Smith et al., 2012). YPQI is a multi-level intervention that velop a quality improvement plan with specific action uses continuous improvement practices to increase stu- steps. Following this administrative audit, the advisor and dent exposure to positive youth development methods. the site director put together teams to observe three to In Rhode Island, development of a statewide quality five program offerings using the Weikart Center’s Youth improvement system based on YPQI began in 2004, Program Quality Assessment (RIPQA Form A). Again, the when the Wallace Foundation awarded a large grant that teams come to consensus and develop an action plan con- allowed for the establishment of an afterschool interme- taining specific steps for improvement. Often these action diary—the Providence After School Alliance (PASA)— steps include sending staff to PASA trainings to improve and made quality an explicit priority. In partnership with specific skills. The quality advisor participates in observa- the Weikart Center, PASA created the Rhode Island tions, guides the site through this entire process, and then Program Quality Assessment (RIPQA), a tool comprising provides five hours of technical assistance or training in the Weikart Center’s validated Youth Program Quality support of the site’s quality improvement action plan. 2 Afterschool Matters Fall 2012 MANAGEMENT PRACTICES AND POLICIES RIPQI Organizational context • Training • Leadership style POINT OF SERVICE • Point-of-service • Climate and culture observations • Structure/governance Improved • Administrative Instructional checklist Administrative practices quality Improved • Quality • Staff development and CHANGE LEADS TO CONTRIBUTES TO youth improvement support Increased outcomes planning • Policies and procedures youth • On-site technical • Administrative engagement assistance functions • Optional Family & civic professional engagement practice development Figure 1. RIPQI Theory of Change PASA has developed a theory of change that gov- valid assessment tool and associated coaching and tech- erns this intervention, shown in Figure 1. The first box nical assistance can have positive effects on the quality on the left represents the intervention itself and the ele- of instructional and management practices in after- ments that comprise it. As the or- school programs (Sinisterra & ganization begins to engage in the Baker, 2010; Smith, Akiva, The Weikart Center, in a intervention, the program manager Blazevski, Pelle, & Devaney, begins to make changes to his or rare experimental study of 2008). The Weikart Center, in a her practice that in turn affect the a continuous improvement rare experimental study of a con- whole organization. This im- tinuous improvement interven- intervention in an provement leads to changes at the tion in an educational context, educational context, point of service. As instructional examined the effectiveness of the quality improves and youth are examined the effectiveness YPQI in 87 afterschool programs more engaged, we expect to see of the YPQI in 87 in five states. Results show that the improvement in youth out- the YPQI had a substantial and afterschool programs in comes the intervention was de- statistically significant effect on five states. Results show signed to produce. both the continuous improvement that the YPQI had a PASA is not the only organi- practices of site managers and the zation to create a quality improve- substantial and statistically instructional practice of front-line ment system based on the YPQI. significant effect on both staff (Smith et al., 2012). Both In fact, to date, more than 70 studies provide critical context for the continuous communities around the country understanding the likely effects of improvement practices of are implementing all or some the RIPQI on manager and staff components of this model,1 pro- site managers and the practice. Our study focuses on viding substantial evidence of ef- instructional practice of how these effects occur, notably in fectiveness. For example, in Palm the words of site managers en- front-line staff. Beach County, Florida, the inter- gaged in the RIPQI process. mediary organization Prime Time Palm Beach County has been implementing a quality Study Overview improvement system based on the YPQI for the past The goal for this study was to test the validity of the theory five years. A recent study of that model demonstrated of change presented above, using two guiding research that a quality improvement system centered around a questions: Devaney, Smith, & Wong UNDERSTANDING THE “HOW” OF QUALITY IMPROVEMENT 3 1. Does the RIPQI process produce change in organiza- Hohmann, 2005), one component of the RIPQA. tions? Observations were conducted during individual program • Is implementation of the RIPQI related to change in offerings over multiple sessions involving the same staff, the quality of instruction and child engagement? the same youth, and the same purpose. Each required at • Is implementation of the RIPQI related to change in least 45 minutes of observation by a reliable rater. The organizational context, administrative practices, and tool measures instructor practice in four key domains: family engagement practices? safe environment, supportive environment, interaction, 2. How does change happen? and engagement. Each domain has several indicators. • What practices do managers employ that may con- Instructors are rated on a three-point scale using a rubric. tribute to change at their site? Two surveys were used for this study, one designed • How do managers transfer, adapt, and extend the for managers, including site coordinators and other ad- RIPQI in organizational settings? ministrators, and one designed for front-line staff who • In what ways are site managers affected by imple- work directly with youth. The surveys were modeled af- mentation? ter those used in the YPQI study, described above, in an effort to create items and subscales in line with the The first set of questions focuses on the first three boxes known reliability and validity of those tools. The re- in the theory of change: implementation of the intervention, sponse rate was about 40 percent for the manager survey, change at the program level, and changes in instructional with 29 managers, representing 21 of the 53 sites, re- quality and youth engagement. The second set of questions sponding. The staff survey had a 26 percent response explores what happens in the spaces between boxes to rate; the 33 staff members who responded represented make change happen. The last step in the theory of change, 14 out of the 53 sites. the effect on youth outcomes, was beyond the scope of Finally, the interviews were conducted using a stan- this study but is an important area for future research. dardized open-ended approach. Lasting about one hour, To answer these questions, we used a mixed-methods they included 15 questions in four key domains: changes approach, employing data from 53 afterschool programs to practice, accountability for implementing change, across Rhode Island funded by the 21st Century changes to leadership style, and overall program improve- Community Learning Centers office at the state ment. Five of the six managers interviewed were chosen Department of Education. Sites are required to partici- because they reported high levels of RIPQI implementa- pate in the RIPQI process every other year; at the time of tion on the survey. We also attempted to use the survey to the study, every site had participated at least once. The identify a low-implementing manager. Only one individ- 53 sites are distributed across the state, with a large con- ual had low enough scores to merit consideration as a centration in Rhode Island’s five “core” cities—Central contrast to the others, but her interview revealed that she Falls, Pawtucket, Newport, Providence, and Woonsocket. reported low levels of implementation and change be- The sites serve all age groups with about half (54 percent) cause she had recently completed the process and had not serving elementary-age students and the remainder serv- yet conducted extension activities or seen change happen ing middle (30 percent) and high school (16 percent) stu- at her site. We therefore simply included this manager’s dents. Sites range in size from 15 to 200 students per day. feedback with that of the other five. Data Sources Data Analysis Our study uses the following sources of data: To analyze the data, we first looked at each data source • Existing instructional quality data collected by trained individually and then began to link sources to answer the observers during 2007–2010 (n = 325 program obser- two research questions. Beginning with observational vations) data, we identified 13 sites with data for two individual • Surveys with program staff (n = 62) and managers program offerings at each of two time points in different (n = 29) program years. We aggregated each site’s ratings for each • In-depth interviews with a subset of managers (n = 6) time point and then compared the two time points to de- who reported a high level of RIPQI implementation scribe an average amount of change for each site. Next we identified 21 instructors from different sites who had ob- Observations were conducted using the Weikart servational ratings in different years and then compared Center’s instrument (for validation evidence see Smith & the two time points to describe an average amount of 4 Afterschool Matters Fall 2012 change for each individual. For each of these samples, the to the time after the RIPQI had been introduced? On the RIPQI had been carried out between the two time points. whole, the answer was yes, although the small sample For survey analysis, we created several subscales from size reduced the power to detect statistically significant the survey items and ran cross-item and cross-survey anal- differences. In nearly all cases, scores went up from the ysis to understand the characteristics of individuals and baseline to the second observation. For the subsample of organizations who reported high and low levels of change 13 sites on which we had instructional quality data at in program practices and higher and lower levels of youth two time points, differences in observed quality were engagement as a result of the RIPQI process. positive, particularly in the domain of safe environment, Finally, to analyze the interview data, we first read where we saw statistically significant change. through each interview transcript, looking for any practice Our best test of baseline-to-post–RIPQI change is or performance changes reported by the site managers we for the 21 individual instructors who were observed interviewed. We then identified key themes that emerged doing the same program at two time points, with across all the interviews. exposure to the RIPQI in between. In these cases, the average score Youth program staff Does Change Happen? change was large and statistically Analysis of the observational as- across Rhode Island significant. While scores improved sessment data, combined with staff reported that in all four key domains, statistically and manager reports on the sur- significant change occurred in the administrative practices veys and interviews, suggest that total score as well as in two and instructional the RIPQI is working. To begin domains: supportive environment experiences are with, we saw high levels of fidelity and interaction. to the RIPQI across all sites, in part improving as a result of because many of its elements are the RIPQI. How Does Change Happen? required. We quantified the level of That the RIPQI is working was implementation by asking managers one question this study set out to and staff about their participation in various elements of answer. Our findings provide evidence supporting the the intervention, such as attending training, conducting theory of change. When fully implemented—that is, observations, assessing administrative practices, and en- when staff embrace the process and engage in activities gaging in quality improvement planning. Out of a total of beyond what is simply required—the RIPQI does appear 21 elements, nine required and 12 not required, the av- to produce measurable change in instructional practice. erage number in which managers participated was 10.5. These findings serve to confirm with local data what More than half (55 percent) participated in 11 or more the more rigorous studies described above suggest: elements. We further distinguished high implementers that the investment in the RIPQI has produced gains from low implementers by looking at the 12 optional or in the quality of afterschool programs across the state. “extension” activities, which required additional effort on However, perhaps the more intriguing finding from this the part of managers and staff. On average, managers par- study is how the RIPQI is working. If we understand the ticipated in 6.75 of these extension activities. how, we can improve training for sites and better prepare Youth program staff across Rhode Island reported quality coaches. that administrative practices and instructional experiences The manager interviews allowed us to further ex- are improving as a result of the RIPQI. Managers and staff plore how administrative practices support changes to reported almost universally (97 percent of managers and instruction. Across the interviews, several themes 81 percent of staff) that the RIPQI produced positive emerged regarding how the RIPQI changed management change in program quality. Fully 72 percent of managers practices and policies. and 67 percent of staff reported that the RIPQI supported youth to become more engaged in program offerings. Changes to Manager Practice In analyzing the observational data, we looked at the Managers reported changes in how they viewed or car- subsamples of 13 sites and 21 individual instructors from ried out their roles. Several talked about being more different sites who had observational data before and af- comfortable in the role of instructional leader, being ter participation in the RIPQI. We asked the simple ques- more able to provide feedback to program instructors, tion: Was there positive change from the first observation freeing up time to provide better supervision, and in gen- Devaney, Smith, & Wong UNDERSTANDING THE “HOW” OF QUALITY IMPROVEMENT 5 eral being more intentional about how they ran their One manager talked about a change she made to the sites. For example, one manager reported: program schedule to allow for more communication I used to just think that I’d hire the facilitators and among staff: they’d know what to do and how to interact with Every day, as an entire group, we meet at 2:00—ev- kids, or teachers would come on board and it’d be, ery single day. And we have a check-in about the “Oh, they’re a school teacher so I don’t really have to day, talk a little bit about the logistics of the after- tell them anything,” but…I think I’m more comfort- noon…and then, there’s that open hour…where ev- able speaking up to facilitators now, and I do it more eryone is paid to be at work to…do lesson planning, often…. I think that I [have] become a stronger lead- get their snacks ready, meet with each other…or er because I’m more intentional about the supervi- someone will have scheduled a time to come and sion and the feedback that I meet with me one-on-one. But we give people in the observa- have that hour set aside every day. One manager explained tions that I do with them. that the RIPQA “gave me Another talked about how she Another core part of changing some way to structure my uses the RIPQA to help herself and the manager role was improving ori- her team set goals: trainings with my staff, entation and training for staff. All six During my supervision that I and it gave them a managers mentioned this element have with them on a monthly and described how incorporating the structure of how to think basis, that’s one of the talking RIPQA into their training allowed about their time in the points that we have, is the them to better prepare their staff to RIPQA process. And how do I classroom with students.” meet expectations. One manager ex- feel that it’s going, are there plained that the RIPQA “gave me any issues going on, is there some way to structure my trainings with my staff, and it gave anything that I think we can strive [for]…. them a structure of how to think about their time in the class- I’m constantly setting goals every month, talking room with students.” Another talked about how she selected about the goals that we’ve met. one indicator from the RIPQA to discuss at each staff meeting. Improvements in Program Structure Improvements to Communication Several managers reported that the RIPQA process pro- One of the most prevalent themes that came out of the vided them with a practice that helps them shape and interviews was improvement to communication at all make improvements to the overall program structure. As levels. Managers reported better communication between one manager put it, the RIPQA “creates procedure. It cre- site managers and staff, site managers and their supervi- ates form. It creates a structure that you can work within sors, the program and parents, and instructors and youth. that is still flexible.” Another reported: Communication was defined broadly but included some I think the biggest thing is just offering a structure of the following types of changes or improvements: that seems to really work...There’s just so much • Improved policies and procedures, clarifying to every- when you’re trying to hire, and staff, and train youth one what was expected of participating youth and staff workers coming from such different backgrounds. I • More intentional and more regular supervision of staff, cannot assume that they’re coming in with a certain including not only formal supervision but also more skill set, and [the RIPQA] has kind of allowed me to informal observations, check-ins, and meetings structure our programming and structure the way • Improved communication with youth, including more we think about how we interact with students. opportunities for youth to voice their opinions and have a say in the program structure through, for ex- The changes to program structure that resulted ample, youth advisory councils, student focus groups, from the process were not purely abstract. One manager and one-on-one conversations with instructors talked more concretely about how the process helped • Better staff meetings that took advantage of the RIPQA her restructure her program: tool and its core indicators One of the main things that came out of our RIPQA • Improved staff connections to family and community, process two years ago was that we were working our more parental involvement kindergarten and first graders way too hard…. We 6 Afterschool Matters Fall 2012 restructured our K and 1 program based on the part? How can we make it better?” —that was some- RIPQA process…. And there was a huge change in thing that kids really did come to me and say, “Hey, behavior, and meltdowns, and kids passing out at guess what? We told them we didn’t like this class 4:30—just falling asleep because they were so ex- this day and they’re going to change it.” And that hausted because we worked them so hard. was a big thing. Improvements in Hiring Practices and The How of Program Improvement Staff Composition These findings describe changes to administrative and A fourth theme that came out of the interviews was the management practices that can lead to improved instruc- impact the RIPQA process had on the composition of the tion and increased youth engagement as described in the staff. All interviewed managers talked about such efforts theory of change. When fully implemented, the RIPQI as rewriting job descriptions to better reflect the quality does appear to produce significant change in instruction- standards, making changes to the organizational chart to al practice, as measured by the observations, as well as in allow for better staffing, creating assistant director posi- greater youth engagement, as reported by managers and tions in order to free up the site manager to spend more staff. In addition, it appears to have an effect on manage- time on quality improvement, and firing staff or using ment practice, as described by the program managers natural transition to eliminate staff who were not com- interviewed for this study. mitted to reflective practice and improvement. One man- So what is actually going on at the site level that ager described this last kind of change: makes change happen? Taken together, the observational After about a year of RIPQA, when I realized that data, survey responses, and interview transcripts begin there were some staff that were either apprehensive to tell a story that mirrors the theory of change laid out or completely just holding back from being a part of above. That is, sites appear to go through a flow of activ- this and moving forward like we were, they were not ity that starts with structural change and ends with im- asked to come back to work this year… They may be provements to instructor practice: wonderful youth workers, but if they’re not aligned with the vision and the needs that your school has, Structural change: then it’s just not the right fit anymore. administrative practices, hiring and firing, policies Improvements in Instruction Of course the changes listed above are valuable only if Organizational and climate change: they eventually have a direct impact on the experience communication, training of young people in the program. Although at least two of the managers felt that the changes they were making had not yet led to improved instruction, others talked Manager-level change: about what they saw changing for the young people in becoming instructional leaders their programs, including: • Improvements to the safety and environment includ- ing more secure entrances and sign-out procedures, Changes to instructional practice: more appropriately sized furniture, and better fire drill youth experience in the program procedures. • Improvements to the quality of interaction between Structural Change the youth and the staff. For example, staff asked more It appears that change begins at the higher levels of open-ended questions and were more intentional administration. The RIPQI provides a framework and about greeting each student; youth voice was solicited context for getting the right staff in place to do the through time built in for feedback and reflection. right jobs. By revising job descriptions, hiring more in- tentionally, firing staff who aren’t a good fit, creating One manager reported on how the program elicited new policies and procedures, and shifting job duties, youth voice: organizations ensure that their staff members are strong The ten-minute, five-minute check-in at the end of a and committed and that they understand exactly what class, “How did this go for you? What’s your favorite is expected of them. Devaney, Smith, & Wong UNDERSTANDING THE “HOW” OF QUALITY IMPROVEMENT 7 Organizational and Climate Change mendations and feedback that lead to the final product: Once the right mix of staff is in place, the RIPQI seems to improvements to instructional practice. provide the staff with a framework for improved com- munication. This communication takes several forms, Changes to Instructional Practice but the most common changes seem to be dramatic im- The final stage in the theory of change that our study ad- provements to orientation and training for new and re- dressed is improvement to instructional practice. This turning staff and more intentional staff meetings. By us- process of change—improving the staffing mix; creating ing the RIPQA to shape new orientations, managers a shared language and common goals; and more inten- ensure that all staff members have a set of shared stan- tionally supporting staff through improved communica- dards to work from and can therefore establish common tion, training, and supervision—takes time. Sites that goals. After establishing a common language at the be- have engaged in the process longer or that have strong ginning of the year, managers then leaders are further along than oth- used aspects of the RIPQA through- As managers become ers. Several managers, but not all, out the year at staff meetings. The did report change at the instructor more comfortable giving standards provided managers with level. Many of the changes manag- feedback, they are likely a structure around which to shape ers described were basic and rela- meetings intentionally. not only to observe their tively easy to achieve, such as The survey data tell us that staff, but also to provide greeting all youth warmly, improv- staff who are most likely to report ing the appropriateness of furni- recommendations changes in their practice are those ture and supplies, and creating a and feedback that lead to who feel most supported by their sense of belonging. However, a few supervisors and who feel they un- the final product: managers referred to development derstand the shared goals of their improvements to of higher-order skills among their organization. It stands to reason, instructors, such as asking more instructional practice. then, that, as the climate of the or- open-ended questions, providing ganization becomes more inten- opportunities for youth to reflect tionally aligned with the RIPQA and staff are receiving on the program, and doing more intentional planning. more training and better support through ongoing super- vision and staff meetings, they will feel more supported Study Limitations and therefore more inclined to enact change. This study has several important limitations. For one, it used existing, but incomplete, observational data col- Manager-Level Change lected as part of a quality improvement system. Not ev- As managers become more certain of their staffing mix, ery site had a complete set of observational data at two create a shared language for the staff, establish clearer time points. We based our analysis on those that did. policies and procedures, and develop an infrastructure A second limitation is the small sample size. As noted for intentional staff meetings, they begin to feel more above, we had a relatively low response rate on the staff confident as instructional leaders. Every manager talked and manager surveys, probably because we distributed about continuing to conduct informal and formal obser- them in June, when many programs were breaking for the vations after the official RIPQI process was over. These summer. By design, interviews were conducted with just managers now have language for giving staff feedback on six individuals. With more time and better response rates, their performance. Many also talked about establishing the data might have yielded different findings. more regular and intentional supervision with their staff, A final limitation is that the study was conducted by using the RIPQA as a guideline. When staff are hired and someone very close to the RIPQI process. Elizabeth trained using a common language, managers can more Devaney created the RIPQI in partnership with the easily provide guided support for their practice. The sur- Weikart Center and has been largely responsible for its vey data suggest that the front-line staff most likely to growth and development into a quality improvement sys- change their practice are those who are involved deeply tem in Rhode Island. She is not an impartial researcher. in the quality improvement process. As managers be- Those surveyed and interviewed knew Elizabeth well and come more comfortable giving feedback, they are likely may have tailored their responses to her. However, her not only to observe their staff, but also to provide recom- closeness to the sites was also a benefit because she was 8 Afterschool Matters Fall 2012 intimately familiar with the RIPQI and the nuances of Acknowledgements implementation. The authors are thankful to the W. T. Grant Foundation Further research is needed to confirm the validity for awarding the Distinguished Fellows grant that made of these findings and to explore what effect additional this research possible. We also thank Samantha Sugar of factors, such as the experience level and education of the Weikart Center for her data analysis support; Shevaun the manager, the longevity of staff, and the program set- Keogh-Walker, Maryclaire Knight, and Kuniko Yasutake ting, may have. Although these findings mirror the edu- for data collection; and the program managers who gave cation literature on administrator effect on teacher their time to be interviewed for this project. Elizabeth practice (Leithwood et al., 2004; Marzano et al., 2003), would also like to thank Dr. Charles Smith and Dr. there may be other ways to understand the flow of fac- Kenneth Wong for serving as mentors throughout the fel- tors that affect instructor improvement, including the lowship and for their guidance on this project. She also effect of formal education and training. The field would thanks Hillary Salmons of the Providence After School benefit from additional research Alliance for providing leave time to exploring the pathways to instructor pursue this fellowship. Managers who can improvement. translate a one-time References Implications assessment and quality C. S. Mott Foundation. (2007). A The purpose of the study was to improvement process into new day for learning: A report from the gain a better understanding of how time, learning, and afterschool task an ongoing, embedded the RIPQI achieves effects on after- force. Washington, DC: Collaborative system of continuous school organizations in Rhode Communications Group. Island. Its findings have implica- improvement are going to Durlak, J. A., & Weissberg, R. P. tions both locally and nationally. be more successful than (2007). The impact of after-school Locally, these findings suggest that those who can’t. programs that promote personal and Rhode Island’s quality improve- social skills. Chicago, IL: ment system is working but is Collaborative for Academic, highly dependent on administra- Social, and Emotional Learning. tors embedding the process and the language of the HighScope Educational Research Foundation. (2005). RIPQA into their organizations. Managers who can trans- Youth program quality assessment: Administration manual. late a one-time assessment and quality improvement pro- Ypsilanti, MI: HighScope Press. cess into an ongoing, embedded system of continuous improvement are going to be more successful than those Intercultural Center for Research in Education & the who can’t. Knowing that, RIPQI decision makers may National Institute on Out-of-School Time. (2005). want to redesign training for new sites and quality coach- Pathways to success for youth: What counts in after-school. es to include strategies for embedding the process into Massachusetts afterschool research study (MARS) report. ongoing program planning. For example, bringing suc- Wellesley, MA: Author. cessful managers into the training to share lessons Leithwood, K., Seashore Louis, K., Anderson, S., & learned and promising strategies may improve imple- Wahlstrom, K. (2004). How leadership influences student mentation at new sites. learning: Review of the research. New York, NY: Wallace Nationally, this study can inform communities that Foundation. are developing and launching quality improvement sys- Sinisterra, D., & Baker, S. (2010). A system that works: tems based on the same or similar tools and practices. A Highlights of effective intervention strategies in a quality clear lesson from this study is that focusing on managers improvement system. Afterschool Matters, 12, 37–44. at the start may be more effective than moving directly Smith, C., Akiva, T., Blazevski, J., Pelle, L., & Devaney, to individual instructors. Without a shared language T. (2008). Final report on the Palm Beach Quality and infrastructure for discussing quality improvement, Improvement System pilot: Model implementation and instructor-level change may not happen or may be program quality improvement in 38 after-school programs. short-lived at best. Change seems to happen on a con- Ypsilanti, MI: HighScope Educational Research tinuum that begins with the administration. Foundation. Devaney, Smith, & Wong UNDERSTANDING THE “HOW” OF QUALITY IMPROVEMENT 9 Smith, C., Akiva, T., Sugar, S. A., Devaney, T., Lo, Y-J., Frank, K., Peck, S., & Cortina, K. (2012). Continuous quality improvement in afterschool settings: Impact findings from the Youth Program Quality Intervention study. Ypsilanti, MI: David P. Weikart Center for Youth Program Quality. Smith, C., & Hohmann, C. (2005). Full findings from the Youth PQA validation study. Ypsilanti, MI: HighScope Educational Research Foundation. Waters, J. T., Marzano, R. J., & McNulty, B. A. (2003). Balanced leadership: What 30 years of research tells us about the effect of leadership on student achievement. Aurora, CO: Mid-continent Research for Education and Learning. Wilson-Ahlstrom, A., & Yohalem, N. (2008). Speaking in on voice: Toward common measures for OST programs and systems. Washington, DC: Forum for Youth Investment. Yohalem, N., & Wilson-Ahlstrom, A. (2009). Measuring youth program quality: A guide to assessment tools (2nd ed.). Washington, DC: Forum for Youth Investment. Note 1Use of other continuous improvement models is also growing. Notably, the Afterschool Program Assessment System (APAS) by the National Institute on Out-of- School Time (NIOST), which is being used in several communities around the country, draws on similar continuous improvement practices and principles. Validation evidence for the APAS is provided in the final report on the Massachusetts Afterschool Research Study (INCRE & NIOST, 2005). 10 Afterschool Matters Fall 2012