Formative Evaluation of the Student Achievement Initiative “Learning Year” Report to the Washington State Board for Community and Technical Colleges and College Spark Washington Davis Jenkins, Todd Ellwein, and Katherine Boswell October 2008 (Revised January 2009) Acknowledgments: Funding for this work was provided by College Spark Washington, by the Ford Foundation as part of the Bridges to Opportunity initiative and by Lumina Foundation for Education as part of Achieving the Dream: Community Colleges Count. The authors wish to thank Kevin Dougherty for his helpful comments on earlier drafts of this report. Address correspondence to: Davis Jenkins Community College Research Center Teachers College, Columbia University 525 West 120th Street, Box 174 New York, New York 10027 Tel.: 212-678-3091 Email: [email protected] Visit CCRC’s website at: http://ccrc.tc.columbia.edu Table of Contents Executive Summary......................................................................................................................1 1. Introduction...........................................................................................................................9 A. The Student Achievement Initiative..............................................................................9 B. Formative Evaluation of the Student Achievement Initiative “Learning Year”..........10 C. Organization of the Report...........................................................................................11 2. Background: The Policy Context for the Initiative.........................................................12 A. Performance Funding: A Tool to Ensure Responsiveness to State Priorities..............12 B. Past Experience with Performance Funding in Washington........................................13 C. Origins of the Student Achievement Initiative.............................................................14 D. Distinctive Features of the Initiative............................................................................15 3. Colleges’ Awareness and Understanding of the Initiative..............................................17 A. How College Personnel Learned About the Initiative.................................................17 B. Level of Awareness and Understanding of the Initiative among College Personnel...19 C. Understanding of the Origins and Goals of the Initiative............................................23 4. Colleges’ Initial Approach to the Initiative......................................................................26 A. Organization of Implementation Efforts......................................................................26 B. Use of the Student Achievement Initiative Database...................................................27 C. Strategies for Increasing Student Achievement...........................................................29 D. Factors Affecting Colleges’ Receptiveness to the Initiative........................................32 5. Colleges’ Views on the Initiative’s Performance Measurement Model.........................34 A. Support for the Initiative’s Performance Measurement Model...................................34 B. Concerns about the Model............................................................................................36 6. Colleges’ Views on the Initiative’s Incentive Funding Model.........................................38 A. Source of Additional Funding......................................................................................38 B. Concerns about Incentive Funding...............................................................................38 C. Views on the Amount of Funding Needed to Encourage Change...............................41 7. Higher Education Policymakers’ Views of the Initiative................................................42 A. Views of the Initiative among External State Policy Makers......................................42 B. State Board Views on the Initiative..............................................................................43 C. Funding the Initiative in a Challenging Fiscal Environment.......................................44 8. Conclusion...........................................................................................................................46 Executive Summary Launch of the Student Achievement Initiative In September 2007, the Washington State Board for Community and Technical Colleges (SBCTC) officially launched the Student Achievement Initiative, a system-wide policy to reward colleges for improvements in student achievement. Developed by a task force comprised of State Board members, college trustees, presidents, and faculty representatives, the policy emphasizes three overarching principles: 1) the initiative should lead to improved educational attainments for students and, in particular, should be aimed at boosting those educational attainments shown by research to be correlated with the earning of higher future wages by students; 2) the initiative should allow colleges sufficient flexibility to improve student achievement according to their local needs; 3) the initiative should result in the identification and implementation of successful practices to improve student achievement system-wide. Under the initiative, Washington’s community and technical colleges will receive financial rewards for increasing the rate at which they accrue “achievement points.” Achievement points are generated when students achieve key attainments in four categories. Colleges each points when students 1) achieve gains in adult basic skills and pre-college remedial courses; 2) complete a college-level math course; 3) earn college credits; or 4) complete a certificate, degree, or apprenticeship training program. The SBCTC staff chose the 2006-07 academic year as the baseline year for measuring the performance of colleges using the achievement point categories. The first “performance year” will be the 2008-09 academic year. In that year and in subsequent years, the performance of colleges will be assessed by comparing the total number of achievement points they generate in a given year with their baseline year point totals. The State Board is requesting $7 million from the legislature for the initiative during the 2009-11 biennium. The State Board designated the 2007-08 academic year as the initiative’s “learning year.” The learning year was designed to enable colleges to better understand their performance across the achievement point categories and to begin developing strategies for improving student attainments in achievement point categories. Each of the state’s 34 community and technical colleges received an allocation of approximately $51,000 in fall 2007 as start-up funding. Throughout the learning year, the State Board staff sought to raise awareness and understanding of the initiative, help colleges learn how to use data on their achievement point performance to identify areas for improvement, and promote sharing of student success strategies across campuses. CCRC Formative Evaluation of the Student Achievement Initiative “Learning Year” 1 The Initiative in Comparative Context Performance funding assumes that financial rewards will serve to change institutional behavior. By choosing measures to gauge college performance, policymakers can both motivate colleges to improve their performance and encourage them to prioritize their efforts according to particular goals or outcomes. In 1978 Tennessee was the first state to adopt a set of performance measures that specifically provided higher education institutions with financial incentives for improving student outcomes. Higher education performance funding reached peak popularity in 2001, when 19 states had implemented such policies. Several other states have experimented with performance funding over the years, but most now have performance reporting without attaching funds. It should be noted that more than a decade ago Washington State implemented a short-lived performance funding system for the postsecondary sector, which had minimal input or buy-in from the state’s colleges and universities. Although the present initiative for Washington State community and technical college shares some important characteristics with earlier performance funding efforts that have taken place across the country, it also has some distinctive features. These include the following: (cid:120) Rather than using a set of performance measures dictated by legislation or external policymakers, the initiative’s measures were recommended by a diverse task force of college representatives and were informed by research (conducted by the SBCTC and the Community College Research Center) on student progression that sought to identify key educational attainments associated with the increased probability of college and labor market success. (cid:120) By rewarding colleges for increasing the rate of intermediate educational attainments (in addition to completing credentials) in a broad continuum of mission areas, including adult basic skills and college remedial education, the initiative seeks to focus colleges’ attention on the progression of all students, regardless of where they begin. (cid:120) Unlike measures such as job placement rates, which are dependent on the economy and other factors, the task force chose performance measures that were more likely to be within a college’s control. (cid:120) The achievement point data supplied to the colleges is intended to enable them not only to measure their performance, but to identify areas for improvement and success and to evaluate the effectiveness of efforts made to improve student success. (cid:120) Colleges are not compared with one another to earn financial rewards, but rather are measured against their own historical performance. The Washington State Board’s efforts to design a set of student-focused performance indicators for community colleges has generated substantial interest from other states. The success or failure of the Student Achievement Initiative to support improved student outcomes will be watched closely. CCRC Formative Evaluation of the Student Achievement Initiative “Learning Year” 2 Formative Evaluation: Scope and Methods Researchers from the Community College Research Center (CCRC) at Teachers College, Columbia University, conducted an independent qualitative review of the Student Achievement Initiative during the 2007-08 learning year. The purpose of this formative evaluation was to assess the extent of awareness and understanding of the initiative among college personnel, examine the initial responses to it by the colleges, and identify opportunities for and potential barriers to the further development of the initiative. The CCRC research team conducted extensive interviews with college personnel through site visits to eight colleges during spring 2008 and conducted phone interviews with college personnel at an additional nine colleges through September 2008. The research team interviewed college trustees, presidents, vice presidents of instruction and student services, institutional research personnel, faculty, and student services staff. In all, we interviewed over 240 individuals connected with the 17 colleges. A second round of in-person and telephone interviews was conducted by CCRC in early fall 2008 with key internal and external stakeholders and with policymakers to better understand the policy context within which the Student Achievement Initiative was conceived and is now being implemented. Among those interviewed were members of the State Board, executive staff from the SBCTC, staff and board members from the Washington Higher Education Coordinating Board, a senior analyst from the governor’s Government Management Accountability and Performance (GMAP) office, and a state senator who has been a leader in issues related to higher education accountability. Formative Evaluation: Summary of Findings (cid:120) Colleges strongly supported the initiative’s goals and the principles of the achievement point framework Among those we interviewed at 17 of the state’s 34 community and technical colleges, there was widespread support for the initiative’s goal of focusing attention and resources on helping all students, including those who start in basic skills and college remedial programs, to successfully progress through their educational programs. There was also strong support for the focus on increasing student progression across intermediate levels of achievement. The faculty, student support staff, and administrators we interviewed generally agreed that this incremental progression framework is much more relevant to their work with students than accountability measures the state has used in the past, which were based primarily on final outcomes. Virtually everyone who was familiar with the research the State Board staff conducted to inform the development of the overall framework and the specific performance measures had high regard for it. CCRC Formative Evaluation of the Student Achievement Initiative “Learning Year” 3 (cid:120) Awareness of the initiative was limited among the colleges’ rank and file While State Board staff did much to try to build awareness of the initiative among the colleges, at the colleges in this study familiarity with the initiative is mostly limited to college leadership, senior administrators, and a small number of college personnel involved in initiative efforts at the colleges. Several colleges were waiting for a better understanding of their performance on the complete 2007-08 data before expanding communication about the initiative to faculty and staff. Senior administrators at several colleges said that, having had to respond to numerous State Board initiatives in recent years, they are reluctant to engage campus-wide constituencies in another external directive until they are assured it is likely to gain support from the state legislature. The lack of awareness and clarity about the initiative should not be too surprising, however. The initiative was developed and is being implemented on a relatively short timeline, and the colleges were still in the first “learning year” of the initiative during our interviews. Still, college personnel at several colleges suggested that clear and transparent communication about the initiative would allay some of the uncertainty and anxiety surrounding the effort and help build support for it. (cid:120) Colleges grappled with their performance data throughout the learning year There was considerable uncertainty among the colleges about exactly how student progression is measured using the achievement point framework. Most colleges struggled throughout the learning year to understand the details of the performance measures. At the time of our interviews, most colleges were still in the process of understanding how their baseline year achievement point totals were calculated. The lack of institutional research capacity hampered several colleges’ ability to analyze the data for a more thorough understanding of their college’s performance. Even colleges with greater IR capacity were having difficulty making sense of the data and figuring out how to use the information to monitor student progression and evaluate efforts to improve student outcomes. (cid:120) Most colleges had not yet used an analysis of their performance data to plan new strategies to improve student achievement Because of these difficulties with the achievement point data, most colleges had not begun to use the Student Achievement Initiative database to develop strategies for increasing student success. Only a handful was planning new strategies to improve their achievement point performance. Most were relying instead on student success activities previously in place at their colleges. While college personnel were optimistic that these existing activities would help to increase student attainment of achievement points, most such efforts were small in scale. College leaders have not yet engaged faculty and staff to figure out what changes in policy and practices are needed to increase the rate at which students attain achievement points on a substantial scale. CCRC Formative Evaluation of the Student Achievement Initiative “Learning Year” 4 (cid:120) At many colleges, student services staff led efforts in response to the initiative Many interviewees we spoke to could not, or at least did not, distinguish between ongoing efforts to support student retention and success at their college and activities planned in response to the Student Achievement Initiative. At least half of the colleges in our study gave primary responsibility for responding to the initiative to their student services division. One reason is that at most of the colleges where we conducted interviews, student services divisions have primary responsibility for efforts to improve student retention and success, so it made sense for them to take the lead on this new effort. Student services personnel were enthusiastic about what they saw as an increased emphasis on student persistence by their colleges. Most indicated that their existing student retention efforts would help to improve their college’s performance on the achievement point measures of the Student Achievement Initiative. (cid:120) Several colleges focused their initial efforts on basic skills and developmental education Several colleges focused their initial work on the initiative on basic skills and developmental education. There were several reasons for this. First, actions of the SBCTC staff responsible for basic skills helped raise awareness of the initiative among basic skills programs directors. Second, there was a widespread perception that colleges could earn more points in the basic skills achievement category than in those related to passing college-level math, completing college credits, or earning credentials. Third, most of the colleges had begun efforts to improve outcomes in basic skills and developmental education before the advent of the Student Achievement Initiative. Interviewees said that the initiative provided additional incentive to strengthen those efforts. College administrators said that there has been a growing recognition in recent years that too few students progress through the various levels of basic skills or developmental education to advance to college-level coursework. And finally, persons involved with basic skills programs were often perceived to be more receptive to the initiative because they are accustomed to being held accountable for student performance. Basic skills directors and instructors at several colleges suggested that existing state and federal performance requirements had given them experience in tracking the progress and outcomes of their students. (cid:120) Some colleges were planning to implement measures with a likely one-time effect Some colleges were planning to implement measures that would likely have a one-time effect on increasing achievement point attainments by their students. Among the approaches proposed were these: conducting “degree audits” or transcript reviews to identify students who achieved the required credits for a certificate or degree but never received one; eliminating certificate or diploma fees and creating academic certificate programs; documenting more rigorously CASAS testing results for basic skills students; and increasing credit requirements for certain shorter occupational certificate programs so they will qualify for achievement points. (cid:120) Several colleges were beginning to link the initiative to strategic planning and accreditation activities. Some college administrators were beginning to align the initiative’s performance measures with their strategic planning and re-accreditation activities. Six colleges had begun early efforts to CCRC Formative Evaluation of the Student Achievement Initiative “Learning Year” 5 incorporate the initiative into their internal strategic planning processes. College personnel at at least eight colleges discussed how the initiative’s focus on documented student progression supported their institution’s re-accreditation efforts. (cid:120) Colleges were concerned that use of the achievement point framework and the incentive funding model may produce unintended effects and place some colleges at a disadvantage While there was general support for the principles of the initiative’s achievement point framework and the model for rewarding achievement point gains, many interviewees voiced a number of concerns. Some interviewees felt that, in practice, their use might unintentionally alter their college’s priorities. For example, some faculty and administrators from career-technical and academic transfer programs expressed concern that the initiative’s heavy focus on basic skills and lack of performance measures for job placements and baccalaureate transfers could draw resources away from their areas. Interviewees frequently suggested that use of the framework and the incentive model might disadvantage certain kinds of colleges, including 1) institutions with a strong academic transfer mission, because the potential for generating the most achievement points is found in the basic skills area; 2) institutions without capacity for significant enrollment growth, because colleges generally might be able to accrue achievement points by increasing enrollments rather than by improving rates of student progression; 3) high performing colleges, which may have less room to improve; and 4) institutions serving a high proportion of disadvantaged students, because it would be difficult for such institutions, who need to provide lots of costly wrap-around services, to earn as many points as colleges with easier-to-serve students. Some colleges were also anxious about how their performance would compare to other colleges (apart from the issue of funding). Concerns were also raised about the potential for “gaming” the system (by, for example, focusing efforts on increased enrollment rather than improved student progression). Interviewees at five colleges noted that, in practice, incentive funding could lead to a lowering of academic standards, and a few interviewees felt that its use could discourage colleges from sharing effective practices. (cid:120) College presidents emphasized that, in order to be effective in improving college performance, the initiative must bring new funding to colleges, over and above base budget funding The potential of significant funding tied to institutional performance has generated concern and anxiety across the colleges, particularly among presidents. Most presidents we interviewed seem to agree that the amount currently being sought by the State Board from the legislature — $7 million for the biennium, or about $100,000 per year per college — is large enough to motivate colleges to make changes in the way they operate, particularly because the funds would be added to their base budgets. Presidents we interviewed were unanimous in the view that to be effective in improving college performance, the initiative must bring new funding to the table over and above base budget funding. College leaders feared a situation where there would be no new funds to support performance funding and that colleges would instead be forced to earn some portion of their base budget funding based on the achievement point measures. Presidents argued CCRC Formative Evaluation of the Student Achievement Initiative “Learning Year” 6 that this would lead to unhealthy competition among the colleges and would significantly reduce support for the initiative from the colleges. Although there were exceptions, most state-level observers we interviewed seemed to agree that funding above and beyond the base budget funding is essential for the initiative to be effective and sustainable. (cid:120) While state policymakers indicated strong support for the initiative’s model of performance accountability and improvement, there is limited awareness of the initiative and no strong champions for it among state legislators. The looming fiscal crisis further threatens the SBCTC’s legislative request for new funding to support the initiative. The state-level stakeholders we interviewed were very enthusiastic about the initiative’s performance measurement model. Indeed, there are strong synergies between the initiative and the governor’s push for more outcome-oriented and accountable state programs. Because accountability pressures across state agencies are not likely to abate, state-level policymakers felt that the Student Achievement Initiative could give the community and technical colleges an advantage as they compete with other sectors for scare state resources. The Higher Education Coordinating Board seems committed to advocating with the legislature for the SBCTC’s request for new funding to support the initiative. However, state-level stakeholders indicated that the initiative is not well known and lacks strong champions among legislators. The State Board’s request to the legislature is threatened by the looming fiscal crisis facing Washington State and the country more generally. A decision by the legislature not to fully fund the initiative could affect future college support. At this point, however, the State Board seems intent on moving ahead with the request. Conclusion The State Board developed a performance measurement system as a practical means of encouraging Washington State community and technical colleges to improve their outcomes over time toward system goals and demonstrating the efficient use of public resources with more attention paid to data and accountability. The Board realized that buy-in from the colleges themselves was essential to such a plan. The evaluation we carried out during the initiative’s “learning year” shows that there is strong support among the colleges for the initiative’s goals and for the principles behind the initiative’s performance measurement model. Yet awareness of and knowledge about the initiative is still weak among many of the rank and file at most colleges. This should not be too surprising, however, given that the initiative is new and still evolving. Likewise, the problems that colleges are experiencing in understanding and using the performance data from the initiative database are not necessarily unexpected. Nevertheless, the need for greater clarity about the database will have to be addressed before the data will be disseminated to wider audiences within colleges and used extensively to identify strategies for improving student success. CCRC Formative Evaluation of the Student Achievement Initiative “Learning Year” 7