-- DRAFT --   The Accidental Revolution: Teacher Accountability, Value-Added, and the Restructuring of the American School System February 7, 2012 Chris Thorn Douglas N. Harris   1 -- DRAFT -- There is little question that test-based school accountability, driven by No Child Left Behind, has been one of the most influential policies in the nation's education history. It has changed what teachers teach, how they teach it, how teachers think and feel about their work, and how school leaders allocate resources. While still early in its development, NCLB’s offspring—Race to the Top—may be having an even greater impact. With more advanced and interconnected data systems spawned by NCLB, it is increasingly possible to evaluate not just whole schools, but individual teachers and leaders on their “value-added” to student achievement. This shift represents a sea change in how schools operate, ending the half-century long autonomy that teachers have come to expect. The balance of power in school decision- making has shifted away from teachers, unions, and schools of education—what some call “the establishment”— toward testing companies, data managers, district department heads, school principals, and state and federal policymakers. These changes have been substantially intentional. The goals of test-based accountability advocates have been to increase accountability for school leaders and individual educators, and to provide the analytic tools to help them respond to those pressures and improve learning. Perhaps without even realizing it, however, educator accountability is re-structuring roles and relationships throughout school districts and yielding entirely new organizations with which districts interact in complex ways. As we show through a series of case studies, this amounts to a major re-structuring of the American school system, and one that has been largely accidental. This restructuring is a natural consequence of the fundamental rethinking of the education system engendered by the shift from school-level to teacher accountability. With school-level accountability, individual teachers are largely insulated from accountability. The worst case— and very rare—scenario with school accountability is that a school will be closed and the 2 -- DRAFT -- teachers dismissed. Teacher accountability in contrast reaches every teacher in every school, as well as a host of indirect and secondary influences that we explore through a series of case studies. When the stakes were low, test designers (mostly for-profit companies) received little scrutiny and their decisions had relatively little influence on the working lives of individual teachers. Our discussion of Hillsborough County shows that the testing companies do not yet have the capacity to meet the demands of these new systems and this means that, for now, assessments will have to be designed in different ways, by other groups. When data were only available at the school-level, a simple spreadsheet was all that was needed for data management. Our discussion of Hillsborough County and Center for Educational Innovation - Public Education Association show that for-profit database management and/or data warehouse companies now play an integral role. This has been driven, indirectly and apparently unintentionally, by educator accountability. Without it, there would be little impetus to change. Educator accountability has generated a need to coordinate. In the past, the assessment director might rarely have spoken with, let alone coordinated with, the director of professional development or human resources. Another recent federal accountability program, the Teacher Incentive Fund, requires that teacher and principal professional development be explicitly targeted at weaknesses identified by individual teacher evaluations. To do that, different district administrators have to talk with one another. The linkage between evaluation results, selection and assignment of appropriate professional support resources, the provision of those services to those individuals across many schools requires a level of coordination and cooperation uncommon to the traditional American school district. Educator accountability is leading to the integration of data from assessments, human resources, professional development, and 3 -- DRAFT -- curriculum departments, making it possible for them to coordinate. This pressure has spawned a breakdown in data silos and management isolation so that districts can succeed in meeting state and federal mandates and to ensure that individual teacher performance measures can be fair and accurate. When teacher evaluation lacked teeth, it mattered relatively little what principals thought of their teachers. New systems of evaluation not only require formal evaluation of individual teachers, but that action be taken to address shortcomings and principals necessarily play an important role in that. Many Teacher Incentive Fund programs also create new roles for teacher leaders – master or mentor teachers, peer evaluators, etc. The organizational changes at the school level required to distribute leadership tasks and embed professional development in the working day are substantial. Our discussion of the three Teacher Incentive Fund projects, which all include teacher and principal evaluation components, suggest that times are changing. To make teacher accountability a reality, many state and federal policymakers have decided to usurp responsibilities for evaluation, compensation, and dismissal from teacher unions. Once these big leaps are taken, it is only a short step to accountability for the programs and institutions that prepare teachers. We discuss the case of the Network of Effective Teaching (NExT) in which participating schools of education (SOEs) provide guarantees of their graduates’ effectiveness. In the past, schools of education could largely do what they wished. Graduates enter teaching, or not, teach well, or not. Perhaps the most positive role of the SOEs was to identify teachers who were committed enough to jump through all the hoops the programs required, even if those hoops did little to prepare teachers for the classroom. Since schools themselves were not focused on effectiveness in the hiring process (Harris, Rutledge, Ingle, & Thompson, 2010), graduates could find jobs in any event. But once we can evaluate individual 4 -- DRAFT -- teachers, it takes little imagination, and almost no statistical knowledge, to place teachers into groups based on where they were prepared and to compare average performance across SOEs and other training institutions. While little research and evaluation on these reforms has been published, we discuss ongoing cases of reform with which we have had direct experience. One of us (Thorn) works for a non-profit organization to which power has shifted: the Value-Added Research Center (VARC) at the University of Wisconsin at Madison. VARC is a supplier of value-added measures and professional development to school districts seeking to use it. The other (Harris) works with states and school districts across the country in an advisory capacity as they implement value- added-based accountability. Drawing on these experiences allows us to provide an up-to-date analysis of what is happening in this fast-moving environment. We also keep our eyes on what these experiences mean for fundamental questions of school reform, governance, and management. Test-based accountability is part of a larger, long-term trend toward performance management in the public sector more broadly (Heinrich & Marschke, 2010; Moynihan, 2010). We continue in the next section by discussing the one element that all the cases have in common: teacher value-added estimates. Because we are interested in how accountability is changing the ways in which schools and districts operate, we leave it to others who have discussed the technical properties of the measures (e.g., Harris & Sass, 2006; Harris, 2009, 2011; Glazerman et al., 2011). We then discuss four cases that have teacher accountability based on value-added at their core. Many of these cases are in their early stages and, while there is little formal research, we draw on first-hand experiences and anecdotes that lead to a set of observations and questions, all of which we believe have important implications for the future of education accountability as well as its effects on the daily lives of educators and student 5 -- DRAFT -- outcomes. It appears to us that the pressure for individual accountability is leading to a possibly widespread and permanent restructuring at every level of the American school system. Value-Added and Observational Measures of Educator Performance At the heart of the teacher accountability movement, and our case studies, are value- added measures of performance. While the statistical models differ from site to site, the basic concept is the same. Rather than rely on snapshots of achievement at a point in time or on changes in school- or grade-level average test score, value-added models take into consideration the prior level of achievement of each child. Children come into classrooms at different starting points. Value-added models capture the progress students make, given where they start. There are a number of different modeling choices that one might consider – the inclusion of student demographics, multiple years of pre-tests, classroom characteristics, etc., but these are extensions of the basic notion of capturing the contribution of the teacher.1 Aside from any modeling challenges (deciding what features are appropriate for a particular use), there are substantial technical difficulties associated with implementing teacher- level value-added models. The biggest problem is having high quality attribution (linkages) between teachers and students by subject taught. This may sound trivial, but most student information systems (SIS) are designed to manage scheduling and basic record keeping. They are not designed to manage the complexity of how learning is delivered to individual students. For example, most SISs indicate only the teacher of record for any given room or course section. This does not include formal or informal team teaching arrangements, ability grouping, ELL or special education aides. If one were to use simple administrative data from the SIS, the resulting value-added results would contain a number of unknown attribution errors that could seriously 1 See (Harris, 2010; Kane & Staiger, 2012) for more. 6 -- DRAFT -- impact the effectiveness scores of the teachers involved. Another problem associated with student linkage is detailed information on dosage – or the amount of instructional time received by each student preparing for any given assessment. Many students receive after school tutoring or other services that are not traditionally captured in SIS records.2 There are several groups working at the state, district, and school level to provide tools to overcome the limitations of SIS data. Battelle for Kids provides support for this at the state and district level through a linkage tool they have developed called BFK•Link®. The National Institute for Excellence in Teaching also provides a comprehensive data collection and linkage system that supports called CODE that supports schools using the Teacher Advancement Program (TAP). Many TIF districts have developed simple online or manual student roster verifications without any intelligent automated processes around it to lessen the burden on users. Imagine a solution that provides teachers with pre-populated spreadsheets that contain course rosters based on the information in the student information system. If a student is missing from the roster, there is no automatic feature search function to help find that student in a building- level list. The verifying teacher must hand enter the information with the resulting chance of error when that data is compared to online data at a later point – spelling differences, typos, etc. all introduce error. At the building level, one of the important roles of the principal is to make sure that no students are “unclaimed” by teachers or that all teachers have been linked to classrooms to reflect team teaching or other more complex organizational models. These tasks are relatively simple tasks in automated linkage systems. The same tasks are quite labor intensive in a manual system. The other major difference in automated systems is the ability to capture the linkage of support services to individual students. Many students are served by a number of other 2 See (Battelle for Kids, 2011; Watson, Witham, & St. Louis, 2010) for more. 7 -- DRAFT -- adults. Dedicated linkage systems can more easily capture these more complex relationships – things that often don’t fit in an SIS. In addition to data quality problems, there are concerns about the quality of the assessments available. Most state-mandated assessments were created to measure particular proficiency cut points at each grade. They are less accurate measuring students who are performing above or below grade level. There are also concerns about the level and types of mastery that can be measured with traditional fill-in-the-bubble assessment formats. Many curricular standards direct teachers to bring students to a level of mastery that allows them to apply concepts learned during the year to novel problems – not just recitation of facts or formulae. There can be a significant mismatch between the level of mastery that can be measured by the available assessment and the educational goals of the school, district, or state.3 Those assessments are most accurate for measuring the performance of student at or near those cut points. The federal assessment consortia – Partnership for Assessment of Readiness for College and Careers (PARCC) and the SMARTER Balanced Assessment Consortium (SBAC) – are intended to address some of the concerns about test quality in the core grades and subjects. The PARCC group will provide both interim and summative, computer-based, performance based assessments in grades 3-8 in mathematics and English language arts based on the Common Core Standards. There will also be grade-based high school assessments in the same subject areas and formative, vertically linked items for grades K-2.4 The SBAC will provide online, computer adaptive summative assessments math and English language arts in grades 3-8 and 11 based on the Common Core Standards. The group will also provide optional interim and formative 3 See (Baker et al., 2010; Koretz, 2008) for more. 4 http://www.parcconline.org/parcc-assessment-design 8 -- DRAFT -- assessment for capturing student progress through the year.5 Both groups will be using common interim assessments. Because almost all states will be implementing these assessments, it will be possible for the first time to compare educator performance across states. This could open up new areas of institutional cooperation and collaboration. As others have pointed out regarding the benefits of common standards and assessments (e.g., Harris & Goertz, 2008), this will facilitate professional development and other bases for school improvement that is both curriculum specific and cuts across states. This could increase the roles of national companies and non-profits who could take advantage of economies of scale (Harris & Taylor, YEAR). Under NCLB, which covers only Math & English/Language Arts in grades 3-8), educational agencies have already begun purchasing commercial assessments in other grades and subjects. In other locations, the creation of new assessments has been an explicit partnership between leadership and teachers and curriculum experts. It may be that localized test design becomes an important trend over the next several years, but it is difficult to see how this will stand up to political demands for standardization and economies of scale – not to mention potential profits that could be leveraged by the large testing companies. Moreover, in our recent experiences working with school districts that are implementing value-added measures for accountability, we also observe a general crumbling of the walls and silos that separate school district administrative activities. While in many cases this has positive implications since many silos have resisted evaluating their own practices or using data at all to make decisions, it is often done with little planning or attention to the staff development needs of the actors involved. Informal discussions with leadership teams across the country suggest that basic assessment and data literacy are critical problems for district staff. Grant requirements for 5 http://www.k12.wa.us/SMARTER/FAQ.aspx 9 -- DRAFT -- Teacher Incentive Fund recipients to link professional development to adult needs identified by evaluation results provide a unique opportunity to link this important source of “inputs” to outputs. The data linkages and data quality needed to routinely evaluate educational interventions are actually being put into place. What is missing in most schools and districts are staff members with the analytical training to design and implement robust evaluation designs. This skill gap stands in the way of schools and districts doing their own research on “what works”. This shift in large scale accountability is based on the assumption that teacher effectiveness measures, both value-added results and observational measures of teacher practice, meet data quality requirements necessary to accurately attribute connections between adults in schools to students by subject or program area. One of the challenges is that the assumptions about validity and reliability at the macro level are likely do not recognize the variability of local programs or practices. The challenge will be to ascertain which validity assumptions are met and what the implications of misfits are for using the accountability data. In addition to the use of measures of student growth on assessments, all of the examples cited here rely on observation as another critical element of understanding effective teaching. There is a strong sense across most scholars who study teacher effectiveness that observational measures of teacher practice are important to understand which practices are most effective at delivering growth in student learning. In untested grades and subjects, observational measures of practice are often the only objective measure available that attributable to the individual teacher. Across TIF sites, adaptations of the Danielson Framework6 are very common. Other frameworks in use include Pianta’s CLASS7 or Marzano’s Teacher Effectiveness Framework8. All TIF grants 6 http://charlottedanielson.com 7 http://curry.virginia.edu/research/centers/castl/class 10