Program on Education Policy and Governance Working Papers Series The Impact of Incentives on Effort: Teacher Bonuses in North Carolina Tom Ahn Jake Vigdor University of Kentucky Duke University PEPG 10-06 Prepared for the PEPG Conference Merit Pay: Will It Work? Is It Politically Viable? Harvard Kennedy School Cambridge, Massachusetts June 3-4, 2010 The Impact of Incentives on Effort: Teacher Bonuses in North Carolina* Tom Ahn University of Kentucky Jake Vigdor Duke University Abstract Teacher effort, a critical component of education production, has been largely ignored in the literature due to measurement difficulties. Using a principal-agent model, North Carolina public school data, and the state's unique accountability system that rewards teachers for school-level academic growth, we show that we can distill effort from teacher absence data and capture its effect on student achievement in a structural framework. We find that: 1. Incentives lead teachers to try harder. The bonus program reduced the number of sick days taken by about 0.6 days for an average teacher. 2. When teachers try harder, students do better. Increased effort of teachers translates into improved student performance. Estimates show that standardized reading scores increased by about 1.3% of a standard deviation and standardized math scores by about 0.9% of a standard deviation. 3. Group-level incentives can actually be more powerful than individual-level incentives. Policy simulations from the model estimates show that an individual bonus program would actually produce weaker incentive effects. While free-rider effects are eliminated, individual incentives push a majority of teachers into one of two categories: those who would qualify for the bonus even without trying and others would not qualify no matter how hard they worked. Performance pay for teachers: are school-level incentives enough? Over the past ten years, researchers have devoted considerable effort to the measurement of the output of schools and teachers, using standardized test scores.1 Our ability to infer the quality of teaching in a school or classroom has developed sufficiently far that school * We grateful to the Center for Child and Family Policy for access to the NCDPI dataset and to the American Enterprise Institute for financial support. 1 See Rivkin, Hanushek, and Kain (2005), Clotfelter, Ladd,and Vigdor (2007), Goldhaber and Anthony (2007), and Rockoff (2004), among many others. districts across the nation, from Denver to Washington DC and many points in between, have put incentive programs in place that make student test score performance a major factor in the evaluation, and in some cases the compensation, of teachers. There are some sticky issues, however, about how to really make a pay-for-performance scheme based on test scores operate.2 For one thing, most school systems don’t conduct standardized tests in every grade, or in every subject. How exactly are we supposed to measure the performance of a kindergarten teacher? Or a high school Spanish teacher? Or a middle school physical education teacher? And if we don't evaluate these teachers' performances, how do we pay them? A second issue with pay-for-performance schemes is that they may lead teachers to fight amongst each other for the best students. Teachers might perceive that certain students hit performance targets more easily; in many cases there is strong evidence to back up this perception. Principals and other administrators might come under pressure to fiddle with classroom assignments. In theory, a commitment to randomly assign students to classrooms would present teachers with a level playing field. But even randomization makes some people lucky and others unlucky sometimes. To complicate matters, many schools have instituted different tracks for students with differing academic abilities. Pay- for-performance would add a financial reason to lobby to teach honors-class students in addition to the laundry list of non-pecuniary benefits. 2 See Neal (2009) for a review of some of the inefficiencies of the pay-for-performance system. Another problem is the “noisy" test results issue. The “noise” we are talking about here is statistical noise, and it is a more severe problem for classroom-sized groups of students relative to school-sized groups. A “noisier” test result makes it difficult to discern whether a higher average score for a group of students relative to another group really means that the former group of students knows more about the subject they were tested on. The smaller the number of test takers, the higher the possibility that one outlying result can throw off the entire result. For instance, if one or two students were ill on the day they took the test, their poor exam scores will significantly drag down the class average and make the teacher look less competent than she really is. These unfortunate students would have a smaller impact on the school average, since they make a smaller proportion of the larger student body. These three Gordian knots – the presence of untested grades and subjects, non-random assignment of students to teachers, and the statistical noise problem in small samples – could be sliced with one modification to pay-for-performance: reward teachers on the basis of all students in the school, rather than just those in their classroom. With school- based incentives, we need not worry about what to do with teachers of odd subjects, or in untested grades. And we need not worry about teachers fighting one another in a zero- sum game, or about statistical noise leading to good teachers going unrewarded. The primary theoretical argument against school-based rewards will be familiar to economics 101 students everywhere: the free-rider problem. Compared with an individual-level incentive, a group-level incentive should have less impact. When your own effort determines my compensation, you have a very strong reason to work hard. When the combined effort of a large group determines your compensation, you may feel at greater liberty to slack off, since most of your reward depends on the actions of other people anyway. It’s the tragedy of the commons, the prisoner’s dilemma – whatever you want to call it. It’s an argument so strong and so intuitive that you don’t hear many education economists saying that school-level rewards are the way to go. Until now. New evidence, derived from the experiences of North Carolina public schools, which have implemented school-based monetary incentives for more than a decade now, indicates that this conventional wisdom – that individual incentives are more powerful than group incentives – is in fact wrong. Yes, Virginia, there is a free rider effect. But what the conventional wisdom fails to incorporate is a powerful countervailing effect, which we might call the “tortoise and hare” effect, borrowing from Aesop’s fables. Consider the following scenario. You are an excellent teacher – one of the best in the business. If the school system sets a bar and promises you rewards if your students exceed it, you know you can exceed the expectation even without trying. Like the hare in the fable, your incentive to try your best is undermined by a sense that your success is inevitable. We may fault the hare for his laziness, but is this really such a surprising response when victory seems assured? The teacher next door, on the other hand, is hopelessly incompetent. You know as well as she does that no matter where the bar is set, her students will almost certainly fall below it. Like the tortoise in the fable, it is only her personal virtue that implores her to exert effort: the incentive means very little. Again, we may cheer the tortoise for his perseverance, but how wise is it to expend effort when there is virtually no chance of success? So, for both you and your neighbor, the individual-level incentive scheme provides almost no incentive to exert greater effort. You are bound to be rewarded no matter what, and your neighbor is destined to fail no matter what. It would be great if both the tortoise and the hare tried their hardest regardless of the competition, but the most likely outcome is that the hare would win walking backwards, and the tortoise would quit before the race even begins. Now suppose we tie you and your neighbor together: your reward will be based not on what you do individually, but the sum total of what you accomplish. All of a sudden, you recognize that the status of your reward is in doubt, and the teacher next door realizes that she now has a realistic shot at the reward; both you and your neighbor are going to have to exert some effort to ensure that the average across your two classrooms exceeds the standard. While the traditional moral of the fable is that “slow and steady wins the race,” perhaps we should reconsider the wisdom of such a match-up in the first place. Rather than race tortoises against hares, we should pair one of each together and judge each pair by their combined time. In this scenario, each competitor faces a stronger incentive to excel, because it is their team’s average time that matters, not their rank within the team. This is a plausible scenario, right? But we just don’t know how common this scenario might be. How often do very good and very poor teachers share the same school? And just how powerful is this free rider effect anyway? The answers lie in the research. To tell you about the research, we first need to spend some time getting to know the setting. The North Carolina State Accountability System The North Carolina ABC accountability program (ABC is an acronym for Accountability, teaching the Basics, and emphasis on local Control) began in the 1996/97 school year. In its inaugural year, teachers in elementary and middle schools were awarded a cash bonus of $1,000 if the school’s average year-over-year improvement in reading and math test scores exceeded the required threshold set by the state. In the following year, the bonus program was extended to high schools, and the award became two-tiered, with teachers receiving $750 in schools that cleared a first threshold referred to as “expected” growth in test scores and $1,500 in schools that cleared a more stringent “exemplary” or “high” growth threshold. 3 Education authorities face a delicate balancing act in setting criteria for bonus payments. If teachers perceive that there is no chance of receiving a bonus, or conversely that the bonus is a sure thing, they have little reason to alter their behavior. This is a basic statement of the “tortoise and hare” effect described above. Fortunately, in North Carolina’s case, teachers in most schools face real uncertainty about the amount of their 3 A complete description of the bonus program and the formulas used to set each school’s threshold can be found in Vigdor (2008). bonus. Figure 1 shows the proportion of schools in the 1999/00 to 2001/02 school years qualifying for $750 or $1500 bonuses. Roughly three-quarters of the schools in the state received bonus payments, but less than half received the full $1,500. The average bonus paid out is roughly $890 (0.23 X $0 + 0.35 X $750 + 0.42 X $1,500 = $890). Vigdor (2009) presents additional evidence that among the schools eligible for any bonus at all, about half receive the full $1,500. There are very few schools that can count on the full $1,500 as a sure thing, and very few for which the $750 standard is completely unattainable. Figure 1: Bonus Receipt by School No Bonus 23% $1,500 Bonus 42% $750 Bonus 35% Incidentally, North Carolina’s system is made possible because the state has a longitudinal data system that can link the performance of individual students as they progress from grade 3 to grade 8. Many other states, unfortunately, have no capacity to link students across years, implying that they can only judge schools by how the students perform in a given year, not by how much they improve in a given year. This limitation forced the federal No Child Left Behind act to focus on proficiency rather than improvement. Why does this matter? A school that serves very low-performing kids and manages to improve their performance dramatically might not be rewarded if their ultimate performance is below the state’s threshold for proficiency. Figuring out what the bonus program accomplishes The North Carolina ABC system is not costless. Their state legislature needs to allocate 90 million dollars or more per year for these performance bonuses. And while there’s a strong economic reason for thinking that performance bonuses improve student performance, there’s no specific guidance regarding how big the impact should be, let alone whether the impact is worth the amount of money being spent on the program.4 So is the program worthwhile? How can we tell? The gold-standard method of evaluating a program such as the ABC initiative would have been to conduct a randomized trial. Schools in North Carolina would have been randomly assigned into two groups: a “treatment group” of schools where teachers were awarded the bonus according to the ABC framework and a “control group” where teachers did not receive the bonus. If the incentives worked as planned, teachers in the treatment group would have exerted higher effort to teach students, and this would have translated into higher scores for students in treatment schools relative to those in control schools. 4 See Figlio and Kenny (2007).