Value-Added Estimates for Phase 1 of the Pennsylvania Teacher and Principal Evaluation Pilot Full Report April 5, 2012 Stephen Lipscomb Hanley Chiang Brian Gill All statistics are calculated by Mathematica unless stated otherwise Mathematica Reference Number: Value-Added Estimates for 06815.300 Phase 1 of the Pennsylvania Submitted to: Teacher and Principal Team Pennsylvania Foundation 100 Pine Street, 9th Floor Evaluation Pilot Harrisburg, PA 17101 Project Officer: Jennifer Cleghorn Full Report Office of the Deputy Secretary of Elementary and Secondary Education April 5, 2012 Pennsylvania Department of Education 333 Market Street Stephen Lipscomb Harrisburg, PA 17126-0333 Hanley Chiang Project Officer: Carolyn C. Dumaresq Brian Gill Submitted by: Mathematica Policy Research 955 Massachusetts Avenue Suite 801 Cambridge, MA 02139 Telephone: (617) 491-7900 Facsimile: (617) 491-8044 Project Director: Stephen Lipscomb All statistics are calculated by Mathematica unless stated otherwise ACKNOWLEDGMENTS This report would not be possible without the support of the Team Pennsylvania Foundation, the Pennsylvania Department of Education, the Bill & Melinda Gates Foundation, and members of the Stakeholder Steering Committee who volunteered their time during Phase 1 of this pilot study. We are especially grateful to Carolyn Dumaresq, Jennifer Cleghorn, Matt Zieger, Pat Hardy, Rich Maraschiello, Sharon Brumbaugh, and Theresa Barnaby for their project leadership, to Harris Zwerling for his insightful comments and suggestions, and to Dave Ream, Thomas Gillin, and Thresa Stafford for providing us with the statewide data used in this report. We also thank Bob Hughes, Suzanne Lane, and the teachers, principals, and superintendents from the four pilot school districts for their important contributions during the pilot toward the improvement of the evaluation system. We would like to recognize several staff at Mathematica Policy Research who made substantive contributions to this report. Clare Wolfendale provided outstanding research support in estimating the value-added models and populating tables of results. We are also grateful to Serge Lukashanets, Xiaofan Sun, and Alena Davidoff-Gore for their contributions to organizing the data, to Duncan Chaplin for his comments and suggestions on an earlier version, to John Kennedy for his editorial assistance, and to Autumn Parker and Eileen Curley for their production support. Any opinions expressed herein are those of the authors and do not necessarily represent the views of the Team Pennsylvania Foundation or the Pennsylvania Department of Education. iii THIS PAGE LEFT BLANK FOR DOUBLE- SIDED PRINTING CONTENTS ACKNOWLEDGMENTS .................................................................................................. iii EXECUTIVE SUMMARY .................................................................................................. xi I. INTRODUCTION ............................................................................................. 1 A. The Pennsylvania Teacher and Principal Evaluation Pilot .......................... 1 B. Description of Value-Added Models ........................................................ 2 II. CHARACTERISTICS OF VAMS ESTIMATED IN THIS REPORT AND THE APPLICABILITY OF EFFECTIVENESS MEASURES TO TEACHERS IN PHASE 1 ......................................................................................................... 7 A. Outcome Measures .................................................................................. 7 B. Teachers with VAM Estimates from Phase 1 of the Pilot ......................... 10 C. Control Variables that Are Included in the VAMs .................................... 12 III. VALUE-ADDED RESULTS FOR TEACHERS ....................................................... 17 A. Variation in Teacher Effectiveness Based on PSSA Outcomes .................. 17 B. Statistical Uncertainty in the Teacher Effectiveness Estimates ................ 20 C. Sensitivity of Measured Effectiveness to Alternative VAM Specifications ........................................................................................ 24 D. Key Characteristics of Teacher Effectiveness Estimates Based on Pilot District Samples and Other Outcomes ............................................ 29 IV. RELATIONSHIPS BETWEEN TEACHER PRACTICES AND VALUE ADDED ............. 33 A. The Phase 1 Teacher Observation Rubric and Score Distribution ............ 33 B. Observation Scores and Value-Added Scores for Phase 1 Teachers with VAM Estimates ................................................................ 36 C. Relationships Between Value-Added and Observation Scores ................. 38 V. VALUE-ADDED RESULTS FOR PRINCIPALS ..................................................... 41 A. An Approach to Estimating Pure Principal Effects ................................... 41 B. The School VAM as the Basis for Evaluating Principals ............................ 44 C. Key Characteristics of School Effectiveness Estimates Based on PSSA Outcomes ..................................................................................... 46 v Contents Mathematica Policy Research (continued) V D. Key Characteristics of School Effectiveness Estimates Based on Outcomes Other than PSSA Scores ......................................................... 50 VI. LOOKING AHEAD TO SUBSEQUENT PHASES OF THE PENNSYLVANIA TEACHER AND PRINCIPAL EVALUATION PILOT .............................................. 57 REFERENCES ................................................................................................. 59 APPENDIX A: TECHNICAL SPECIFICATIONS OF THE VAMS APPENDIX B: DATA SOURCES AND SAMPLE CHARACTERISTICS APPENDIX C: TECHNICAL RESULTS FROM VALUE-ADDED ANALYSES vi TABLES 1 Number of Teachers with Effectiveness Estimates Reported Based on the Number of Cohorts in the VAM and Share of Reported Estimates that Are Statistically Different from the Average ....................................................... xiv I.1 Strengths and Limitations of Value-Added Models Relative to Other Evaluation Methods ....................................................................................... 4 II.1 Outcomes Considered in Value-Added Models for Teacher and School Evaluation in this Report ................................................................................ 8 II.2 Summary of Teacher Evaluation Pilot, Phase 1 .............................................. 11 II.3 Number of Teachers from Phase 1 with at Least One VAM Estimate that Can Be Used for Correlating Value-Added with Teacher Practices in Chapter IV ................................................................................................... 12 II.4 Baseline Measures for Value-Added Models Estimated in this Report, by Outcome ..................................................................................................... 13 II.5 Student and Classroom Control Variables Included in VAMs Estimated for this Report................................................................................................... 15 III.1 Distribution of Teacher VAM Estimates for Selected PSSA Outcomes ............ 17 III.2 Teacher VAM Estimates in Recent Studies for the 85th Percentile of Effectiveness Relative to the 50th Percentile, Reported in Standard Deviations of Student Test Scores ................................................................ 19 III.3 Number of Teachers with Effectiveness Estimates Reported and Share of Reported Estimates that Are Statistically Different from the Average, by Number of Cohorts Used in Estimation ........................................................ 23 III.4 Implied Percentage of Variation in Teacher Value-Added Within Districts and Schools ................................................................................................. 25 III.5 Counts and Percentages of Grade 5 Math and Reading Teachers in Effectiveness Quartiles Based on 3-Cohort Teacher VAMs that Include and Exclude Same-Subject Baseline Scores ......................................................... 27 III.6 Grade 4 Through 8 Math and Reading Teachers in Pilot Districts in Effectiveness Quartiles Based on Teacher VAMs with Fall or Spring Baselines Using PSSA Outcome Data ............................................................ 29 III.7 Key Characteristics of Teacher Effectiveness Estimates Based on Selected Non-PSSA Tests Administered in the Pilot Districts ...................................... 30 III.8 Key Characteristics of Teacher Effectiveness Estimates Based on PSSAs in 3rd and 11th Grades in the Pilot Districts .................................................... 32 IV.1 Danielson Framework Domains and Components, by Priority and Additional Components for the Pennsylvania Pilot ........................................ 33 vii Tables Mathematica Policy Research IV.2 Final Ratings in Pennsylvania and Chicago, by Number and Percentage of Teachers ...................................................................................................... 35 IV.3 Sample Characteristics of Nonpilot and Pilot Teachers ................................. 36 IV.4 Regression Coefficients Indicating the Standard Deviation Increase in Teacher Value-Added that Is Predicted for a One-Unit Increase in Rubric Scores ......................................................................................................... 39 V.1 Counts and Percentages of Principals in Effectiveness Quartiles Based on Principal Transitions Model and School VAM ................................................ 46 V.2 Distribution of School Effectiveness Estimates for Selected PSSA Outcomes . 47 V.3 Number of Schools with Effectiveness Estimates Reported and Share of Reported Estimates that Are Statistically Different from the Average, by Number of Cohorts Used in Estimation ........................................................ 50 V.4 Key Characteristics of School Effectiveness Estimates Based on Selected Tests Administered in the Pilot Districts ...................................................... 52 V.5 Key Characteristics of School Effectiveness Estimates Based on Nonassessment Outcomes ........................................................................... 54 B.1 Data Sources ............................................................................................. B-2 B.2 Descriptive Statistics on Student Characteristics, 2010-2011 ..................... B-3 B.3 Baseline and Analysis Student Sample Sizes for Teacher and School VAMs, by Outcome ............................................................................................... B-4 B.4 Number of Teachers and Principals with VAM Estimates Reported from Multicohort and Single-Cohort VAMs ......................................................... B-6 C.1 Sample Characteristics of Outcome Measures and Teacher VAMs Based on State Samples ............................................................................................ C-1 C.2 Estimated Regression Coefficients from Selected Three-Cohort PSSA Teacher VAMs ............................................................................................ C-2 C.3 Effect Sizes for Three-Cohort Teacher VAMs Expressed in Terms of One Year of Learning ........................................................................................ C-3 C.4 Sample Characteristics of Outcome Measures and Teacher VAMs Based on Phase 1 Samples ........................................................................................ C-4 C.5 Sample Characteristics of Outcome Measures and School VAMs Based on State Samples ............................................................................................ C-5 C.6 Sample Characteristics of Outcome Measures and School VAMs Based on Phase 1 Samples ......................................................................... C-6 viii FIGURES 1 Distribution of Teacher Effectiveness for 5th-Grade Math PSSA Scores ........ xiii 2 Distribution of Final Rating Scores for Phase 1 Teachers ............................. xvi III.1 Distribution of Teacher Effectiveness and 95 Percent Confidence Intervals of Teacher Effectiveness Estimates for 5th-Grade Math PSSA Scores ............ 21 III.2 Distribution of Teacher Effectiveness Estimates and 95 Percent Confidence Intervals of Teacher Effectiveness Estimates for 8th-Grade Reading PSSA Scores ......................................................................................................... 22 IV.1 Distribution of Final Rating Scores for Phase 1 Teachers .............................. 34 IV.2 Distribution of Average Rating Scores for Phase 1 Teachers with VAM Estimates ..................................................................................................... 37 IV.3 Distribution of VAM Scores for Phase 1 Teachers with VAM Estimates .......... 38 V.1 Distribution of School Effectiveness Estimates and 95 Percent Confidence Intervals of School Effectiveness Estimates for Math PSSA Grade 5 Scores .... 49 V.2 Distribution of School Effectiveness Estimates for 9th-Grade Holding Power .......................................................................................................... 55 ix THIS PAGE LEFT BLANK FOR DOUBLE- SIDED PRINTING