RESEARCH BRIEF | SPRING 2021 ELEVATING TEACHER EFFECTIVENESS IN A RURAL COMMUNITY Findings from a Teacher Incentive Fund Grant Elevating Teacher Effectiveness in a Rural Community This study examines the impact of a federal Teacher Incentive Fund grant award that expanded a partnership between Cross County School District and NIET to create systems that support ongoing, continuous improvement for teachers. Through the implementation of the TAP System for Teacher and Student Advancement, the district is implementing a coherent and integrated set of strategies that are rooted in a common vision for instructional excellence. Results from the first three years of the grant indicate progress towards increasing teacher collegiality, improving teacher effectiveness, increasing teacher retention, and impacting student achievement. During the grant period, teachers reported increased levels of collegiality in their school. Additionally, the percentage of teachers rated as effective has increased and retention of these effective teachers has improved. Finally, student performance on state standardized assessments improved in all subjects in Cross County School District, surpassing that in comparison districts. Background Cross County School District in Cherry Valley, Arkansas, was awarded a five-year Teacher Incentive Fund (TIF) grant in fall 2016 to maintain and enhance implementation of the TAP System for Teacher and Student Advancement, a comprehensive approach for enhancing teacher and school effectiveness. The TIF grant aims to increase teacher effectiveness and student achievement through supporting, developing, and implementing a sustainable performance-based compensation system and human capital management system that includes evaluation and support for teachers. The TAP System serves as Cross County School District’s human capital management system. A primary goal of this grant is to raise teacher excellence and student achievement. This study reviews progress at the district level during the first three years of the grant. Education in Rural Communities Cross County School District and many other school districts serving rural communities struggle to attract, develop and then keep effective teachers (DeFeo et al., 2017; Garcia & Weiss, 2019; Latterman & Steffes, 2017; Ulferts, 2016, Sutcher et al., 2016). Schools in rural communities face unique challenges due to their remoteness from economic and cultural centers. While studies and reports cite “rural” as a contributing factor to teacher shortages and turnover (Aragon, 2016; Lazarev et al., 2017; Sutcher et al., 2016), more recent studies find that teacher turnover rates in rural locales do not differ significantly from rates in suburban and urban locales. Instead, non-traditional teacher preparation, lack of administrative support, maximum district salaries (not beginning salaries), school size, and percentage of students of color and low-income students were all found to be highly predictive of teacher turnover (Carver-Thomas & Darling-Hammond, 2019). Schools in rural communities across the United States, including in Cross County, exhibit many, or most, of these predictive factors. Another study showed that insufficient pay, poor collegiality, lack of leadership support, and poor school culture and morale were the major factors driving teachers away (Geiger & Pivovarova, 2018). Further, studies of the effects of teacher turnover have shown that high-poverty schools serving remote rural communities rely more on lateral entry and provisional teachers to fill positions than the average school (Sorenson & Ladd, 2020). While Cross County School District cannot change its community or student demographics, it can address teacher working conditions. Cross County School District’s proposed solution to teacher turnover is to continuously improve the quality of instruction of all teachers in the district through (1) the creation of formal structures for teacher leadership, (2) regular, job-embedded professional learning, (3) opportunities for additional compensation, and (4) a system for educator support, observation and feedback tied to high expectations and real-time needs of teachers and students. 1 Cross County School District Cross County is located in the northeastern corner of Arkansas, with the nearest large city, Memphis, Tennessee, about 56 miles east. Cross County School District serves a rural community of about 3,600 residents from around 1,550 total households (National Center for Education Statistics, n.d.-a, n.d.-b). The district serves about 580 students, about 10% of whom identify as racial/ethnic minorities and over 70% of whom are eligible for free or reduced-price meals. The district has approximately 50 teachers split evenly between the elementary school and the high school (Arkansas Department of Education, 2019a). From 2010-11 to 2014-15, annual teacher turnover in Cross County School District schools ranged from 10% to 26%, which is at or above the national annual turnover rate (Snyder et al., 2019). Applicants for Cross County School District teaching positions often come from alternative/non-traditional teacher preparation programs. As such, new teachers are often underprepared to teach, and they struggle with their responsibilities and do not stay for more than one or two years. The most effective teachers are also drawn to larger city schools due to significantly higher salaries and more robust cultural and social communities. To address its workforce instability, Cross County School District began implementing the TAP System and sustained implementation using their Teacher Incentive Fund award in 2016, which aims to build local capacity for improving teacher effectiveness. The TAP System for Teacher and Student Advancement For two decades, hundreds of schools have implemented the TAP System. The theory of action for the TAP System is that applied professional development delivered via teacher leaders using a rigorous rubric of evaluation complemented by performance-based compensation will lead to improved teacher effectiveness, which will lead to improved student achievement. The TAP System theoretical framework consists of four aligned core elements (National Institute for Excellence in Teaching, n.d.). • Multiple career paths. In TAP schools, skilled teachers have the opportunity to serve as master and mentor teachers, receiving additional compensation for providing high levels of support to career teachers. Along with administrators, master and mentor teachers form a leadership team to deliver school-based professional support and appraise teachers’ performance. • Ongoing applied professional growth. Teachers in TAP schools participate in weekly cluster group meetings, led by master and mentor teachers, in which they examine student data, engage in collaborative planning and learn instructional strategies that have been field-tested in their respective schools. Professional development continues into each classroom as master teachers model lessons, observe classroom instruction, and support career teachers in the improvement of their teaching methods. • Instructionally focused accountability. Teachers in TAP schools are observed in classroom instruction several times a year by multiple, trained observers. Student growth analysis complements these classroom observations, rounding out a multi-measure system of teacher appraisals. Observation results guide both formative feedback for one-on-one mentoring sessions and plans for cluster group meetings. • Performance-based compensation. Teachers in TAP schools have the opportunity to earn annual bonuses based on their observed skills, knowledge and responsibilities, the average academic growth of students in their classroom, and the entire school’s average growth in achievement. Master and mentor teachers receive additional compensation in recognition of their additional support roles and responsibilities. This brief presents findings from a quantitative case study evaluating Cross County School District’s implementation of their TIF grant over the first three years. The study specifically focuses on progress towards (1) increasing teacher collegiality, (2) improving teacher effectiveness, (3) increasing teacher retention, and (4) impacting student achievement. 2 Finding 1. Increased Teacher Collegiality The TAP System implements processes to increase teacher collaboration and collegiality. The career ladder, cluster group meetings, and even the teacher evaluation process bring teachers together in a mutually supportive and enriching environment. When asked about collegiality in their school on NIET’s annual survey, Cross County School District teachers have agreed quite strongly that they have a high level of collegiality in their schools (see Figure 1). 2018-19 8% 90% 2017-18 10% 85% 2016-17 18% 77% 0% 20% 40% 60% 80% 100% % Moderately Agree % Strongly Agree Figure 1. Cross County School District teachers’ perceptions of high levels of collegiality in their school, 2016-17 to 2018-19. Finding 2. Increased Teacher Effectiveness At its core, the TAP System works to raise instructional effectiveness throughout the curriculum. The TIF grant targets advancing teacher effectiveness and retaining effective teachers in the district. The dual approach works well for Cross County School District. Cross County School District teachers are observed four times a year by multiple certified raters using the NIET Teaching Standards Rubric. They receive a Skills, Knowledge, and Responsibilities (SKR) score based on these observations and a responsibility survey (see Appendix for details on the methodology). To be considered effective, teachers must receive an overall score above a fixed benchmark. As shown in Figure 2, from the year before grant implementation (2015-16) to the third year of the grant (2018- 19), the percentage of effective teachers increased by 19 points. 94% 100% 88% 84% 75% 75% Percentage of Effective 50% Teachers 25% 0% 2015-16 2016-17 2017-18 2018-19 School Year Figure 2. Percentage of effective teachers, 2015-16 to 2018-19. “The focus on data-driven instructional decisions molds the culture of our school into one where teachers and their students strive for greatness.” —Jessica Stacy Principal, Cross County Elementary Technology Academy 3 Finding 3. Improved Teacher Retention The TAP System supports performance-based compensation for effective teachers and collegiality among all teachers through ongoing support from teacher leaders. The combination of these two TAP System elements has helped to decrease turnover of effective teachers. Examination of teacher rosters and effectiveness data (SKR scores) reveals that from 2016-17 to 2019-20 the percentage of effective teachers retained by Cross County School District rose by 21 points (see Figure 3). 96% 100% 84% 86% 75% Percentage 75% of Effective Teachers 50% Retained 25% 0% 2016-17 2017-18 2018-19 2019-20 School Year Figure 3. Percentage of effective teachers retained the following school year, 2016-17 to 2019-20. 4 Finding 4. Improved Student Achievement Within the TAP System theory of action, success with advancing teacher effectiveness and retaining effective teachers leads to improved student performance. The Cross County School District and NIET partnership focuses on building the capacity of teachers. These efforts ensure that change is lasting across the schools and that it impacts not only the current group of students, but those to follow. From the year before grant implementation (2015-16) through the third year of the grant (2018-19), the percentage of Cross County School District students passing the state assessment increased for all tested subjects: English, reading, mathematics, and science (see Figure 4). When compared with the average performance of districts with similar demographic characteristics and passing rates on the 2015-16 state assessment, Cross County School District outperformed them on every subject during the second year of the grant (2017-18), and sustained that increase through the third year for all subjects. English Reading 74% 74% 80% 80% 69% 69% 60% 68% 68% 70% 70% 60% 44% 44% Percentage Percentage 36% 36% 40% 40% of Students of Students Passing State Passing State 36% 38% 38% 33% Assessment 20% Assessment 20% 0% 0% 2015-16 2016-17 2017-18 2018-19 2015-16 2016-17 2017-18 2018-19 School Year School Year Cross County School District Comparison Cross County School District Comparison Mathematics Science 80% 80% 55% 60% 46% 48% 50% 60% 46% 46% Percentage Percentage 38% 39% 40% 40% of Students 46% 44% 46% 47% of Students Passing State Passing State 37% 37% 38% 39% Assessment 20% Assessment 20% 0% 0% 2015-16 2016-17 2017-18 2018-19 2015-16 2016-17 2017-18 2018-19 School Year School Year Cross County School District Comparison Cross County School District Comparison Figure 4. Percentage of students passing the state assessment, 2015-16 to 2018-19. 5 Conclusion Cross County School District works with NIET to strengthen the teacher workforce through a coherent and integrated set of strategies that are rooted in a common vision for instructional excellence. As a result, the district has experienced successes across key indicators of school improvement. Teachers report increased collaboration and collegiality over the course of the TIF grant. Improvements in teacher effectiveness and retention have occurred alongside improvement in student achievement, particularly in the last two years when retention of effective teachers topped 85%. Cross County School District also performed better than matched comparison districts in all tested subjects during those last two years. Cross County School District historically faced challenges in the recruitment, development, and retention of effective teachers. The TIF grant has allowed Cross County School District to leverage school, district, and national nonprofit resources to overcome these challenges through continuous improvement of their implementation of the TAP System. The results after three years of grant implementation indicate improved teacher collegiality, teacher effectiveness, teacher retention, and student achievement. 6 References Aragon, S. (2016). Teacher shortages: What we know. Teacher Shortage Series (Teacher Shortage Series). [Brief]. Education Commission of the States. https://eric.ed.gov/?id=ED565893 Arkansas Department of Education (2019a). Cross County School District. Retrieved from https://myschoolinfo.arkansas.gov/Districts/Detail/1901000 Arkansas Department of Education (2019b). ACT Aspire. Retrieved from http://www.arkansased.gov/divisions/learning-services/assessment/act-aspire. Carver-Thomas, D., & Darling-Hammond, L. (2019). The trouble with teacher turnover: How teacher attrition affects students and schools. Education Policy Analysis Archives, 27(36). doi:10.14507/epaa.27.3699 DeFeo, D. J., Tran, T., Hirshberg, D., Cope, D., & Cravez, P. (2017). The cost of teacher turnover in Alaska. Center for Alaska Education Policy Research, University of Alaska Anchorage. https://scholarworks.alaska.edu/handle/11122/7815 Garcia, E., & Weiss, E. (2019). The teacher shortage is real, large and growing, and worse than we thought: The first report in “The Perfect Storm in the Teacher Labor Market” series. Economic Policy Institute. https://www.epi.org/publication/the-teacher-shortage-is-real-large-and-growing-and-worse-than-we- thought-the-first-report-in-the-perfect-storm-in-the-teacher-labor-market-series/ Geiger, T., & Pivovarova, M. (2018). The effects of working conditions on teacher retention. Teachers and Teaching, 24, 1–22. doi:10.1080/13540602.2018.1457524 Latterman, K., & Steffes, S. (2017, October). Tackling teacher and principal shortages in rural areas. Legis Brief, National Conference of State Legislatures, 25(40). https://www.ncsl.org/research/education/tackling- teacher-and-principal-shortages-in-rural-areas.aspx Lazarev, V., Toby, M., Zacamy, J., Lin, L., & Newman, D. (2017). Indicators of successful teacher recruitment and retention in Oklahoma rural school districts (REL 2018–275). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. https://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2018275.pdf National Center for Education Statistics (n.d.-a). Common Core Data Public school district data for the 2018- 2019, 2019-2020 school years. https://nces.ed.gov/ccd/districtsearch/ National Center for Education Statistics (n.d.-b). Education Demographics and Geographic Estimates, ACS-ED District Demographic Dashboard 2014-18, Cross County School District, AR. https://nces.ed.gov/Programs/Edge/ACSDashboard/0507740 National Institute for Excellence in Teaching (n.d.). TAP implementation handbook. Santa Monica, CA. Retrieved from https://www.niet.org/our-work/brochures/category/tap-implementation-handbook Snyder, T. D., de Brey, C., & Dillow, S. A. (2019). Digest of Education statistics 2018 (NCES 2020-009). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. https://nces.ed.gov/pubs2020/2020009.pdf Sorensen, L. C., & Ladd, H. F. (2020). The hidden costs of teacher turnover. AERA Open, 6(1), 2332858420905812. https://doi.org/10.1177/2332858420905812 Sutcher, L., Darling-Hammond, L., & Carver-Thomas, D. (2016). A coming crisis in teaching? Teacher supply, demand, and shortages in the U.S. Palo Alto, CA: Learning Policy Institute. https://learningpolicyinstitute.org/product/coming-crisis-teaching Ulferts, J. D. (2016). A brief summary of teacher recruitment and retention in the smallest Illinois rural schools. Rural Educator, 37(1), 14–24. doi:10.35608/ruraled.v37i1.292 7 Appendix: Methodology Data Sources All data in the study come from Cross County School District, NIET, and the Arkansas Department of Education. Specifically, annual teacher rosters are available from Cross County School District; annual teacher survey results and teacher effectiveness data at the teacher level are available from NIET; and districtwide enrollment, free or reduced-price meals percentage, and student achievement data are publicly available from the Arkansas Department of Education website. Measures Teacher Collegiality. The TAP Teacher Attitude Survey is administered electronically through a third-party vendor who warehouses all responses. The Collegiality factor represents a composite of nine questions that ask about teachers’ overall satisfaction with their colleagues, school leaders, and the school environment. In addition, this construct measures the degree to which teachers feel supported, respected, and valued as contributors within the school community. The percentage of teachers who report agreement with these statements is reported. The survey response rate for Cross County School District teachers ranged from 94% to 100% over the three- year period. Teacher Effectiveness. Skills, Knowledge, and Responsibility (SKR) scores measure teacher effectiveness. Multiple certified raters observe teachers four times per year on 19 indicators of instructional practice. A weighted average of observation scores and seven responsibility indicators creates an overall SKR score for each teacher. The SKR scores range from a 1.0 (unsatisfactory performance) to 5.0 (exemplary performance) in half point increments. An overall score above a fixed benchmark signifies an effective teacher. Effective Teacher Retention. Teacher rosters and effectiveness data (SKR scores) identify effective teachers who are retained each school year. The teacher retention rate divides the number of effective teachers who return to the same school or district in the current year by the total number of effective teachers in the school or district in the previous year. Student Achievement. The Arkansas Board of Education uses ACT Aspire as the statewide summative assessment. The ACT Aspire is a series of end-of-year tests in the following subjects: English, reading, mathematics, and science. The assessments are for all public school students in grades 3-10, unless qualified for an alternative assessment (Arkansas Department of Education, 2019b). The Arkansas Department of Education publicly reports these data as percentages of students passing the state assessment (i.e., on-track to be college and career ready). Comparison Groups To contextualize student achievement, a series of comparison groups were selected using 2015-16, the year before the grant started, as the base year. The comparison districts were selected based on demographic characteristics comparable to Cross County School District: enrollment (526 - 726 students), free or reduced- price meals percentage (53% - 83%), and percentage passing the state assessments (English, 63% - 73%; reading, 30% - 42%; mathematic, 40% - 52%; and science, 32% - 41%). The ranges for the percentage passing the state assessments were selected to place Cross County School District at the center of the distribution of districts, which resulted in Cross County School District’s percentage passing being approximately equal to the comparison group average in 2015-16. 8 Website: Contact: Address: www.niet.org 1-800-575-NIET 1801 West End Avenue, Suite 920 Nashville, TN 37203 The contents of this report were developed under a grant from the U.S. Department of Education. However, the contents do not necessarily represent the policy or endorsement of the U.S. Department of Education. 9