Inside the Black Box of Teacher Evaluation in Rural Settings: Implementation Measures and Their Associations with Student Outcomes

Background: Race to the Top era teacher evaluation reforms sought to improve teacher effectiveness and, ultimately, student achievement through either accountability or developmental mechanisms (Donaldson, 2021). Quasi-experimental teacher evaluation studies find mixed results in urban and large-scale settings (Dee & Wyckoff, 2015; Steinberg & Sartain, 2015; Taylor & Tyler, 2012; Bleiberg et al., 2023). This study seeks to examine two important gaps in the teacher evaluation literature. First, evaluation reforms have been at the state and district levels; however, implementation is largely at the discretion of principals. This decentralization raises questions about implementation (Bleiberg et al., 2023), therefore it is crucial to extend our limited knowledge of implementation and its impact on important outcomes (Cohen et al., 2020). Research on evaluation implementation is comprised of interview and survey studies and has found that implementation varies widely (Donaldson & Woulfin, 2018; Kraft & Gilmour, 2016). We complement prior interview studies to provide quantitative-based insight into evaluation implementation, thereby equipping practitioners and decision-makers with more knowledge about which aspects of evaluation may be the most important. Second, this study focuses on evaluation in rural settings. The importance of this knowledge is underscored by previous findings suggesting that location matters. One study finds that the implementation of a reformed system benefitted higher-performing and lower-poverty schools more (Steinberg & Sartain, 2015), while another system was found to benefit schools that were lower-achieving, less white, and had less experienced teachers (Hunter & Bowser, 2021a). The former study examined an urban context, while the latter was a rural context, suggesting heterogeneous effects by urbanicity. Further evidence suggests that rural principals face different challenges in implementation than their urban counterparts (Hazi, 2016; Gilles, 2017; Klocko & Justis, 2019). It is critical to examine rural teacher evaluation because 53% of U.S. districts are rural (School Superintendents Association, 2017) and when making research-informed decisions, policymakers prefer studies in settings like their own (Nakajima, 2021); this means that most district-level policymaking bodies (rural school boards) do not have evaluation research they view as applicable to their schools. Purpose: Given the importance of understanding 1) evaluation implementation in rural settings, this study answers the following research question: To what extent are implementation measures associated with rural student math and reading achievement? Implementation measures include 1) number of observations received per teacher, 2) days lapsed between observations received, and 3) evaluator differentiation in number of observations received across teachers. These measures are important because: 1) Higher concentrations of observations and feedback may not result in instructional improvements (Hunter, 2019; Hunter & Springer, 2022). However, this may be due to principals dealing with the burden of observations (Kraft & Gilmour, 2016; Hunter & Springer, 2022). In contrast to previously studied systems, the evaluation system under examination in this study recommends more frequent yet substantially shorter observations. We therefore examine the number of observations received per teacher per year. 2) Days lapsed between observations (i.e., dispersion) for a given teacher is theoretically important as teachers require time to synthesize feedback and implement instructional improvements. To our knowledge, our study will be the first to examine dispersion as a measure of evaluation system implementation. 3) Novice and low performing teachers need the most support, however recent evidence finds that all teachers exhibit similar rates of performance score growth (Hunter, Curby, & Bowser, under review). This is problematic because it suggests that novice and low performing teachers receive inequitable instructional supports. Therefore, higher differentiation in the number of observations received across teachers may be indicative of increased instructional support for novice and low performing teachers. Context: The Network for Educator Effectiveness (NEE) is a low-stakes teacher evaluation system designed for rural schools. As of 2018-19, 216 districts and 864 schools across Missouri used NEE. There were 1,424 unique evaluators who observed 22,180 teachers. Over 90% of the districts that adopted NEE are rural. Data: Unique NEE administrative data from 2015-16 through 2018-19 include: observation-level data for all NEE schools, unique teacher and observer identifiers, date of observation, teacher performance indicators (e.g., questioning) observed per observation, and teacher-by-indicator-by-observation scores. We also use Missouri administrative data, including teacher, principal, and student demographics and student achievement. Methods: We answer our research question in two ways. First, Hunter and Bowser (2021) find that switching into NEE from Missouri's system is not associated with changes in achievement. However, this may mask heterogeneity from within-school variation in implementation based on previously discussed measures. We rely on school and year fixed effects to examine heterogeneity by 1) number of observations received and 2) dispersion of observations. The outcome of interest is student math/reading achievement scores and treatment is an indicator equal to 1 when a school switches into the NEE system from the Missouri system and 0 otherwise. Second, we use student mobility between schools that exhibit low differentiation in the number of observations received across teachers and those that exhibit high differentiation in the number of observations received across teachers to estimate the association between implementation differentiation and achievement within the NEE system. y_isdt= ??switch?_ist+X_isdt A+ W_sdt B+ ?_t+ ?_s+ ?_isdt where: 1. ?switch?_sdt = 1 if student i in school s in year t switches from a low differentiating school in year t-1 to a high differentiating school in year t and 0 if student i in school s in year t switches from a low differentiating school to another low differentiating school 2. ?switch?_sdt = 1 if student i in school s in year t switches from a high differentiating school in year t-1 to a low differentiating school in year t and 0 if student i in school s in year t switches from a high differentiating school to another high differentiating school We include student (X_isdt) and school characteristics (W_sdt), year fixed effects (?_t), and school fixed effects (? _s). 1 Model specifications following Cowan, J., Goldhaber, D. and Theobald, R. (2022).