Search

Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999-2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (>85%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models., Comment: Health Serv Outcomes Res Method (2022)

Most states' teacher evaluation systems have changed substantially in the past decade. New evaluation systems typically require school leaders to observe teachers' classrooms two to three times a school year instead of once (Doherty & Jacobs, 2015). The feedback that school leaders provide to teachers after these observations is a key but understudied step in the teacher evaluation cycle. The feedback and subsequent professional development are intended to help teachers change their instructional practices and improve student achievement (Correnti & Rowan, 2007; DeNisi & Sonesh, 2011; Taylor & Tyler, 2012). However, little is known about the feedback that school leaders provide to teachers following classroom observations or about how to train leaders to make that feedback more effective. This study examined the impact of disseminating a detailed checklist intended to structure an effective feedback conference between a school leader and a teacher following a classroom observation. The feedback conference checklist is a modified version of one created by the Carnegie Foundation for the Advancement of Teaching (Tang & Chow, 2007). The checklist, along with short testimonial videos, was a low-cost, low-intensity intervention provided to a randomly selected half of 339 participating New Mexico principals in fall 2015 by the study team. These principals' schools constituted the treatment group. Principals in the treatment group schools received an email with an attachment containing a guide and a 24-item feedback conference checklist, plus a hyperlink to a three-minute testimonial video featuring a principal. Principals were encouraged to distribute the checklist to other school leaders and to use the checklist in all their feedback conferences in the 2015/16 school year. Principals were also asked to distribute the checklist to all their teachers in order to promote greater teacher participation in the feedback conference. The study team also emailed the same checklist plus a hyperlink to a three-minute testimonial video featuring a teacher to up to 10 randomly sampled teachers in each treatment group school. The other half of the principals in the study schools formed the control group. Each of the control group principals received a two-page principal guide as an email attachment in fall 2015. The two-page guide reprised the five tips about feedback included in the summer 2015 New Mexico Public Education Department--sponsored professional development for principals and informed principals about the study. In addition, the study team sent up to 10 randomly sampled teachers in each control group school a two-page teacher guide summarizing the teacher evaluation system (Skandera, 2013) and teachers' right to receive post-observation feedback. All principals and teachers in both the treatment group and the control group who consented to be in the study were asked to complete an online survey (one for principals, another for teachers) in spring 2015 and again in spring 2016. The main outcomes of the study were principals' and teachers' reports of the impacts of the checklist and testimonial video on the perceived quality of feedback conferences following formal classroom observations; principals' recommendations for and teachers' take-up of professional development; and the quality of teachers' subsequent instructional practices as measured by principals' formal classroom observation scores and teachers' self-reported scores. Additional exploratory outcomes included the impact of the checklist on student achievement (school-average math and English language arts scores on the Partnership for Assessment of Readiness for College and Careers assessment) and school report card grades (reported as an A, B, C, D, or F of multiple measures of a school's student achievement) compiled annually by the New Mexico Public Education Department. The study also documented how many recipients reported using the checklist and what they thought about it. The checklist had few clear impacts on the quality of feedback, professional development outcomes, instructional practice, or student achievement. There were two exceptions: teachers who received the checklist reported that their principals were less likely to dominate the feedback conferences, and they reported that they were more likely to follow their principal's professional development recommendations. Use of the checklist in the treatment group was moderate: 77 percent of principals surveyed who received the checklist reported viewing it, and 58 percent said they used it with one or more teachers. At the same time, 29 percent of control group principals (who were not emailed the checklist) reported that they had seen the checklist, and 10 percent reported using it with one or more teachers. The relatively moderate use of the checklist by treatment group principals, combined with the reports by some control group school leaders that they were using it, implies that the estimated impacts of using the checklist would be larger than the estimated impacts of receiving it. Though distribution of the feedback conference checklist to principals and teachers had a few modest impacts, this study indicates that distributing the checklist is unlikely by itself to substantially alter feedback conferences, teachers' classroom practices, or student achievement, at least during the first school year in which the checklist is used. This study suggests that only a fraction of school leaders are likely to use the checklist if it is distributed in the low-cost manner followed in this study. But the checklist may also have failed to help principals overcome common barriers to effective feedback, such as providing critical comments to teachers or recommending appropriate professional development. The study results raise the possibility that additional (or different) investments might be necessary to improve school leaders' feedback conferences with teachers--for example, pairing training with written guidance.

TNTP (formerly The New Teacher Project), an organization committed to ending educational inequities by promoting the recruitment, training, and retention of high-quality teachers and school leaders, implemented its Teacher Effectiveness and Certification (TEACh) initiative in three urban school districts. Through TEACh, TNTP works with school districts to develop a within-district process to recruit, prepare, and certify teacher candidates, as well as hire and support them in their first year. As part of an evaluation of this initiative, RAND investigated each district program's implementation and costs, the effects of TEACh on the recruitment and retention of teachers, and the relative performance of those teachers. This is the final report for that evaluation. Key findings include: (1) Communication was vital to the implementation of TEACh; (2) The impact of TEACh was clearest in its supply of racially diverse teachers for hard-to-staff district roles; and (3) The cost of TEACh was somewhat higher than that of similar programs, but the costs of various components in different districts varied widely.

Across the country, school districts, their stakeholders, and policymakers have become increasingly concerned about suspensions, particularly about suspending students from elementary school and disproportionately suspending ethnic/racial minority students. Suspended students are less likely to graduate, possibly because they miss the instructional time they need to advance academically. Restorative practices have gained buy-in in the education community as a strategy to reduce suspension rates. Proactively improving relationships among students and staff and building a sense of community in classrooms and schools may make students less inclined to misbehave. And addressing severe misbehavior through a restorative approach may help students realize the impacts of their actions and make them less likely to offend again. This study of the implementation of restorative practices in the Pittsburgh Public Schools district (PPS) in school years 2015-16 and 2016-17 represents one of the first randomized controlled trials of the effects of restorative practices on classroom and school climates and suspension rates. The authors examined a specific restorative practices program -- the International Institute for Restorative Practices' SaferSanerSchools™ Whole-School Change program -- implemented in a selected group of PPS schools under a program called Pursuing Equitable and Restorative Communities, or PERC. The researchers found that PERC achieved several positive effects, including an improvement in overall school climates (as rated by teachers), a reduction in overall suspension rates, and a reduction in the disparities in suspension rates between African American and white students and between low- and higher-income students. Key Findings: Effects of the Pursuing Equitable and Restorative Communities (PERC) program in Pittsburgh Public Schools: (1) Implementation of restorative practices through PERC improved overall school climates, as rated by teachers; (2) Implementation of restorative practices reduced the average suspension rate: During the study period, average suspension rates decreased in both PERC and non-PERC schools, but rates decreased more in PERC schools; (3) Suspension rates of African American students and of those from low-income families also went down in PERC schools, shrinking the disparities in suspension rates between African American and white students and between low- and higher-income students; (4) Academic outcomes did not improve in PERC schools, and actually worsened for grades 6-8; and (5) Arrest rates among PERC schools did not decrease. Recommendations: (1) Given the constraints on teachers' time, emphasize restorative practices that can be woven into the school day; (2) Ensure that school leaders understand and can model restorative practices, including by providing mandatory professional development, books and other materials, and coaching on restorative practices; (3) Establish a mechanism for school staff to meet at least once per month as a professional learning community on restorative practices; (4) Ensure that leaders at the district level can coordinate this work; (5) Set, and update, clear expectations regarding the use of restorative practices; and (6) Implement data collection systems to collect accurate information on all types of behavioral incidents and remedies.