Search

Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999-2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (>85%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models., Comment: Health Serv Outcomes Res Method (2022)

The world is becoming increasingly complex, both in terms of the rich sources of data we have access to as well as in terms of the statistical and computational methods we can use on those data. These factors create an ever-increasing risk for errors in our code and sensitivity in our findings to data preparation and execution of complex statistical and computing methods. The consequences of coding and data mistakes can be substantial. Openness (e.g., providing others with data code) and transparency (e.g., requiring that data processing and code follow standards) are two key solutions to help alleviate concerns about replicability and errors. In this paper, we describe the key steps for implementing a code quality assurance (QA) process for researchers to follow to improve their coding practices throughout a project to assure the quality of the final data, code, analyses and ultimately the results. These steps include: (i) adherence to principles for code writing and style that follow best practices, (ii) clear written documentation that describes code, workflow and key analytic decisions; (iii) careful version control, (iv) good data management; and (iv) regular testing and review. Following all these steps will greatly improve the ability of a study to assure results are accurate and reproducible. The responsibility for code QA falls not only on individual researchers but institutions, journals, and funding agencies as well., Comment: 29 pages, 6 figures

In fall 2017, Propel Schools initiated the expansion of one of its schools, Propel Montour. Originally a single K-8 school with two classrooms per grade, Montour added a new high school and expanded into separate elementary and middle schools over four years, adding a classroom to each grade. In this report, the authors used difference-in-difference and doubly robust regression models to examine the academic and behavioral experiences of both continuing and expansion Montour students from fall 2017 through the onset of the coronavirus disease 2019 (COVID-19) pandemic in spring 2020. The analyses did not find evidence that the academic and behavioral experiences of either the continuing or expansion students fell below what would have been expected absent expansion. Although the expansion experiences of a single charter management organization (CMO) network might be idiosyncratic to that entity and the locale and population it serves, this report should be of interest to similarly structured CMOs, the organizations from which they might seek expansion funding, those involved in school choice policy, and state and local education leaders and policymakers. [This report was supported through a subcontract with Propel Schools Foundation.]

This report presents findings from a 2016 American Teacher Panel survey, which gathered information about teachers' instructional materials for mathematics, their understanding of their mathematics standards, and their standards-aligned practices. The authors also consider how teachers' instructional materials might support their understanding of their standards and practice. This report follows up on findings from a 2015 survey and is intended to support decisionmaking regarding policies and related to state standards and teachers' instructional practice. The authors found that most of the materials that teachers reported using regularly for their instruction during the 2015-2016 school year were not highly aligned with Common Core State Standards. In addition, they were not always able to identify approaches and content aligned with their state standards for mathematics. Over one-third of teachers reported that their students engaged in various standards-aligned practices to a great extent. The authors found positive and significant relationships between teachers' use of standards-aligned instructional materials and their standards-aligned practices, as well as some limited relationships between teachers' standards-aligned materials and their understanding of their standards. The report concludes by suggesting that teachers need a better understanding of their state standards if they are to engage their students in practices aligned with those standards. Furthermore, evidence from this report indicates that instructional materials closely aligned with state standards may support teachers' understanding of their standards and their standards-aligned practices. Key Findings: (1) Most of the materials that teachers reported using regularly for their instruction during the 2015-2016 school year were not highly aligned with the Common Core State Standards: (a) About 16 percent of elementary mathematics teachers and 5 percent of secondary mathematics teachers reported regularly using materials with a high degree of alignment; and (b) Teachers in states that have not formally adopted the Common Core were not using highly aligned materials, by the authors' definition; (2) A majority of teachers were able to identify Common Core-aligned topics at their grade level: (a) At the same time, many teachers also indicated that many topics not aligned with the Common Core should be addressed at their grade level; and (b) Less than one-third of teachers were able to order two sets of standards from the lowest to the highest grade level, although most teachers could order at least one set of standards correctly; (3) Over one-third of teachers reported that their students engaged in various standards-aligned practices to a great extent: (a) On the other hand, roughly 20 percent of teachers reported asking students to look for and make use of structure or construct viable arguments, two Standards for Mathematical Practice emphasized by the Common Core; (b) Teachers using at least one aligned main material more frequently reported their students engaging to a great extent in standards-aligned practices than teachers not using at least one aligned main material; and (c) Teachers with more vulnerable students were less likely to report engaging their students in standards-aligned practices than teachers serving less vulnerable students. Recommendations: (1) By providing teachers with materials that are closely aligned with state standards, states and districts may encourage more standards-aligned practices; and (2) Provision of aligned curricula is only the first step in supporting teachers' high-quality, standards-aligned instruction. States and districts should clarify, via professional development and other means, the topics and sequencing of those topics teachers should be teaching at their grade level, as well as mathematics approaches and concepts closely aligned with state standards.

The basic concept of personalized learning (PL) -- instruction that is focused on meeting students' individual learning needs while incorporating their interests and preferences -- has been a longstanding practice in U.S. K-12 education. Options for personalization have increased as personal computing devices have become increasingly affordable and available in schools and developers created software to support individual student learning. In recent years, it has become more common for schools to embrace schoolwide models of PL. We collected data from schools in the Next Generation Learning Challenges (NGLC)'s Breakthrough School Models program. Our study seeks to describe the practices and strategies these schools used to implement PL, understand some of the challenges and facilitators, and consider these alongside achievement findings to discern patterns that may be informative. Teachers and students reported higher levels of many aspects of personalization than their counterparts in a national sample. These included time for one-on-one tailored support for learning; using up-to-date information on student progress to personalize instruction and group students; students tracking their own progress; competency-based practices; and flexible use of staff, space, and time. However, some more-difficult-to-implement aspects did not appear to differ from practices in schools nationally, such as student discussions with teachers on progress and goals; keeping up-to-date documentation of student strengths, weaknesses, and goals; and student choice of topics and materials. We estimate study students gained about 3 percentile points in mathematics relative to a comparison group of similar students. In reading, there was a similar trend, though it was not statistically significant. Low-performing and high-performing students appeared to benefit.