Search

The District of Columbia Public Schools (DCPS) has prioritized efforts to support students' social and emotional learning (SEL) competencies, such as perseverance and social awareness. To measure students' SEL competencies and the school experiences that promote SEL competencies (school climate), DCPS began administering annual surveys to students, teachers, and parents in 2017/18. DCPS partnered with the Mid-Atlantic Regional Educational Laboratory to study how the district could use these surveys to improve students' outcomes. The study found the following: (1) Students' SEL competencies and school experiences are the most favorable in elementary school and the least favorable in middle school and the beginning of high school. This pattern suggests that schools might provide targeted supports before or during grades 6-10 to promote SEL competencies and school experiences when students need the most support; (2) The trajectories of students' SEL competencies and school experiences differed in different schools, to a similar degree as trajectories in academic measures like test scores. To understand why changes in SEL competencies and school experiences differ across schools, DCPS could explore differences in practices between schools with better and worse trajectories. In addition, DCPS could provide targeted support to schools with lower levels of positive change; (3) Of the SEL competencies and school experiences in DCPS's survey, self-management--how well students control their emotions, thoughts, and behavior--is most related to students' later academic outcomes. Programs or interventions that target self-management might have the most potential for improving students' outcomes compared to those that target other SEL competencies or school experiences; (4) In statistical models designed to predict students' future academic outcomes, SEL competency and school experience data add little accuracy beyond prior academic outcomes (such as achievement test scores and attendance) and demographic characteristics. Prior academic outcomes and demographic characteristics predict later outcomes with a high degree of accuracy, and they may implicitly incorporate the SEL competencies and school experiences. These findings suggest that DCPS would not need to use SEL competencies and school experiences to identify whether or not students are at risk of poor academic outcomes; and (5) Student, teacher, and parent reports on SEL competencies and school experiences are positively related across schools, but they also exhibit systematic differences, suggesting that some respondent groups may not be aligned in their view of SEL competencies and school experiences. These differences may serve as a tool to help DCPS target efforts to improve communication among students, teachers, and parents.

In 2014, The Wallace Foundation launched the Principal Supervisor Initiative (PSI), a four-year, $24 million effort to redefine principal supervision in six urban school districts. The PSI aimed to help districts overhaul a position traditionally focused on administration, operations, and compliance to one dedicated to developing and supporting principals to be effective instructional leaders in their schools. Principal supervisors are a potential point of leverage for supporting and developing principals' effectiveness, but little is known about the effectiveness of this approach. The overarching hypothesis of the PSI was that changing the role of principal supervisors from overseeing operations to developing principals' instructional leadership practices could drive improvement in principal effectiveness. This report describes districts' experiences with the PSI across the years of the initiative and analyzes its effects on teachers' perceptions of principals' performance. Specifically, the report addresses the following questions: (1) What is the theory of change for revising the principal supervisor's role?; (2) What were districts' and principal supervisors' experiences with the PSI?; (3) What were principals' experiences with the PSI?; (4) How did the initiative affect teachers' perceptions of principals' performance, and how did teachers' perceptions of principals' performance and the effects of the PSI vary across districts and principal supervisors?; (5) How do key components of the PSI relate to changes in teachers' perceptions of principals' performance?; and (6) How did the principal supervisor role in PSI districts compare with that in other urban districts? The report also discusses the sustainability of changes made during the PSI and lessons learned for revising the principal supervisor role. The analyses of data from site visits, interviews, and surveys focused on districts', supervisors', and principals' experiences, successes, and challenges throughout the initiative; lessons learned; and plans for sustaining the accomplishments of the PSI after the initiative ended. The authors integrated the descriptive survey results with the qualitative data. The PSI brought substantial changes to all six districts. The initiative was successful in changing the principal supervisor role to one that focused on providing support and guidance to schools and principals. Principals' perceptions of their supervisors' effectiveness improved over the course of the initiative. Districts restructured their central offices to better support schools and meet their needs. Changes districts made during the PSI led to a shared understanding of the principal supervisor role and equipped principal supervisors to better support principals. However, the PSI did not improve teachers' perceptions of principals' performance, on average.

Helping principals improve their leadership practices is a common use of federal funds and one way to improve instruction and student achievement. This study sought to better understand the effectiveness of an intensive principal professional development program focused primarily on helping principals conduct structured observations of teachers' classroom instruction and provide targeted feedback based on those observations. The report found that the program did not change principal practices as intended or improve students' achievement. However, improvements in some of the targeted principal practices were positively associated with improved student achievement. Attention on how to change these practices might be a potential avenue for improving the program. [For the one page highlights to this study, "The Effects of a Principal Professional Development Program Focused on Instructional Leadership. Study Highlights. NCEE 2020-0002," see ED599490. For the appendices to this study, "The Effects of a Principal Professional Development Program Focused on Instructional Leadership: Appendices. NCEE 2020-0002," see ED599491.]

The four appendices in this report accompany the full report "The Effects of a Principal Professional Development Program Focused on Instructional Leadership. NCEE 2020-0002," (ED599489). They include: (1) The Study's Principal Professional Development Program; (2) Study Design, Data Collection, and Analytic Methods; (3) Supplemental Tables; and (4) Interpreting Study Findings: Alternatives to P-Values and Statistical Significance. [For the study highlights to this report, "The Effects of a Principal Professional Development Program Focused on Instructional Leadership. Study Highlights. NCEE 2020-0002," see ED599490.]

This article develops design-based ratio estimators for clustered, blocked randomized controlled trials (RCTs), with an application to a federally funded, school-based RCT testing the effects of behavioral health interventions. We consider finite population weighted least squares estimators for average treatment effects (ATEs), allowing for general weighting schemes and covariates. We consider models with block-by-treatment status interactions as well as restricted models with block indicators only. We prove new finite population central limit theorems for each block specification. We also discuss simple variance estimators that share features with commonly used cluster-robust standard error estimators. Simulations show that the design-based ATE estimator yields nominal rejection rates with standard errors near true ones, even with few clusters.

Evaluators of education interventions increasingly need to design studies to detect impacts much smaller than the 0.20 standard deviations that Cohen (1988) characterized as "small." For example, an evaluation of Response to Intervention from the Institute of Education Sciences (IES) detected impacts ranging from 0.13 to 0.17 standard deviations (Balu et al. 2015), and IES' evaluation of the Teacher Incentive Fund detected impacts of just 0.03 standard deviations (Chiang et al. 2015). The drive to detect smaller impacts is in response to strong arguments that in many contexts, impacts once deemed "small" can still be meaningful (Kane 2015). Though based on a compelling rationale, the drive to detect smaller impacts may create a new challenge for researchers: the need to guard against relatively smaller biases. When studies were designed to detect impacts of 0.20 standard deviations or larger, it may have been reasonable for researchers to regard small biases as ignorable. For example, a bias of 0.03 standard deviations might have been ignorable in a study that could only detect an impact of 0.20 standard deviations. But in a study designed to detect much smaller impacts, such as Chiang et al. (2015) in which the impact estimate was 0.03 standard deviations, a bias of 0.03 standard deviations is no longer small--it is enormous. The authors define asymdystopia as a context in which a larger sample size is not necessarily better and could even be worse from the perspective of controlling the Type 1 error rate. The authors examine the potential for asymdystopia as studies are powered to detect smaller impacts, where even small biases may lead to false inferences about the existence or magnitude of an impact. The authors focus on the potential for bias from attrition in the case of randomized controlled trials (RCTs) and bias from regression misspecification in the case of regression discontinuity designs (RDDs). While the methodological details are distinct, in both cases they are unpacking a source of bias that may become increasingly problematic when studies are designed to detect smaller impacts. The two main research questions are: (1) How problematic is attrition bias in RCTs as studies are powered to detect smaller impacts? and (2) How problematic is functional form misspecification bias in RDDs as studies are powered to detect smaller impacts? Overall, the findings suggest that biases that might have once been reasonably ignorable can pose a real threat in evaluations that are powered to detect small impacts. This paper identifies and quantifies some of these biases, and shows that they are important to consider when designing evaluations and when analyzing and interpreting evaluation findings. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Evaluators of education interventions are increasingly designing studies to detect impacts much smaller than the 0.20 standard deviations that Cohen (1988) characterized as "small." While the need to detect smaller impacts is based on compelling arguments that such impacts are substantively meaningful, the drive to detect smaller impacts may create a new challenge for researchers: the need to guard against smaller biases. The purpose of this paper is twofold. First, we examine the potential for small biases to increase the risk of making false inferences as studies are powered to detect smaller impacts, a phenomenon we refer to as asymdystopia. We examine this potential for two of the most rigorous designs commonly used in education research--randomized controlled trials (RCTs) and regression discontinuity designs (RDDs). Second, we recommend strategies researchers can use to avoid or mitigate these biases.

A new design-based theory has recently been developed to estimate impacts for randomized controlled trials (RCTs) and basic quasi-experimental designs (QEDs) for a wide range of designs used in social policy research (Imbens & Rubin, 2015; Schochet, 2016). These methods use the potential outcomes framework and known features of study designs to connect statistical methods to the building blocks of causal inference. They differ from model-based methods that have commonly been used in education research, including hierarchical linear model (HLM) methods and robust cluster standard error (RCSE) methods for clustered designs. In comparison to model-based methods, the design-based methods tend to make fewer assumptions about the nature of the data and also more explicitly account for known information about the experimental and sampling designs. While these theoretical differences suggest the corresponding estimates might differ, it is unclear how much of a practical difference it makes to use design-based methods versus more conventional model-based methods. This study addresses this question by re-analyzing nine past RCTs in the education area using both design- and model-based methods. The study uses real data, rather than simulated data, to better explore the differences that would arise in practice. In order to investigate the full scope of differences between the methods, the study uses data generated from different types of randomization designs commonly used in social policy research: (1) non-clustered designs in which individuals are randomized; (2) clustered designs in which groups are randomized; (3) non-blocked designs in which randomization is conducted for a single population; and (4) blocked (stratified) designs in which randomization is conducted separately within partitions of the sample. The study conducts the design-based analyses using "RCT-YES," a free software package funded by the Institute of Education Sciences (IES) that applies design-based methods to a wide range of RCT designs (www.rct-yes.com). This report focuses on two analyses that compare model- and design-based methods, both of which suggest there is little substantive difference in the results between the two methods. For both analyses, the study uses a reference model-based method that is similar to the one used in the original evaluation. In the first analysis, the study compares the reference model-based method to a design-based method with underlying assumptions that most closely align with those of the reference model-based method. In the second analysis, the report presents a sensitivity check that compares the reference model-based method to an alternative design-based method. In particular, the alternative method is based on the default settings in the" RCT-YES" software, which correspond to an alternative set of plausible assumptions. The findings from both analyses suggest that model- and design-based methods yield very similar results in terms of the magnitude of impact estimates, statistical significance of the impact estimates, and implications for policy. To contextualize the differences in impact estimates between design- and model-based methods, the report also presents a third analysis, which compares estimates from two commonly used model-based methods: (1) HLM methods; and (2) linear models with ordinary least squares (OLS) assumptions and RCSE to account for clustering. Importantly, this analysis suggests that the differences between the design- and model-based methods (with similar assumptions) are no greater than the differences that would arise between commonly used, model-based methods. The study suggests that researchers should select estimators with assumptions that best suit the goals of their study regardless of whether they use a design- or model-based approach. Moreover, researchers should consider the trade offs between different assumptions, and how these assumptions affect the interpretation of findings. Appended are: (1) Hierarchical linear model methods; and (2) Detailed description of studies and results. [For related reports see: "What Is Design-Based Causal Inference for RCTs and Why Should I Use It? NCEE 2017-4025" (ED575014)and "Multi-Armed RCTs: A Design-Based Framework. NCEE 2017-4027 (ED575022).]