Search

Researchers face many choices when conducting large-scale multisite individually randomized control trials. One of the most common quantities of interest in multisite RCTs is the overall average effect. Even this quantity is non-trivial to define and estimate. The researcher can target the average effect across individuals or sites. Furthermore, the researcher can target the effect for the experimental sample or a larger population. If treatment effects vary across sites, these estimands can differ. Once an estimand is selected, an estimator must be chosen. Standard estimators, such as fixed-effects regression, can be biased. We describe 15 estimators, consider which estimands they are appropriate for, and discuss their properties in the face of cross-site effect heterogeneity. Using data from 12 large multisite RCTs, we estimate the effect (and standard error) using each estimator and compare the results. We assess the extent that these decisions matter in practice and provide guidance for applied researchers.

Community colleges are a large sector of postsecondary education. In 2016-2017, the United States had nearly 1,000 public 2-year postsecondary institutions (community colleges), serving almost nine million students, representing 39% of all undergraduates. The majority of entering community college students require developmental (or remedial) math. Success rates in the developmental math course sequence and college more broadly are discouragingly low. Policymakers, practitioners, and researchers alike are eagerly searching for reforms to improve success rates, but there is a dearth of causal evidence on the effectiveness of most proposed reforms. We sought to answer the following question: what effect does a modularized, computer-assisted, self-paced approach to developmental math (compared with a more "traditional" direct-instruction course alternative) have on students' likelihood of completing the developmental math course sequence? Findings from a randomized controlled trial (n =1,403) are presented. The program was well implemented; however, we did not find evidence that this approach was superior to the "traditional" math class. Although these results are disappointing, they are important because modularization and self-paced computer-assisted instruction are popular reforms.

The present article considers a fundamental question in evaluation research: "By how much do program effects vary across sites?" The article first presents a theoretical model of cross-site impact variation and a related estimation model with a random treatment coefficient and fixed site-specific intercepts. This approach eliminates several biases that can arise from unbalanced sample designs for multisite randomized trials. The article then describes how the approach operates, explores its assumptions, and applies the approach to data from three large welfare-to-work trials. The article also illustrates how to report cross-site impact findings and presents diagnostics for assessing these findings. To keep the article manageable, it focuses on experimental estimates of effects of program assignment (effects of intent to treat), although the ideas presented can be extended to analyses of multisite quasi-experiments and experimental estimates of effects of program participation (complier average causal effects).

Multisite trials, in which individuals are randomly assigned to alternative treatment arms within sites, offer an excellent opportunity to estimate the cross-site average effect of treatment assignment (intent to treat or ITT) "and" the amount by which this impact varies across sites. Although both of these statistics are substantively and methodologically important, only the first has been well studied. To help fill this information gap, we estimate the cross-site standard deviation of ITT effects for a broad range of education and workforce development interventions using data from 16 large multisite randomized controlled trials. We use these findings to explore hypotheses about factors that predict the magnitude of cross-site impact variation, and we consider the implications of this variation for the statistical precision of multisite trials.

In the "individually randomized group treatment" (IRGT) experimental design, individuals are first randomly assigned to a treatment arm or a control arm, but then within each arm, are grouped together (e.g., within classrooms/schools, through shared case managers, in group therapy sessions, through shared doctors, etc.) to receive services. In the common setting where the second-stage grouping is not experimentally controlled, we demonstrate that the standard error estimators for the estimated treatment effects from both random and fixed effects models are biased because parameters required to compute the standard errors are not identifiable from the data. We discuss implications for designing and analyzing IRGTs and provide suggestions for mitigating the bias.

Community colleges play a vital role in higher education, enrolling more than one in every three postsecondary students. While their market share has grown over the past 50 years, students' success rates remain low. Consequently, community college stakeholders are searching with mounting urgency for strategies that increase rates of success. We evaluate the effects of one such strategy, learning communities, from a randomized trial of over 1,500 students at a large urban college in the City University of New York (CUNY) system. We find that the program's positive effects on short-term academic progress (credit accumulation) are maintained seven years after random assignment. We find some limited evidence that the program positively affected graduation rates, particularly for those students without remedial English needs, over this time period. While there is not clear evidence that the program improved economic outcomes, this article concludes by offering sobering reflections on trying to detect the effects of higher education interventions on future earnings.

In some experimental evaluations of classroom-level interventions it is not practically feasible to randomly assign teachers to experimental conditions. Given such restrictions, researchers may randomly assign students to experimental conditions and consider the teacher to be a part of the intervention. However, in an individually randomized evaluation of a classroom-level intervention, unless teachers are randomized to experimental conditions, it will not be clear whether differences in outcomes between program and control group students are a result of the core components of the intervention or the teachers (i.e., teacher effects). This article clarifies the correct interpretation of "program impacts" when this study design is used. In addition, using the magnitude of estimated teacher effects from past research, this article demonstrates that, if teachers are not randomly assigned to experimental conditions, it is difficult to establish whether the program works or whether the types of teachers selected to teach in program classrooms are simply more or less effective than their control group counterparts. The significant implications of the correct causal interpretation are discussed, and the limitations of this research design are explored. (Contains 2 figures, 2 tables and 8 footnotes.)