Search

Developed by the City University of New York (CUNY), the Accelerated Study in Associate Programs (ASAP) is a comprehensive program that provides students with up to three years of financial and academic support and other services. In return, students are expected to enroll in classes full time and participate in essential program services. An earlier experimental evaluation found that ASAP nearly doubled graduation rates three years after students entered the study. This paper summarizes the effects on education and financial aid outcomes over an eight-year follow-up period, or five years after the end of program services. This paper also presents the educational investment in students associated with providing these services and any educational investment in students from the CUNY community colleges after the program services were no longer provided. Through the eight-year follow-up period, ASAP's effects on associate's degree receipt persisted, indicating that the program not only helped students graduate faster, but also helped some students who would have never graduated without the program earn a degree. ASAP also led to a significant increase in the amount of financial aid students received through federal and state grants. The net cost (the difference between the total program group costs and the total control group costs) averaged $13,838 per program group member over eight years. CUNY spent, on average, an additional $9,162 per degree earned for students offered ASAP, compared with the control group students over the eight-year follow-up period. The findings also show that the program resulted in increased financial aid from the Pell Grant Program and the New York State Tuition Assistance Program, and reduced the grant dollars these programs invested per degree earned.

Community colleges are a large sector of postsecondary education. In 2016-2017, the United States had nearly 1,000 public 2-year postsecondary institutions (community colleges), serving almost nine million students, representing 39% of all undergraduates. The majority of entering community college students require developmental (or remedial) math. Success rates in the developmental math course sequence and college more broadly are discouragingly low. Policymakers, practitioners, and researchers alike are eagerly searching for reforms to improve success rates, but there is a dearth of causal evidence on the effectiveness of most proposed reforms. We sought to answer the following question: What effect does a modularized, computer-assisted, self-paced approach to developmental math (compared with a more "traditional" direct-instruction course alternative) have on students' likelihood of completing the developmental math course sequence? Findings from a randomized controlled trial (n = 1,403) are presented. The program was well implemented; however, we did not find evidence that this approach was superior to the "traditional" math class. Although these results are disappointing, they are important because modularization and self-paced computer-assisted instruction are popular reforms.

A solid understanding of fractions is the cornerstone for acquiring proficiency with rational numbers and paves the way for learning advanced mathematical concepts, such as algebra. Fraction difficulties limit not only students' educational and vocational opportunities but also their ability to solve everyday problems. Students who exit 6th grade with inadequate understanding of fractions may experience far-reaching repercussions that lead to lifelong avoidance of mathematics. This paper presents the results of a randomized controlled trial (RCT) focusing on the first two cohorts of a larger efficacy investigation aimed at building fraction sense in students with mathematics difficulties. Teachers implemented an evidence-informed fraction sense intervention (FSI) within their 6th-grade intervention classrooms. The lessons draw from research in cognitive science as well as mathematics education research. Employing random assignment at the classroom level, multilevel modeling revealed a significant effect of the intervention on posttest fractions scores, after controlling for pretest fractions scores, working memory, vocabulary, proportional reasoning, and classroom attentive behavior. Students in the FSI group outperformed their counterparts in the control group with noteworthy effect sizes on most fraction measures. Challenges associated with carrying out school-based intervention research are addressed. [This is the online first version of an article published in "Journal of Experimental Child Psychology."]

Multisite trials that randomize individuals (e.g., students) within sites (e.g., schools) or clusters (e.g., teachers/classrooms) within sites (e.g., schools) are commonly used for program evaluation because they provide opportunities to learn about treatment effects as well as their heterogeneity across sites and subgroups (defined by moderating variables). Despite the rich opportunities they present, a critical step in ensuring those opportunities is identifying the sample size that provides sufficient power to detect the desired effects if they exist. Although a strong literature base for conducting power analyses for the moderator effects in multisite trials already exists, software for power analysis of moderator effects is not readily available in an accessible platform. The purpose of this tutorial paper is to provide practical guidance on implementing power analyses of moderator effects in multisite individual and cluster randomized trials. We conceptually motivate, describe and demonstrate the calculation of statistical power and minimum detectable effect size difference (MDESD) using highly accessible software. We conclude by outlining guidelines on power analysis of moderator effects in multisite individual randomized trials (MIRTs) and multisite cluster randomized trials (MCRTs). [This is the online first version of an article published in "Journal of Experimental Education."]

In lower- and middle-income countries, including Ghana, students in rural areas dramatically underperform their urban peers. Rural schools struggle to attract and retain professionally trained teachers (GES 2012; World Bank 2012). We explore one potential solution to the problem of teacher recruitment: distance instruction. Through a cluster randomized controlled trial, we estimate the impact of a program that broadcasts live instruction via satellite to rural primary school students. The program equipped classrooms in 70 randomly selected Ghanaian schools with the technology required to connect to a studio in Accra. An additional 77 schools served as the control. Instructors in Accra provided math and English lessons to classrooms in the treatment group. The model is interactive, and students in satellite classes could communicate in real time with their remote teachers. We estimate significant gains (p<0.05) in rural students' numeracy and foundational literacy skills. We find no impact on attendance and classroom time-on-task (as measured through unannounced classroom observations), suggesting that these gains may result from improved instructional quality rather than from increased instruction time.

Recent studies show that texting-based interventions can produce educational benefits in children across a range of ages. We study the effects of a text-based program for kindergarten parents, distinguishing a general program from one that adds differentiation and personalization based on the child's developmental level. Children in the differentiated and personalized program were 50 percent more likely to read at a higher level (p<0.01) compared to the general group; and their parents reported engaging more in literacy activities by 0.31 standard deviations (p<0.01) compared to the control group. Effects were driven by children further from average levels of baseline development.

Online credit recovery will likely expand in the coming years as school districts try to address increased course failure rates brought on by the coronavirus pandemic. Some researchers and policymakers, however, raise concerns over how much students learn in online courses, and there is limited evidence about the effectiveness of online credit recovery. This article presents findings from a multisite randomized study, conducted prior to the pandemic, to expand the field's understanding of online credit recovery's effectiveness. Within 24 high schools from a large urban district, the study randomly assigned 1,683 students who failed Algebra 1 or ninth grade English to a summer credit recovery class that either used an online curriculum with in-class teacher support or the school's business-as-usual teacher-directed class. The results suggest that online credit recovery had relatively insignificant effects on student course experiences and content knowledge, but significantly lower credit recovery rates for English. There was limited heterogeneity in effects across students and schools. Non-response on the study-administered student survey and test limit our confidence in the student experience and content knowledge results, but the findings are robust to different approaches to handling the missing data (multiple imputation or listwise deletion). We discuss how the findings add to the evidence base about online credit recovery and the implications for future research. [This paper will be published in "Journal of Research on Educational Effectiveness."]

Although results showing coaching effectiveness are accumulating, coaching is often included with other forms of Professional Development (PD) support including teacher in-service (Powell, Diamond, Burchinal, & Koehler, 2010; Kretlow et al., 2011), access to an annotated video library (Allen et al., 2011), and access to ongoing learning communities (Gallucci, Van Lare, Yoon, & Boatright, 2010; Matsumura, Garnier, & Spybrook, 2012). The presence of multiple intervention components obscures the unique effect of coaching and makes drawing conclusions about coaching effectiveness impossible. There is a critical need for identifying effective, sustainable approaches for teacher PD in delivering instruction to foster science practice skills. The purpose of this research study was to: (1) determine the effects of a professional development intervention comprised of a summer institute and follow-up technology-delivered instructional coaching on teacher and student science practice knowledge, skills, self-efficacy, and engagement; and (2) isolate specific effects of coaching when combined with more traditional teacher workshops. The intervention model described here was designed to equip middle and high school science teachers with knowledge and skills to use a guided scientific inquiry approach to teach science practice skills integrated into content as specified by Next Generation Science Standards (NGSS). The project involved a randomized controlled trial aimed at addressing the following research question: What is the impact of a summer institute focused on guided scientific inquiry with follow-up coaching (treatment) versus no professional development (control) on (a) teacher science practices knowledge, skills, self-efficacy, and beliefs and (b) student science practices knowledge, skills, engagement and self-efficacy? A secondary question involved the independent effects of the summer institute and coaching: What were the separate effects of the summer institute and coaching on teacher and student outcomes? The study was conducted with 124 science teachers (63 treatment and 61 control) from 110 rural schools (61 treatment and 49 control) in Nebraska and Iowa. Results from this study show the promise of coaching and its value added to traditional teacher in-service. While this study shows the promise of coaching in impacting teacher change, more research is needed to investigate what specific aspects of the coaching process (i.e., rapport and trust between teacher and coach, coach qualifications, teacher self-reflection) are most responsible for these effects.

Researchers often include covariates when they analyze the results of randomized controlled trials (RCTs), valuing the increased precision of the estimates over the potential of inducing small-sample bias when doing so. In this paper, we develop a sufficient condition which ensures that the inclusion of covariates does not induce small-sample bias in the effect estimates. Using this result as a building block, we develop a novel approach that uses machine learning techniques to reduce the variance of the average treatment effect estimates while guaranteeing that the effect estimates remain unbiased. The framework also highlights how researchers can use data from outside the study sample to improve the precision of the treatment effect estimate by using the auxiliary data to better model the relationship between the covariates and the outcomes. We conclude with a simulation, which highlights the value of using the proposed approach.

Emergent technologies present platforms for educational researchers to conduct randomized controlled trials (RCTs) and collect rich data on study students' performance, behavior, learning processes, and outcomes in authentic learning environments. As educational research increasingly uses methods and data collection from such platforms, it is necessary to consider the most appropriate ways to analyze this data to draw causal inferences from RCTs. Here, we examine whether and how analysis results are impacted by accounting for multilevel variance in samples from RCTs with student-level randomization. We propose and demonstrate a method that leverages auxiliary non-experimental "remnant" data collected within a learning platform to inform analysis decisions. Specifically, we compare five commonly-applied analysis methods to estimate treatment effects while accounting for, or ignoring, class-level factors and observed measures of confidence and accuracy to identify best practices under real-world conditions. We find that methods that account for groups as either fixed effects or random effects consistently outperform those that ignore group-level factors, even though randomization was applied at the student level. However, we found no meaningful differences between the use of fixed or random effects as a means to account for groups. We conclude that analyses of online experiments should account for the naturally-nested structure of students within classes, despite the notion that student-level randomization may alleviate group-level differences. Further, we demonstrate how to use remnant data to identify appropriate methods for analyzing experiments. These findings provide practical guidelines for researchers conducting RCTs in educational technologies to make more informed decisions when approaching analyses. [This paper will be published in the "Educational Technology Research and Development."]

A randomized control trial was conducted to study whether providing 10th grade students with information about the returns to upper secondary and tertiary education, and a source of financial aid for tertiary education, can contribute to improve student performance. The study finds that the intervention had no effects on the probability of taking a 12th grade national standardized exam three years after, a proxy for on-time high school completion, but a positive and significant impact on learning outcomes and self-reported measures of effort. The effects are larger for girls and students from households with a relatively high income. These findings are consistent with a simple model where time discount determines the increase in effort and only students with adequate initial conditions are able to translate increased effort into better outcomes. [This paper is a product of the Education Global Practice Group.]

National and state educational mandates require students achieve proficiency in not only science content, but also "science inquiry", or those process skills associated with science (National Research Council, 2011; Next Generation Science Standards, 2013). Science inquiry instruction has been shown to improve student achievement and process skills (e.g., Llewellyn, 2002; Schroeder, Scott, Olson, Huang, & Lee, 2007; Minner, Levy, & Century, 2010). Guided scientific inquiry is a student-centered, teacher-facilitated approach to science instruction where students are guided toward, but not directly presented, scientific concepts. Students are led by the teacher to develop the concept (Trowbridge, Bybee, & Powell, 2004). Guided science inquiry has been found effective in promoting student achievement (e.g., Wilson, Taylor, Kowalski, & Carlson, 2010). However, research has not yet specifically examined the value of guided scientific inquiry among students drawn from the special education population. This study sought to answer two research questions. First, how did teachers involved in a study examining implementation of guided science inquiry qualitatively describe their experience implementing guided science inquiry among their students from special education populations? Second, did teachers find guided science inquiry effective to use with students from special education populations? This study sought to answer two research questions. First, how did teachers involved in a study examining implementation of guided science inquiry qualitatively describe their experience implementing guided science inquiry among their students from special education populations? Second, did teachers find guided science inquiry effective to use with students from special education populations? The results suggest that guided science inquiry instruction can be effectively implemented with students from the special education population. The majority of teacher responses suggested that a guided science inquiry approach provides these students with greater understanding, inclusion, and engagement with their science lessons than do traditional instructional methods. Guided science inquiry's flexibility allows for differentiated instruction. Limited concerns with the approach reported for these students included a need to adapt instruction in order to meet their needs and difficulties among students grasping its occasionally abstract structure.

This study examined differences in kindergarten children's behavioral functioning across rural and non-rural communities, as well as explored the effect of risk in each setting (i.e., rural, nonrural) on these children's behavior. Data from 116 kindergarten students and their parents and teachers was analyzed. Multileveling modeling was used to explore the main and interaction effects of community setting and cumulative risk on parent and teacher reports of children's behavioral functioning. Findings suggested that community setting or cumulative risk did not predict kindergarten children's behavior; however, risk differentially predicted children's behavior based on rural or non-rural community status. Particularly, cumulative risk predicted poorer student outcomes upon entrance in kindergarten in rural communities than non-rural communities. Limitations of the study and implications for practice are discussed.

In many experimental evaluations in the social and medical sciences, individuals are randomly assigned to a treatment arm or a control arm of the experiment. After treatment assignment is determined, individuals within one or both experimental arms are frequently grouped together (e.g., within classrooms or schools, through shared case managers, in group therapy sessions, or through shared doctors) to receive services. Consequently, there may be within-group correlations in outcomes resulting from (1) the process that sorts individuals into groups, (2) service provider effects, and/or (3) peer effects. When estimating the standard error of the impact estimate, it may be necessary to account for within-group correlations in outcomes. This article demonstrates that correlations in outcomes arising from nonrandom sorting of individuals into groups leads to bias in the estimated standard error of the impact estimator reported by common estimation approaches. [This paper was supported by funding from the Judith Gueron Fund for Methodological Innovation in Social Policy Research at MDRC.]

This paper considers recent efforts to conduct experimental and quasi-experimental evaluations of career and technical education programs. It focuses on understanding the counterfactual, or control population, for these program evaluations, discussing how the educational experiences of the control population might vary from those of the treated population and the ways in which the treatment and control populations used for evaluation may differ from each other. The paper begins by discussing the key identification strategies and the associated assumptions used to identify program effects, including regression and propensity score matching, instrumental variables, regression discontinuity designs, randomized controlled trials, and lottery-based admissions. It then presents a series of case studies evaluating the counterfactual for specific studies that used each identification strategy.

Randomized controlled trials in lower-income countries have demonstrated ways to increase learning, in specific settings. This study uses a large-scale, nationwide RCT in Ghana to show the external validity of four school-based interventions inspired by other RCTs. Even though the government implemented the programs within existing systems, student learning increased across all four models, more so for female than male students, and many gains persisted one year after the program ended. Three of the four interventions had a similar cost effectiveness. The intervention that directly targeted classroom teachers increased the likelihood that teachers were engaged with students. [Additional funding was provided for this evaluation from the International Growth Centre and the Children's Investment Fund Foundation.]

Power in multilevel models remains an area of interest to both methodologists and substantive researchers. In two-level designs, the total sample is a function of both the number of level-2 (e.g., schools) clusters and the average number of level-1 (e.g., classrooms) units per cluster. Traditional multilevel power calculations rely on either the arithmetic average or the harmonic mean when estimating the average number of level-1 units across clusters of unbalanced size. The current study evaluates and contrasts these two approaches with simulation-based power estimates in two-group two-level cluster randomized controlled trial designs with unbalanced cluster sizes. Results from the Monte Carlo study demonstrated the largest differences in simulated versus the two forms of calculated power occurred in study designs with large variability in the number of level-1 units sampled. Overall, power was less sensitive to the level-2 sample size or the effect size, regardless of the imbalance in cluster size. Implications of these findings for the design of cluster randomized trials are discussed. [This paper will be published in "The Journal of Experimental Education."]

Randomized A/B tests in educational software are not run in a vacuum: often, reams of historical data are available alongside the data from a randomized trial. This paper proposes a method to use this historical data--often highdimensional and longitudinal--to improve causal estimates from A/B tests. The method proceeds in two steps: first, fit a machine learning model to the historical data predicting students' outcomes as a function of their covariates. Then, use that model to predict the outcomes of the randomized students in the A/B test. Finally, use design-based methods to estimate the treatment effect in the A/B test, using prediction errors in place of outcomes. This method retains all of the advantages of design-based inference, while, under certain conditions, yielding more precise estimators. This paper will give a theoretical condition under which the method improves statistical precision, and demonstrates it using a deep learning algorithm to help estimate effects in a set of experiments run inside ASSISTments. [For the full proceedings, see ED593090.]

In cluster randomized evaluations, a treatment or intervention is randomly assigned to a set of clusters each with constituent individual units of observations (e.g., student units that attend schools, which are assigned to treatment). One consideration of these designs is how many units are needed per cluster to achieve adequate statistical power. Typically, researchers state that "about 30 units per cluster" is the most that will yield benefit towards statistical precision. To avoid rules of thumb not grounded in statistical theory and practical considerations, and instead provide guidance for this question, the ratio of the minimum detectable effect size (MDES) to the larger MDES with one less unit per cluster is related to the key parameters of the cluster randomized design. Formulas for this subsequent difference effect size ratio (SDESR) at a given number of units are provided, as are formulas for finding the number of units for an assumed SDESR. In general, the point of diminishing returns occurs with smaller numbers of units for larger values of the intraclass correlation. [This paper will be published in "American Journal of Evaluation."]

Studies of small-scale "model" early-childhood programs show that high-quality preschool can have transformative effects on human capital and economic outcomes. Evidence on the Head Start program is more mixed. Inputs and practices vary widely across Head Start centers, however, and little is known about variation in effectiveness within Head Start. This paper uses data from a multi-site randomized evaluation to quantify and explain variation in effectiveness across Head Start childcare centers. I answer two questions: (1) How much do short-run effects vary across Head Start centers? and (2) To what extent do inputs, practices, and child characteristics explain this variation? To answer the first question, I use a selection model with random coefficients to quantify heterogeneity in Head Start effects, accounting for non-compliance with experimental assignments. Estimates of the model show that the cross-center standard deviation of cognitive effects is 0.18 test score standard deviations, which is larger than typical estimates of variation in teacher or school effectiveness. Next, I assess the role of observed inputs, practices and child characteristics in generating this variation, focusing on inputs commonly cited as central to the success of model programs. My results show that Head Start centers offering full-day service boost cognitive skills more than other centers, while Head Start centers offering frequent home visiting are especially effective at raising non-cognitive skills. Head Start is also more effective for children with less-educated mothers. Centers that draw more children from center-based preschool have smaller effects, suggesting that cross-center differences in effects may be partially due to differences in counterfactual preschool options. Other key inputs, including the High/Scope curriculum, teacher education, and class size, are not associated with increased effectiveness in Head Start. Together, observed inputs explain about one-third of the variation in Head Start effectiveness across experimental sites.

We present the results of three large-scale randomized controlled trials (RCTs) carried out in Chicago, testing interventions to reduce crime and dropout by changing the decision-making of economically disadvantaged youth. We study a program called Becoming a Man (BAM), developed by the non-profit Youth Guidance, in two RCTs implemented in 2009-10 and 2013-15. In the two studies participation in the program reduced total arrests during the intervention period by 28-35%, reduced violent-crime arrests by 45-50%, improved school engagement, and in the first study where we have follow-up data, increased graduation rates by 12-19%. The third RCT tested a program with partially overlapping components carried out in the Cook County Juvenile Temporary Detention Center (JTDC), which reduced readmission rates to the facility by 21%. These large behavioral responses combined with modest program costs imply benefit-cost ratios for these interventions from 5-to-1 up to 30-to-1 or more. Our data on mechanisms are not ideal, but we find no positive evidence that these effects are due to changes in emotional intelligence or social skills, self-control or "grit," or a generic mentoring effect. We find suggestive support for the hypothesis that the programs work by helping youth slow down and reflect on whether their automatic thoughts and behaviors are well suited to the situation they are in, or whether the situation could be construed differently. [A full list of sponsors of this project can be found on the NBER web site: http://www.nber.org/papers/w21178.ack.]

A prior cluster randomized controlled trial (RCT) compared outcomes for a comprehensive school intervention (schoolMAX) to typical educational programming (services-as-usual [SAU]) for 103 children with autism spectrum disorder (ASD) without intellectual disability. The schoolMAX intervention was superior to SAU in improving social-cognitive understanding (emotion-recognition), social/social-communication skills, and ASD-related impairment (symptoms). In the current study, a range of demographic, clinical, and school variables were tested as potential moderators of treatment outcomes from the prior RCT. Moderation effects were not evident in demographics, child IQ, language, or ASD diagnostic symptoms, or school SES. Baseline externalizing symptoms moderated the outcome of social-cognitive understanding and adaptive skills moderated the outcome of ASD-related symptoms (no other comorbid symptoms or adaptive skills ratings moderated outcomes on the three measures). Overall, findings suggest that the main effects of treatment were, with two exceptions, unaffected by third variables. [This paper was published in "Journal of Abnormal Child Psychology" v48 p1105-1114 May 2020.]

Introduction: Youth in disadvantaged urban areas are frequently exposed to chronic stress and trauma, including housing instability, neighborhood violence, and other poverty-related adversities. These exposures increase risk for emotional, behavioral, and academic problems and ultimately, school dropout. Schools are a promising setting in which to address these issues; however, there are few universal, trauma-informed school-based interventions for urban youth. Methods/Design: Project POWER (Promoting Options for Wellness and Emotion Regulation) is a randomized controlled trial testing the impact of RAP Club, a trauma-informed intervention for 8th graders that includes mindfulness as a core component. Students in 32 urban public schools (n = 800) are randomly assigned to either RAP Club or a health education active control group. We assess student emotional, behavioral, and academic outcomes using self-report surveys and teacher ratings at baseline, post-intervention, and 4-month follow up. Focus groups and interviews with students, teachers, and principals address program feasibility, acceptability, and fidelity, as well as perceived program impacts. Students complete an additional self-report survey in 9th grade. Schools provide students' academic and disciplinary data for their 7th, 8th, and 9th grade years. In addition, data on program costs are collected to conduct an economic analysis of the intervention and active control programs. Discussion: Notable study features include program co-leadership by young adults from the community and building capacity of school personnel for continued program delivery. In addition to testing program impact, we will identify factors related to successful program implementation to inform future program use and dissemination. [This paper was published in "Contemporary Clinical Trials" Mar 2020.]