Search

We provide evidence about college financial aid from an eight-year randomized trial where high school ninth graders received a $12,000 merit-based grant offer. The program was designed to be free of tuition/fees at community colleges and substantially lower the cost of four-year colleges. During high school, it increased students' college expectations and low-cost effort, but not higher-cost effort, such as class attendance. The program likely increased two-year college graduation, perhaps because of the free college framing, but did not affect overall college entry, graduation, employment, incarceration, or teen pregnancy. Additional analysis helps explain these modest effects and variation in results across prior studies.

Online credit recovery will likely expand in the coming years as school districts try to address increased course failure rates brought on by the coronavirus pandemic. Some researchers and policymakers, however, raise concerns over how much students learn in online courses, and there is limited evidence about the effectiveness of online credit recovery. This article presents findings from a multisite randomized study, conducted prior to the pandemic, to expand the field's understanding of online credit recovery's effectiveness. Within 24 high schools from a large urban district, the study randomly assigned 1,683 students who failed Algebra 1 or ninth grade English to a summer credit recovery class that either used an online curriculum with in-class teacher support or the school's business-as-usual teacher-directed class. The results suggest that online credit recovery had relatively insignificant effects on student course experiences and content knowledge, but significantly lower credit recovery rates for English. There was limited heterogeneity in effects across students and schools. Non-response on the study-administered student survey and test limit our confidence in the student experience and content knowledge results, but the findings are robust to different approaches to handling the missing data (multiple imputation or listwise deletion). We discuss how the findings add to the evidence base about online credit recovery and the implications for future research. [This paper will be published in "Journal of Research on Educational Effectiveness."]

In 2015, Abt Associates received a grant from the Institutes for Education Sciences (IES) for a five-year "Development and Innovation" study of PTC. The purposes of the study were to gauge progress in implementing PTC and to develop and test improvements where needed. Fein et al. (2020) summarize the IES study's approach and findings. A subsequent grant from Arnold Ventures provided support for extending the two analyses--to three follow-up years for Study 1 and to four years for Study 2. This report provides findings from these longer-term analyses. Study 1 found no difference in average earnings or months enrolled in college in follow-up Years 2 and 3 between young adults assigned to PTC and their control group counterparts. (As expected, the PTC group earned less and spent more time in college than the control group in Year 1, when participants were still in the program.) The results also show modest increases in receipt of credentials (mostly short-term certificates based on credit earned at partner college during PTC).

Narration has been shown to be a foundational skill for literacy development in school-age children. Elementary teachers routinely conduct classroom lessons that focus on reading decoding and comprehension, but they rarely provide instruction in oral narration (Hall et al., 2021). This multisite randomized controlled trial was designed to rigorously evaluate the efficacy of the "Supporting Knowledge of Language and Literacy" ("SKILL") intervention program for improving oral narrative comprehension and production. Three hundred fifty-seven students who were at-risk for language and literacy difficulties in Grades 1-4 in 13 schools across seven school districts were randomly assigned to the "SKILL" treatment condition or a business as usual (BAU) control condition. "SKILL" was provided to small groups of two to four students in 36 thirty-minute lessons across a 3-month period. Multilevel modeling with students nested within teachers and teachers nested within schools revealed students who received the "SKILL" treatment significantly outperformed students in the BAU condition on measures of oral narrative comprehension and production immediately after treatment. Oral narrative production for the "SKILL" treatment group remained significantly more advanced at follow-up testing conducted 5 months after intervention ended. Improvements in oral narration generalized to a measure of written narration at posttest and the treatment advantage was maintained at follow-up. Grade level did not moderate effects for oral narration, but it did for reading comprehension, with a higher impact for students in grades 3 and 4. [This is the online version of an article published in "Journal of Educational Psychology."]

Student-teacher relationships are important to student outcomes and may be especially pivotal at the high school transition and for minoritized racial/ethnic groups. Although interventions exist to improve student-teacher relationships, none have been shown to be effective among high school students or in narrowing racial/ethnic disparities in student outcomes. This study was conducted to examine the effects of an equity-explicit student-teacher relationship intervention (Equity-Explicit Establish Maintain Restore, or E-EMR) for high school teachers and students. A cluster-randomized pilot trial was conducted with 94 ninth grade teachers and 417 ninth grade students in six high schools. Teachers in three schools were randomized to receive E-EMR training and follow-up supports for one year. Teachers in three control schools conducted business as usual. Student-teacher relationships, sense of school belonging, academic motivation, and academic engagement were collected via student self-report in September and January of their ninth-grade year. Longitudinal models revealed non-significant main effects of E-EMR. However, there were targeted benefits for students who started with low scores at baseline, for Asian, Latinx, multicultural, and (to a lesser extent) Black students. We also found some unexpected effects, where high-performing and/or advantaged groups in the E-EMR condition had less favorable outcomes at post, compared to those in the control group, which may be a result of the equity-explicit focus of E-EMR. Implications and directions for future research are discussed. [This is the online first version of an article published in "School Mental Health."]

At every stage since the start of the pandemic, decisions across education and childcare have been informed by the scientific and medical evidence -- both on the risks of coronavirus (COVID-19) infection, transmission and illness, and on the known risks to children and young people not attending education settings -- balancing public health and education considerations. This report sets out the evidence informing the Government's decision to revisit the guidance on the use of face coverings within secondary schools and colleges in England -- temporarily extending their recommended use in communal areas to also include classrooms and teaching spaces for those in year 7 and above. In making this decision, the Government has balanced education and public health considerations, including the benefits in managing infection and transmission, against any educational and wider health and wellbeing impacts from the recommended use of face coverings. Evidence shows that face coverings can contribute to reducing transmission of COVID19 primarily by reducing the emission of virus-carrying particles when worn by an infected person. The government will continue to evaluate the data relating to all COVID-19 measures, including in education settings.

This document summarises key findings from the quantitative strands of a review of the Education Endowment Foundation (EEF) evaluations that had reported from the establishment of EEF in 2011 up to January 2019. The quantitative strands summarised include meta-analyses of effect sizes reported for attainment outcomes and descriptive analyses of cost-effectiveness and attrition. Additionally, descriptive univariate analyses of the explanatory variables used in the review are summarised. Complete findings can be found in the main report. The review conducted qualitative interviews to explore perceptions on effective scale-up and developed and piloted an IPE quality measure, and these are reported separately. [For "Review of EEF Projects. Evaluation Report," see ED625480. For "Review of EEF Projects. Technical Annex," see ED625484. For "Review: Scale-Up of EEF Efficacy Trials to Effectiveness Trials," see ED625479. For "Review: EEF Implementation and Process Evaluation (IPE) Quality Pilot," see ED625478.]

This report presents findings from exploratory, descriptive meta-analyses of effect sizes reported by the first 82 EEF evaluations that used a randomised controlled trial (RCT) or clustered RCT impact evaluation design published up to January 2019. The review used a theoretical framework derived from literature with five overarching themes to group explanatory variables: (1) Intervention; (2) Implementation & fidelity; (3) Theory & evidence; (4) Context; and (5) Evaluation design. Meta-analyses of effect sizes reported for intention-to-treat (ITT) analyses of primary attainment outcomes, ITT analyses of secondary attainment outcomes and free school meals (FSM) subsample analyses of primary or secondary attainment outcomes are reported. Effect sizes reported for psychological outcomes were also examined but not included in the meta-analyses because of the distinct and diverse nature of these outcomes. Also presented are findings from trial/evaluation-level descriptive analyses of cost effectiveness of interventions and overall pupil-level attrition. [For "Review of EEF Projects. Summary of Key Findings," see ED625481. For "Review of EEF Projects. Technical Annex," see ED625484. For "Review: Scale-Up of EEF Efficacy Trials to Effectiveness Trials," see ED625479. For "Review: EEF Implementation and Process Evaluation (IPE) Quality Pilot," see ED625478.]

This case study describes a small randomized controlled trial (RCT) comparing alternative strategies for monitoring and supporting academic achievement in Year Up's Professional Training Corps (PTC) program. Year Up is a nonprofit organization dedicated to preparing economically disadvantaged young adults for well-paying jobs with advancement potential. As part of the study, Year Up and its research partners designed and tested strategies for more quickly identifying and supporting participants struggling with their college coursework. To ensure the credibility of the study findings, the team randomly assigned participants either to a coach who would use the new monitoring and support strategies or to a coach who would follow existing practices, which did not place much emphasis on academics. Over two cycles of testing, the study found strong evidence that the alternative strategies tested substantially improved success in courses and, thus, advancement to internships. Three factors contributed to the resulting evidence being actionable: (1) it focused on low- to no-cost, field-initiated practice changes that could be implemented widely; (2) site staff could tailor strategies to local needs and opportunities; and (3) the research team encouraged local staff to modify the strategies being tested between enrollment cohorts. [This case study was commissioned by the Actionable Evidence Initiative and led by Project Evident. Funding from the GreenLight Fund also supported this work.]

This study investigates the impact of Education Endowment Foundation (EEF)-funded trials on pupils eligible for free school meals. Although similar analysis is conducted during each individual evaluation, this report conducts a meta-analysis using data from 88 trials and over half a million pupils to reach conclusions. The report contributes to the evidence about what type of approaches may be effective at reducing the attainment gap, which influences the choice of interventions that EEF chooses to trial and scale up. This study investigates the immediate impact of EEF-funded interventions on attainment in literacy and mathematics based on the following research questions: (1) Do EEF trials improve literacy/mathematics attainment of pupils eligible for FSM? (2) What broad types of interventions are more beneficial for mathematics and literacy outcomes of FSM pupils? and (3) Do FSM pupils improve their literacy/mathematics attainment more or less from EEF-funded interventions compared to their non-FSM peers? [This report was written in partnership with the Durham Research Methods Centre (DRMC).]

Some education programs' early positive effects disappear over time. Other programs have unanticipated positive long-term effects. Foundations warn of the dangers of putting too much weight on in-program effects, which, they say, often fade after a program ends. This Issue Focus tackles the topic of post-program effects in postsecondary education. Are in-program effects--that is, the effects observed while the program was active--maintained once the program ends? Do they grow and improve? Or do they fade out? This investigation capitalizes on two decades of rigorous program evaluations conducted by MDRC, including approximately 25 postsecondary programs, to consider those questions. The Higher Education Randomized Controlled Trials (THE RCT) draw from a database of student-level data from 31 MDRC projects, sampling 67,400 stu­dents. The results of THE RCT are striking: During the year after these programs ended, effects on academic progress were consistently maintained. There is no evidence of discernible fade-out, which provides encouraging information about the lasting value of many postsecondary programs and the value of evidence collected from them. There was also no evidence of post-program growth, perhaps prompting the need to develop programs that equip students for success over the long term.

Most community college students are referred to developmental education courses to build basic skills. These students often struggle in these courses and college more broadly. CUNY Start is a prematriculation program for students assessed as having significant remedial needs. CUNY Start students delay matriculation for one semester and receive time-intensive instruction in math, reading, and writing with a prescribed pedagogy delivered by trained teachers. The program aims to help students complete remediation and prepare for college-level courses. This article describes the results of an experiment at four community colleges (n [is approximately equal to] 3,800). We estimate that over three years, including one semester that students spent in the program and two-and-a-half years after the program was complete, CUNY Start substantially increased college readiness, slightly increased credit accumulation, and modestly increased graduation rates (by increasing participation in CUNY's highly effective ASAP).

Power in multilevel models remains an area of interest to both methodologists and substantive researchers. In two-level designs, the total sample is a function of both the number of level-2 (e.g., schools) clusters and the average number of level-1 (e.g., classrooms) units per cluster. Traditional multilevel power calculations rely on either the arithmetic average or the harmonic mean when estimating the average number of level-1 units across clusters of unbalanced size. The current study evaluates and contrasts these two approaches with simulation-based power estimates in two-group two-level cluster randomized controlled trial designs with unbalanced cluster sizes. Results from the Monte Carlo study demonstrated the largest differences in simulated versus the two forms of calculated power occurred in study designs with large variability in the number of level-1 units sampled. Overall, power was less sensitive to the level-2 sample size or the effect size, regardless of the imbalance in cluster size. Implications of these findings for the design of cluster randomized trials are discussed. [This paper will be published in "The Journal of Experimental Education."]

Objective: The current study reports functional outcomes from a multi-site randomized trial of a cognitive-behavioral treatment program for college students diagnosed with ADHD. Methods: A sample of emerging adults (N = 250; ages 18 to 30) currently attending college were comprehensively evaluated and diagnosed with ADHD (M age = 19.7; 66% female, 6.8% Latino, 66.3% Caucasian). Participants were randomized to either a two-semester intervention (Accessing Campus Connections and Empowering Student Success (ACCESS)) or a delayed treatment condition. Participants were assessed with measures of academic, daily life, and relationship functioning prior to treatment, at the end of the first semester, and after the second semester of treatment. Results: Multi-group latent growth curve models revealed moderate effect size improvements on self-report measures of study skills and strategies, as well as on self-report measures of time management, daily functioning, and overall well-being for participants in ACCESS. Importantly, treatment effects were maintained or increased in some cases from the end of the first semester to the end of the second semester. Improvements in self-reported interpersonal functioning were not significantly different across condition and neither condition demonstrated significant change over time in educational record outcomes (GPA and number of credits earned). Conclusions: ACCESS appears to promote improvements in self-reported general well-being and functioning, time management, and study skills and strategies. However, improvements in interpersonal relationships and objective academic outcomes such as GPA were not observed. Clinical implications and future directions for treating ADHD on university and college campuses are discussed. [This is the online version of an article published in "Journal of Clinical Child & Adolescent Psychology."]

This paper considers recent efforts to conduct experimental and quasi-experimental evaluations of career and technical education programs. It focuses on understanding the counterfactual, or control population, for these program evaluations, discussing how the educational experiences of the control population might vary from those of the treated population and the ways in which the treatment and control populations used for evaluation may differ from each other. The paper begins by discussing the key identification strategies and the associated assumptions used to identify program effects, including regression and propensity score matching, instrumental variables, regression discontinuity designs, randomized controlled trials, and lottery-based admissions. It then presents a series of case studies evaluating the counterfactual for specific studies that used each identification strategy.

Developed by the City University of New York (CUNY), the Accelerated Study in Associate Programs (ASAP) is a comprehensive program that provides students with up to three years of financial and academic support and other services. In return, students are expected to enroll in classes full time and participate in essential program services. An earlier experimental evaluation found that ASAP nearly doubled graduation rates three years after students entered the study. This paper summarizes the effects on education and financial aid outcomes over an eight-year follow-up period, or five years after the end of program services. This paper also presents the educational investment in students associated with providing these services and any educational investment in students from the CUNY community colleges after the program services were no longer provided. Through the eight-year follow-up period, ASAP's effects on associate's degree receipt persisted, indicating that the program not only helped students graduate faster, but also helped some students who would have never graduated without the program earn a degree. ASAP also led to a significant increase in the amount of financial aid students received through federal and state grants. The net cost (the difference between the total program group costs and the total control group costs) averaged $13,838 per program group member over eight years. CUNY spent, on average, an additional $9,162 per degree earned for students offered ASAP, compared with the control group students over the eight-year follow-up period. The findings also show that the program resulted in increased financial aid from the Pell Grant Program and the New York State Tuition Assistance Program, and reduced the grant dollars these programs invested per degree earned.

A prior cluster randomized controlled trial (RCT) compared outcomes for a comprehensive school intervention (schoolMAX) to typical educational programming (services-as-usual [SAU]) for 103 children with autism spectrum disorder (ASD) without intellectual disability. The schoolMAX intervention was superior to SAU in improving social-cognitive understanding (emotion-recognition), social/social-communication skills, and ASD-related impairment (symptoms). In the current study, a range of demographic, clinical, and school variables were tested as potential moderators of treatment outcomes from the prior RCT. Moderation effects were not evident in demographics, child IQ, language, or ASD diagnostic symptoms, or school SES. Baseline externalizing symptoms moderated the outcome of social-cognitive understanding and adaptive skills moderated the outcome of ASD-related symptoms (no other comorbid symptoms or adaptive skills ratings moderated outcomes on the three measures). Overall, findings suggest that the main effects of treatment were, with two exceptions, unaffected by third variables. [This paper was published in "Journal of Abnormal Child Psychology" v48 p1105-1114 May 2020.]

Policymakers, experts and advocates have promoted many different types of education reform over the past few decades, but what is the evidence about the efficacy of these programs? EdChoice partnered with Hanover Research to find out what research has been conducted in nine major education reform areas focusing on outcomes related to student achievement or education attainment. The best methodology available to researchers for generating "apples-to-apples" comparisons is a randomized control trial (RCT), which researchers also refer to as random assignment studies or experimental studies. For this summary only RCTs were reviewed--not studies using other methods--because experiments comparing treatment and control groups allow researchers to identify reform/policy/program effects while minimizing bias from unobservable factors. The brief shares the assessment of the RCT studies only examining two types of outcomes, either participating students' achievement or measures of educational attainment. [Reviews conducted by Hanover Research Editing and coding by EdChoice.]

Introduction: Youth in disadvantaged urban areas are frequently exposed to chronic stress and trauma, including housing instability, neighborhood violence, and other poverty-related adversities. These exposures increase risk for emotional, behavioral, and academic problems and ultimately, school dropout. Schools are a promising setting in which to address these issues; however, there are few universal, trauma-informed school-based interventions for urban youth. Methods/Design: Project POWER (Promoting Options for Wellness and Emotion Regulation) is a randomized controlled trial testing the impact of RAP Club, a trauma-informed intervention for 8th graders that includes mindfulness as a core component. Students in 32 urban public schools (n = 800) are randomly assigned to either RAP Club or a health education active control group. We assess student emotional, behavioral, and academic outcomes using self-report surveys and teacher ratings at baseline, post-intervention, and 4-month follow up. Focus groups and interviews with students, teachers, and principals address program feasibility, acceptability, and fidelity, as well as perceived program impacts. Students complete an additional self-report survey in 9th grade. Schools provide students' academic and disciplinary data for their 7th, 8th, and 9th grade years. In addition, data on program costs are collected to conduct an economic analysis of the intervention and active control programs. Discussion: Notable study features include program co-leadership by young adults from the community and building capacity of school personnel for continued program delivery. In addition to testing program impact, we will identify factors related to successful program implementation to inform future program use and dissemination. [This paper was published in "Contemporary Clinical Trials" Mar 2020.]

This article assesses the likely generalizability of educational treatment-effect estimates from regression discontinuity designs (RDDs) when treatment assignment is based on academic pretest scores. Our assessment uses data on outcome and pretest measures from six educational experiments, ranging from preschool through high school, to estimate RDD generalization bias. We then compare those estimates (reported as standardized effect sizes) with the What Works Clearinghouse (WWC) standard for acceptable bias size ([less than or equal] to 0.05[delta]) for two target populations, one spanning a half-standard deviation pretest-score range and another spanning a full-standard deviation pretest-score range. Our results meet this standard for all 18 study/outcome/pretest scenarios examined given the narrower target population, and for 15 scenarios given the broader target population. Fortunately, two of the three exceptions represent pronounced "ceiling effects" that can be identified empirically, making it possible to avoid unwarranted RDD generalizations, and the third exception is very close to the WWC standard. [This is the online version of an article published in "Journal of Research on Educational Effectiveness" (ISSN 1934-5747).]

A growing body of research documents the effectiveness of classroom management programs on a range of student outcomes, yet few early-career teachers receive training on these practices prior to entering the classroom. Moreover, few studies have attended to how variations in teacher distress or level of classroom misbehavior affects training benefits. This study reports findings from a randomized trial of a teacher training program that combined two evidence-based programs (Good Behavior Game [GBG] and MyTeachingPartnerTM [MTP]) to determine their impact on novice teachers and their students. In addition, the current study reports findings on moderated impacts by initial teacher distress as well as the overall classroom level of misbehavior. The sample included 188 early-career teachers (grades K-3) in their first three years of teaching from three large, urban school districts. Analyses indicated that intervention had no main effects, but yielded moderated impact depending on the combination of the baseline levels of classroom disruptive behavior and teacher distress; it appears that the program impacts were greatest in the highest risk circumstance (i.e., high teacher stress and elevated challenging student behaviors). For those classrooms, those assigned to intervention evidence improved behavior and student achievement compared to control counterparts by the spring of the training year, relative to the fall baseline (d = 0.18-0.70 depending on outcome). This study is significant in that it highlights effects during a critical window of training and coaching for early career teachers and the need to consider teacher and classroom contextual factors that may moderate professional development efforts. [This paper was published in "Journal of School Psychology" v78 p75-95 2020.]

Many public and private initiatives have focused on improving prekindergarten (PK) through Grade 12 mathematics education, given that the mathematics content learned before college is foundational for later learning and college success across STEM subjects. Although decades of educational research have studied PK-12 mathematics interventions, the field lacks a comprehensive understanding of which interventions work, for whom, and under what conditions. This meta-analytic project aimed to examine the heterogeneity in mathematics intervention effects by synthesizing 25 years of randomized experiments of interventions designed to improve mathematics achievement. It describes intersections of interventions, intervention components, and outcomes that appear especially promising for future intervention research. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Disruptive behavior problems frequently emerge in the preschool years and are associated with numerous, long-term negative outcomes, including comorbid disorders. First Step is a psychosocial early intervention with substantial empirical evidence supporting its efficacy among young children (Walker et al., 2014). The present study reports on a validation study of the revised and updated First Step early intervention, called First Step Next (Walker, Stiller et al. 2015), conducted within four preschool settings. One hundred sixty students at risk for school failure, and their teachers, were randomized to intervention and control conditions. Results indicated coach and teacher adherence to implementing the core components of the program was excellent. Teachers and parents had high satisfaction ratings. For the three First Step Next pro-social domains, Hedges' g effect sizes ranged from 0.34 to 0.91. For the problem behavior domain, children who received the First Step Next intervention had significant reductions in teacher and parent-reported problem behavior as compared to children randomized to the control condition. For the problem behavior domain, Hedges' g effect sizes ranged from 0.33 to 0.63, again favoring the intervention condition. All of the domains were statistically significant. This study builds on the evidence base supporting the First Step intervention in preschool settings (Feil et al., 2014; 2016; Frey et al., 2015). [This paper will be published in "Exceptional Children."]