Search

In this randomized control study, we evaluated science teachers' pedagogical practices via classroom observations following ongoing, intensive, and structured instructional support sessions. These sessions included virtual professional development (VPD) and virtual mentoring and coaching (VMC) that accompanied a literacy-infused science curriculum. Using a low-inference observational instrument, we explored the direct impact of VPD and VMC on fifth-grade science teachers' observed time allocation in a randomized controlled trial (RCT) validation study. The observations were collected three times during science instruction from 121 teachers in 68 schools from 35 public school districts in the U.S. state of Texas, during the 2017-2018 school year. Preliminary findings revealed pedagogical differences in time allocation among teachers between treatment and control classrooms. We identified improved instructional practices within treatment classrooms, which suggests the intervention had a positive effect by enhancing the quality of pedagogy as well as the content-area instruction in science.

This review aims to systematically summarize existing evidence to determine the effectiveness of cognitive-based interventions (CBIs) on psychological health and well-being among parents of children with developmental disabilities (DD). Six databases were searched to identify eligible randomized controlled trials (RCTs) from their inception to April 2023. The revised Cochrane Risk of Bias tool for RCTs was applied to assess the risk of bias and the certainty of evidence was evaluated using the Grading of Recommendation, Assessment, Development and Evaluation. Meta-analyses were conducted using a random-effects model. Twenty-five RCTs involving 1915 participants were identified. The results indicated that CBIs reduced parental stress levels (Hedges' g = - 0.69), depressive symptoms (g = - 0.95), anxiety levels (g = - 0.78), and parental distress (g = - 0.29), and improved parental well-being (g = 0.62) and parent-child relationships (g = 0.43) postintervention compared with the active/inactive control groups. Subgroup analysis of the effectiveness of interventions using mindfulness-based interventions and cognitive behavioural therapy showed positive effects. The favourable intervention duration and participant targets were also identified in this review. Furthermore, the effects of CBIs were impacted by the different types of DD among the children. This review highlighted the positive effects of CBIs on parental stress levels, depressive symptoms, anxiety levels, parental distress levels, parental well-being levels, and parent-child relationships. Future well-designed RCTs are needed to further investigate the effects of MBIs and CBT interventions on children with DD and their parents, as well as the factors and mechanisms of action affecting the efficacy of these interventions.

Objective: To determine if treatment of co-occurring adult ADHD and Cannabis Use Disorder (CUD) with extended-release mixed amphetamine salts (MAS-ER) would be effective at improving ADHD symptoms and promoting abstinence. Method: A 12-week randomized, double-blind, two-arm pilot feasibility trial of adults with comorbid ADHD and CUD (n = 28) comparing MAS-ER (80 mg) to placebo. Main outcomes: ADHD: [greater than or equal to] 30% symptom reduction, measured by the Adult ADHD Investigator Symptom Rating Scale (AISRS). CUD: Abstinence during last 2 observed weeks of maintenance phase. Results: Overall, medication was well-tolerated. There was no significant difference in ADHD symptom reduction (MAS-ER: 83.3%; placebo: 71.4%; p = 0.65) or cannabis abstinence (MAS-ER: 15.4%; placebo: 0%; p = 0.27). MAS-ER group showed a significant decrease in weekly cannabis use days over time compared to placebo (p < 0.0001). Conclusions: MAS-ER was generally well-tolerated. The small sample size precluded a determination of MAS-ER's superiority reducing ADHD symptoms or promoting abstinence. Notably, MAS-ER significantly reduced weekly days of use over time.

Cluster randomized trials (CRTs) are commonly used to evaluate the causal effects of educational interventions, where the entire clusters (e.g., schools) are randomly assigned to treatment or control conditions. This study introduces statistical methods for designing and analyzing two-level (e.g., students nested within schools) and three-level (e.g., students nested within classrooms nested within schools) CRTs. Specifically, we utilize hierarchical linear models (HLMs) to account for the dependency of the intervention participants within the same clusters, estimating the average treatment effects (ATEs) of educational interventions and other effects of interest (e.g., moderator and mediator effects). We demonstrate methods and tools for sample size planning and statistical power analysis. Additionally, we discuss common challenges and potential solutions in the design and analysis phases, including the effects of omitting one level of clustering, non-compliance, heterogeneous variance, blocking, threats to external validity, and cost-effectiveness of the intervention. We conclude with some practical suggestions for CRT design and analysis, along with recommendations for further readings.

Background: Physical activity (PA) may benefit academic performance, but it is unclear what kind of classroom-based PA is optimal for learning. Aim: We studied the effects of physically active maths lessons on children's maths performance and maths-related effects, and whether gender and previous mathematical or motor skills modify these effects. Sample: A total of 22 volunteered teachers and their pupils with signed consent (N = 397, mean age: 9.3 years, 51% females) participated in a 5-month, teacher-led, multi-arm, cluster-randomized controlled trial. Methods: The intervention included a PAL group (20 min of physically active learning in each 45-min lesson), a breaks group (two 5-min PA breaks in each 45-min lesson) and a control group (traditional teaching). Maths performance was assessed with a tailored curriculum-based test. Maths-related enjoyment, self-perceptions and anxiety were measured with a self-reported questionnaire. The individual-level intervention effects were tested via covariate-adjusted linear mixed-effect models with school classes serving as random effects. Results: Changes in maths performance or self-perceptions did not differ between the intervention groups. Maths anxiety in learning situations increased in the PAL group (effect .28, 95% CI = .01--.56); there was no change in the other groups. Subgroup analyses suggested that maths anxiety increased in the PAL group among children in the two lowest tertiles of motor skills. It decreased in the highest tertile. Enjoyment decreased in the breaks group among pupils in the lowest motor skill tertile. Conclusions: Physically active maths lessons did not affect maths performance or self-perceptions but had divergent effects on maths anxiety and enjoyment, depending on motor skills.

This article examines the implementation, participation rates, and potential determinants of participation in the digital addiction prevention program "ready4life." A two-arm cluster-randomized trial recruited German vocational students via class-based strategies. Intervention group received 16 weeks of in-app coaching; the control group received health behavior information, with coaching offered after 12 months. Potential determinants of participation were analyzed based on class and individual characteristics. Out of 525 contacted schools, 35 participated, enrolling 376 classes. Implementation during the pandemic required flexible adjustments, with 49.7% of introductions conducted in person, 43.1% digitally via online streaming, and 7.2% received a video link via email. Despite challenges, 72.3% of the vocational students downloaded the app, and 46.7% gave informed consent. Participation rates were highest among (associate) professionals, vocational grammar school classes, classes introduced by females, younger individuals, members of the project team, and classes introduced face-to-face. Female gender, lower social competencies, lifetime cannabis use, higher problematic internet use, and higher perceived stress were associated with higher individual participation. The study highlights the importance of proactive outreach and personalized interventions for addiction prevention programs in vocational schools. While reached students aligned with the aims of the app, tailored recruitment strategies could enhance engagement among under-represented groups.

Violence is a major public health problem globally, with the highest rates in low- and middle-income countries (LMICs) in the Americas and southern Africa. Parenting programmes in high-income countries can diminish risk for violence, by reducing risk factors such as child aggression and harsh parenting, and increasing protective factors such as child cognitive development and school readiness. However, there is critical need to identify low-cost programmes with replicable benefits that work in real-world LMICs contexts. A three-arm, randomised, single-blind trial evaluated effects of two low-cost, group-based parenting programmes recommended for LMICs (ACT: Raising Safe Kids; DBS: dialogic book-sharing) on child aggression (primary outcome), child development, parenting, maltreatment, and stress. Participants were 369 children with medium-high levels of aggression (mean age 3.1 years at baseline) in poor households. Interventions were implemented in city health and education services in southern Brazil. Maternal reports, filmed observations, child tasks, and hair cortisol were assessed at baseline, 1-month post-intervention, and 8-month follow-up. Intention-to-treat analyses compared each of ACT and DBS with a control group. Three hundred sixty-eight (99.7%) participants completed follow-up assessments 8 months after the interventions. There was no effect of ACT (standardised mean difference, SMD 0.11, 95% CI - 0.05, 0.27) or DBS (SMD 0.05, 95% CI - 0.11, 0.21) on the primary outcome of child aggression. ACT reduced harsh parenting behaviour post-intervention (SMD - 0.23; 95% CI - 0.46, 0.01), but not at follow-up. DBS improved book-sharing practices at both time points (e.g., maternal sensitivity at follow-up SMD 0.33; 95% CI 0.08, 0.57). There were no benefits of either programme for other parenting, child development, or stress outcomes. Two parenting programmes in Brazil had small effects on parenting practices but did not reduce child aggression or several other important risk/protective factors for violence. Effective early interventions that reduce violence in real-world LMIC settings are highly desirable but may be challenging to achieve.

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing--remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

Discussions of randomised controlled trials (RCTs) in education that do not show an impact regularly focus on the intervention and how it failed to impact on expected measures, with typologies identifying persistent critical points of failure. This paper uses one such RCT--the Integrating English programme--to exemplify the application of a new model to explain failure in RCTs. To do so, the paper develops a set of categories of context drawing on the wider social evaluation field: backdrop, design, operation and interpretation. Thus, the paper exposes critical weak points in the commission and interpretation, as well as the implementation, of an RCT. Our aim is to work towards more robust evaluations by demonstrating that it is not simply the programme design, implementation and evaluation that can contribute to a lack of impact; there can be more fundamental system issues at play.

The purpose of this study is to explore the views of autistic adults on randomised controlled trials, specifically on processes such as randomisation and blinding, to understand the barriers and facilitators for recruiting autistic people to randomised controlled trials involving medications. We conducted one-to-one interviews with 49 autistic adults. Interviews were audio-recorded and analysed thematically. The participants found randomised controlled trial processes acceptable and linked positive attitudes towards randomised controlled trial participation to autistic peoples' heightened sense of fairness and preference for evidence-driven knowledge. However, randomised controlled trial designs may be incompatible with a (1) preference for a controlled predictable world, (2) perceived vulnerability at physical and mental health levels and (3) history of misunderstanding and exclusion, crucially from healthcare professionals. Suggestions that emerged from our findings include efforts to co-produce research to nurture trust and adapting communication practices to improve access to trials. Autistic people are a highly motivated group to work with research teams to mitigate barriers to randomised controlled trial participation. We explored what psychosocial determinants play a role in the acceptability of randomised controlled trials to test interventions to improve quality of life and mental health in autistic adults. The study provides useful information that may help the design and conduct of more accessible trials with and for the autistic community.

Background: A volitional help sheet (VHS) is an intervention for promoting implementation intentions. This study was the first to test the effectiveness of a VHS for increasing university students' lecture attendance. Aims: To develop a VHS to increase university students' lecture attendance and test its effectiveness at increasing the proportion of lectures attended, and promoting the maintenance of lecture attendance, over an 11-week teaching semester. Sample and Method: One hundred and seventy-eight undergraduate students enrolled in a psychology degree programme were allocated at random to a VHS or active control condition. Prior to intervention, measures of goal intention to attend lectures and trait conscientiousness were collected using self-report, online questionnaires. Over the following 11-week teaching semester, attendance at synchronous (live) online lectures was measured. Results: The VHS condition attended a greater proportion of lectures and maintained their lecture attendance for longer than did the active control condition. These effects were not sensitive to underlying goal intentions, although the sample means on the measures of goal intention were approaching ceiling. Trait conscientiousness increased the effects of the VHS on the proportion of lectures attended. Conclusions: VHSs constitute useful interventions for increasing and maintaining university students' lecture attendance.

Implementation of evidence-based practices for autistic students must be informed by robust research literature. Randomized-controlled trials (RCTs) are often considered a "gold standard" methodology for determining the effectiveness of interventions. However, the complex nature of schools presents challenges for implementing successful RCTs. We review common challenges that researchers may face when implementing RCTs of interventions for autistic students in school settings. Additionally, we provide recommendations to researchers for avoiding and addressing such challenges, based on our own experience conducting a school-based RCT. Limitations and directions for future research are also discussed.

Many autistic children access some form of early intervention, but little is known about the value for money of different programs. We completed a scoping review of full economic evaluations of early interventions for autistic children and/or their families. We identified nine studies and reviewed their methods and quality. Most studies involved behavioral interventions. Two were trial-based, and the others used various modelling methods. Clinical measures were often used to infer dependency levels and quality-adjusted life-years. No family-based or negative outcomes were included. Authors acknowledged uncertain treatment effects. We conclude that economic evaluations in this field are sparse, methods vary, and quality is sometimes poor. Economic research is needed alongside longer-term clinical trials, and outcome measurement in this population requires further exploration.

Background: Although existing studies have shown that both repetitive transcranial magnetic stimulation (rTMS) and music therapy have advantages in the treatment of non-fluent aphasia, the efficacy of the combination of these two methods remains to be investigated. Aims: To investigate the clinical efficacy of low-frequency rTMS combined with music therapy on language function and depression in patients with non-fluent aphasia after stroke. Methods & Procedures: A single-blind parallel randomised controlled trial was conducted. Sixty patients (mean duration = 93.78 days) with non-fluent aphasia after stroke were randomly divided into a traditional therapy group (n = 20), a music therapy group (n = 20) and a combined therapy group (n = 20, 1 Hz). The language function and depression were evaluated before and 3 weeks after treatment with the Chinese version of the Western Aphasia Battery scale, Boston Diagnostic Aphasia Examination scale and Stroke Aphasic Depression Questionnaire Hospital Version scale. Outcomes & Results: The combined therapy group was significantly better in all outcomes than the traditional therapy group and was significantly better in depression than the music therapy group. The music therapy group was significantly better in repetition and depression than the traditional therapy group. Language improvement was positively correlated with depression improvement. For adverse events, only two patients in the combined therapy group showed slight dizziness during rTMS treatment and their symptoms improved after rest. Conclusions & Implications: Our preliminary randomised controlled study indicates that low-frequency rTMS combined with music therapy is feasible and safe in improving language function and depression in non-fluent aphasia patients after stroke.

Risk of bias (RoB) assessment is essential to the systematic review methodology. The new version of the Cochrane RoB tool for randomized trials (RoB 2) was published in 2019 to address limitations identified since the first version of the tool was published in 2008 and to increase the reliability of assessments. This study analyzed the frequency of usage of the RoB 2 and the adequacy of reporting the RoB 2 assessments in non-Cochrane reviews published in 2020. This meta-research study included non-Cochrane systematic reviews of interventions published in 2020. For the reviews that used the RoB 2 tool, we analyzed the reporting of the RoB 2 assessment. Among 3880 included reviews, the Cochrane RoB 1 tool was the most frequently used (N = 2228; 57.4%), followed by the Cochrane RoB 2 tool (N = 267; 6.9%). From 267 reviews that reported using the RoB 2 tool, 213 (79.8%) actually used it. In 26 (12.2%) reviews, erroneous statements were used to indicate the RoB 2 assessment. Only 20 (9.4%) reviews presented a complete RoB 2 assessment with a detailed table of answers to all signaling questions. The judgment of risk of bias by the RoB 2 tool was not justified by a comment in 158 (74.2%) reviews. Only in 33 (14.5%) of reviews the judgment in all domains was justified in the accompanying comment. In most reviews (81.7%), the RoB was inadequately assessed at the study level. In conclusion, the majority of non-Cochrane reviews published in 2020 still used the Cochrane RoB 1 tool. Many reviews used the RoB 2 tool inadequately. Further studies about the uptake and the use of the RoB 2 tool are needed.

Meta-analyses are systematic summaries of research that use quantitative methods to find the mean effect size (standardized mean difference) for interventions. Critics of meta-analysis point out that such analyses can conflate the results of low- and high-quality studies, make improper comparisons and result in statistical noise. All these criticisms are valid for low-quality meta-analyses. However, high-quality meta-analyses correct all these problems. Critics of meta-analysis often suggest that selecting high-quality RCTs is a more valid methodology. However, education RCTs do not show consistent findings, even when all factors are controlled. Education is a social science, and variability is inevitable. Scholars who try to select the best RCTs will likely select RCTs that confirm their bias. High-quality meta-analyses offer a more transparent and rigorous model for determining best practices in education. While meta-analyses are not without limitations, they are the best tool for evaluating educational pedagogies and programs.

We describe an activity that introduces school-aged children to clinical trials, that presents the terminology associated with randomized controlled trials, and that reveals how the findings from clinical trials are applicable to everyone everywhere.

Underrepresentation of historically marginalized populations in clinical trials continues to threaten the validity of health intervention research. Evidence supports the merits of intercept and other proactive forms of recruitment for achieving more equitable representation. However, researchers also report lower retention and adherence to protocols among these populations, particularly in longitudinal studies. Few studies have compared recruitment methods for longitudinal randomized trials testing health interventions, with even fewer having done so for trials involving ecological momentary assessment (EMA). As intervention research integrates EMA and other data collection approaches requiring substantial participant effort, it is critical to better understand the effectiveness and implications of strategies to improve the representativeness of health research. This secondary data analysis compared outcomes of proactive and reactive recruitment strategies (mobile lab intercepts and internet/flyer advertising, respectively) in study inclusion, task completion, and retention within a 14-day randomized controlled trial that used EMA to evaluate cigarette package health messages. Proactive recruitment resulted in higher proportions of participants with low income and education, limited health literacy, and of diverse racial/ethnic makeup. However, this recruitment method also resulted in lower task completion, especially in the second week of the trial period, and lower retention, although group differences were not explained by participant sociodemographic characteristics targeted by inclusion efforts. We conclude that proactive recruitment via intercepts is an effective strategy for health intervention research that aims to include stakeholders from historically marginalized groups but that researchers and funders must recognize these methods require additional resources, considerations, and capacity to address non-trivial challenges to successful participation.

McKinney (2018) has argued that for the scholarship of teaching and learning (SoTL) to advance within a discipline, the integration of SoTL must be closely examined and opportunities for growth in SoTL must be recognized and discussed. To that end, this paper reflects on the degree to which SoTL is integrated into communication sciences and disorders (CSD) by examining a variety of topics: perspectives and theories historically valued by our discipline, existing supports for SoTL at various levels (i.e., individual teacher-scholars, departments, institutions, and the CSD discipline as a whole), the application of SoTL findings in teaching and learning. Four specific recommendations are made because of this examination and reflection.

Randomized A/B tests within online learning platforms represent an exciting direction in learning sciences. With minimal assumptions, they allow causal effect estimation without confounding bias and exact statistical inference even in small samples. However, often experimental samples and/or treatment effects are small, A/B tests are underpowered, and effect estimates are overly imprecise. Recent methodological advances have shown that power and statistical precision can be substantially boosted by coupling design-based causal estimation to machine-learning models of rich log data from historical users who were not in the experiment. Estimates using these techniques remain unbiased and inference remains exact without any additional assumptions. This paper reviews those methods and applies them to a new dataset including over 250 randomized A/B comparisons conducted within ASSISTments, an online learning platform. We compare results across experiments using four novel deep-learning models of auxiliary data and show that incorporating auxiliary data into causal estimates is roughly equivalent to increasing the sample size by 20% on average, or as much as 50-80% in some cases, relative to t-tests, and by about 10% on average, or as much as 30-50%, compared to cutting-edge machine learning unbiased estimates that use only data from the experiments. We show that the gains can be even larger for estimating subgroup effects, hold even when the remnant is unrepresentative of the A/B test sample, and extend to post-stratification population effects estimators.

Although experimental evaluations have been labeled the "gold standard" of evidence for policy (U.S. Department of Education, 2003), evaluations without an analysis of costs are not sufficient for policymaking (Monk, 1995; Ross et al., 2007). Funding organizations now require cost-effectiveness data in most evaluations of effects. Yet, there is little guidance on how to integrate research on costs into efficacy or effectiveness evaluations. As a result, research proposals and papers are disjointed in the treatment of costs, implementation, and effects, and studies often miss opportunities to integrate what is learned from the cost component into what is learned about effectiveness. To address this issue, this paper uses common evaluation frameworks to provide guidance for integrating research on costs into the design of field experiments building on the ingredients method (Levin et al., 2018). The goal is to improve study design, resulting in more cohesive, efficient, and higher-quality evaluations.

The purpose of this study was to examine the effects of MoBeGo, a mobile self-monitoring app, on the initial and sustained academic engagement and disruptive behavior of third- to eighth-grade students with challenging behavior. Student-teacher pairs (N = 57) were randomly assigned to the treatment (MoBeGo) or control (business-as-usual) condition. We conducted systematic direct observation of students' behavior throughout prebaseline, baseline, intervention, and postintervention conditions of the study. Multivariate multilevel models revealed differential improvement for the MoBeGo group in student outcomes (less disruptive behavior; more academic engagement) from baseline to intervention, as well as successful postintervention effects for disruptive behavior. Limitations, future directions, and implications for practice are discussed.

Randomized controlled trials (RCTs) in education are common as the design allows for an unbiased estimate of the overall impact of a program. As more RCTs are completed, researchers are also noting that an overall average impact may mask substantial variation across sites or groups of individuals. Mixed methods can provide insight and help in unpacking some of the reasons for these variations in impact. This article contributes to the field of mixed methods research by integrating mixed methods into a recently developed conceptual framework for understanding variations in impact. We model the use of this approach within the context of an RCT for online courses that found differences in impact across courses.

Longitudinal models typically emphasize between-person predictors of change but ignore how growth varies "within" persons because each person contributes only one data point at each time. In contrast, modeling growth with multi-item assessments allows evaluation of how relative item performance may shift over time. While traditionally viewed as a nuisance under the label of "item parameter drift" (IPD), IPD may be of substantive interest if it reflects how learning manifests on different items or subscales at different rates. In this study, we apply the Explanatory Item Response Model to assess IPD in a causal inference context. Simulation results show that when IPD is not accounted for, both parameter estimates and standard errors can be affected. We illustrate with an empirical application to the persistence of transfer effects from a content literacy intervention , revealing how researchers can leverage IPD to achieve a more fine-grained understanding of how vocabulary learning develops over time.

Multisite cluster randomized trials (MCRTs), in which, the intermediate-level clusters (e.g., classrooms) are randomly assigned to the treatment or control condition within each site (e.g., school), are among the most commonly used experimental designs across a broad range of disciplines. MCRTs often align with the theory that programs are delivered at a cluster-level (e.g., teacher professional development) and provide opportunities to explore treatment effect heterogeneity across sites. In designing experimental studies, a critical step is the statistical power analysis and sample size determination. However, the statistical tools for power analysis of moderator effects in three-level MCRTs are not available. In this study, we derived formulas for calculating the statistical power and the minimum detectable effect size difference (MDESD) with confidence intervals for investigating the effects of various moderators in three-level MCRTs. We considered the levels of the moderators (level-1, -2, and -3), the scales of the moderators (binary and continuous), and random and nonrandomly varying slopes of the (moderated) treatment effects. We validated our formulas through Monte Carlo simulations. Finally, we conclude with directions for future work.

Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment of major depression and binge eating disorder or bulimia nervosa (N = 212) to illustrate the limits of the intraclass coefficient. Randomly selected therapy sessions were rated for therapeutic alliance quality by independent observers using the well-validated Vanderbilt Psychotherapy Process Scale and Vanderbilt Therapeutic Alliance Scale. Scores on subscales related to therapist behavior were restricted, indicating consistent alliance-supportive actions. Inter-rater reliability estimates were low despite high agreement between raters, however. The use of Bland-Altman plots to visualize agreement and data spread is suggested as a useful tool for researchers, consistent with the ideal of exploring reliability from a number of perspectives.

Sulforaphane has been reported to possibly improve core symptoms associated with autism spectrum disorders from mostly small size studies. Here we present results of a larger randomized clinical trial (N = 108) in China. There were no significant changes in caregiver rated scales between sulforaphane and placebo groups. However, clinician rated scales showed a significant improvement in the sulforaphane group, and one third of participants showed at least a 30% decrease in score by 12 weeks treatment. The effects of sulforaphane were seen across the full range of intelligence and greater in participants over 10 years. Sulforaphane was safe and well-tolerated even for young children. The inconsistent results between caregiver and clinician rated scales suggest more clinical trials are needed to confirm our findings.

Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods--lasso regression and Bayesian Additive Regression Trees (BART)--using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.

Although the use of the cluster randomized trial (CRT) design to evaluate vaccines, public health interventions or health systems is increasing, the ethical issues posed by the design are not adequately addressed, especially in low- and middle-income country settings (LMICs). To help reveal ethical challenges, qualitative interviews were conducted with key stakeholders experienced in designing and conducting two selected CRTs in Malawi. The 18 interviewed stakeholders included investigators, clinicians, nurses, data management personnel and community workers who were invited to share their experiences related to implementation of CRTs. Data analysis revealed five major themes with ethical implications: (1) The moral obligation for health care providers to participate in health research and its compensation; (2) Suboptimal care services compromising the integrity of CRT; (3) Ensuring scientific validity and withholding care service; (4) Obtaining valid consent and permission for waiver of consent; and (5) Inadequate risk assessment for trial participation. Understanding key ethical issues posed by CRTs in Malawi could improve ethical review and research oversight of this particular study design.

Meta-analyses of treatment effects in randomized control trials are often faced with the problem of missing information required to calculate effect sizes and their sampling variances. Particularly, correlations between pre- and posttest scores are frequently not available. As an ad-hoc solution, researchers impute a constant value for the missing correlation. As an alternative, we propose adopting a multivariate meta-regression approach that models independent group effect sizes and accounts for the dependency structure using robust variance estimation or three-level modeling. A comprehensive simulation study mimicking realistic conditions of meta-analyses in clinical and educational psychology suggested that imputing a fixed correlation 0.8 or adopting a multivariate meta-regression with robust variance estimation work well for estimating the pooled effect but lead to slightly distorted between-study heterogeneity estimates. In contrast, three-level meta-regressions resulted in largely unbiased fixed effects but more inconsistent prediction intervals. Based on these results recommendations for meta-analytic practice and future meta-analytic developments are provided.

Individual participant data (IPD) meta-analyses of randomised trials are considered a reliable way to assess participant-level treatment effect modifiers but may not make the best use of the available data. Traditionally, effect modifiers are explored one covariate at a time, which gives rise to the possibility that evidence of treatment-covariate interaction may be due to confounding from a different, related covariate. We aimed to evaluate current practice when estimating treatment-covariate interactions in IPD meta-analysis, specifically focusing on involvement of additional covariates in the models. We reviewed 100 IPD meta-analyses of randomised trials, published between 2015 and 2020, that assessed at least one treatment-covariate interaction. We identified four approaches to handling additional covariates: (1) Single interaction model (unadjusted): No additional covariates included (57/100 IPD meta-analyses); (2) Single interaction model (adjusted): Adjustment for the main effect of at least one additional covariate (35/100); (3) Multiple interactions model: Adjustment for at least one two-way interaction between treatment and an additional covariate (3/100); and (4) Three-way interaction model: Three-way interaction formed between treatment, the additional covariate and the potential effect modifier (5/100). IPD is not being utilised to its fullest extent. In an exemplar dataset, we demonstrate how these approaches lead to different conclusions. Researchers should adjust for additional covariates when estimating interactions in IPD meta-analysis providing they adjust their main effects, which is already widely recommended. Further, they should consider whether more complex approaches could provide better information on who might benefit most from treatments, improving patient choice and treatment policy and practice.

Comprehensive evaluation of treatment effects is aided by considerations for moderated effects. In educational research, the combination of natural hierarchical structures and prevalence of group-administered or shared facilitator treatments often produces three-level partially nested data structures. Literature details planning strategies for a variety of experimental designs when moderation effects are of interest but has yet to establish power formulas for detecting moderation effects in three-level partially nested designs. To address this gap, we derive and assess the accuracy of power formulas for detecting the different types of moderation effects possible in these designs. Using Monte Carlo simulation studies, we probe power rates and adequate sample sizes for detecting the different moderation effects while varying common influential factors including variance in the outcome explained by covariates, magnitude of the moderation effect, and sample sizes. The power formulas developed improve the planning of experimental studies with partial nesting and encourage the inclusion of moderator variables to capture for whom and under what conditions a treatment is effective. Educational researchers also have some initial guidance regarding adequate sample sizes and the factors that influence detecting moderation effects in three-level partially nested designs.

Traditionally, cluster randomized controlled trials are analyzed with the average intervention effect of interest. However, in populations that contain higher degrees of heterogeneity or variation may differ across different values of a covariate, which may not be optimal. Within education and social science contexts, exploring the variation in magnitude of treatment effect at different points in the population can indicate where the intervention is most effective rather than assuming an average effect. Data from [Owen, K.L., et al., 2021. Implementation support improves outcomes of a fluency-based mathematics strategy: A cluster-randomized controlled trial. Journal of research on educational effectiveness, 14 (3), 523-542.] were reanalyzed using three modeling approaches: conditional mean-modeling reporting the average treatment effect using linear mixed models, and two quantile regression-based methods. Quantile regressions report the quantile treatment effects at different percentiles: 10th, 25th, 50th, 75th and 90th. For the Quantile approaches, a significant intervention effect in the median to upper quantiles was found and linear quantile mixed model showed improved fit over the other approaches. An improved picture of intervention effects may be apparent using quantile regression methods when analyzing cluster randomized trials that have heterogeneous error variance. In particular, the linear quantile mixed model shows improved model fit allowing a multilevel framework to include random effects. There is considerable scope to extend this framework to incorporate more complex RCT designs.

External validity is an important parameter that needs to be considered for decision making in health research, but no widely accepted measurement tool for the assessment of external validity of randomized controlled trials (RCTs) exists. One of the most limiting factors for creating such a tool is probably the substantial heterogeneity and lack of consensus in this field. The objective of this study was to reach consensus on a definition of external validity and on criteria to assess the external validity of RCTs included in systematic reviews. A three-round online Delphi study was conducted. The development of the Delphi survey was based on findings from a previous systematic review. Potential panelists were identified through a comprehensive web search. Consensus was reached when at least 67% of the panelists agreed to a proposal. Eighty-four panelists from different countries and various disciplines participated in at least one round of this study. Consensus was reached on the definition of external validity ("External validity is the extent to which results of trials provide an acceptable basis for generalization to other circumstances such as variations in populations, settings, interventions, outcomes, or other relevant contextual factors"), and on 14 criteria to assess the external validity of RCTs in systematic reviews. The results of this Delphi study provide a consensus-based reference standard for future tool development. Future research should focus on adapting, pilot testing, and validating these criteria to develop measurement tools for the assessment of external validity.

When randomized control trials are not possible, quasi-experimental methods often represent the gold standard. One quasi-experimental method is difference-in-difference (DiD), which compares changes in outcomes before and after treatment across groups to estimate a causal effect. DiD researchers often use fairly exhaustive robustness checks to make sure the assumptions of the DiD are met. However, less thought is often put into the approach to score item responses from the outcome measure used. For example, surveys are often scored by adding up the item responses to produce sum scores, and achievement tests often rely on scores produced by test vendors, which frequently employ a unidimensional item response theory (IRT) scoring model that implicitly assumes control and treatment participants are exchangeable (i.e., that they come from the same distribution). In this study, several IRT models that parallel the DiD design in terms of groups and timepoints are presented, and their performance is examined. Results indicate that using a scoring approach that parallels the DiD study design reduces bias and improves power, though these approaches can also lead to increased Type I error rates.

Though previously unacknowledged, a SMART (Sequential Multiple Assignment Randomized Trial) design uses both regression discontinuity (RD) and randomized controlled trial (RCT) designs. This combination structure creates a conceptual symbiosis between the two designs that enables both RCT- and previously unrecognized, RD-based inferential claims. In Phase 1, based on a quantitative cut-point, a SMART design yields successful or unsuccessful participants after the initial RCT. In Phase 2, interventions are randomly assigned to successful and unsuccessful, Phase 1 subgroups, thereby yielding both RCT and RD results. Like RCT-based inferences, many but not all newly recognized, RD-based SMART inferences are potentially unbiased. In particular, comparisons between RCT and RD outcomes are potentially problematic. Principles from the within-study comparison literature are used to inform SMART-based research, thereby introducing otherwise ignored inferential and procedural possibilities. Awareness of RD designs embedded within SMARTs encourages previously unrecognized inferential claims, though potential internal validity threats and limitations must be carefully weighed. Finally, new, SMART-like sequences of quasi-experiments and experiments can be tested and embedded logic applied.

Online credit recovery will likely expand in the coming years as school districts try to address increased course failure rates brought on by the coronavirus pandemic. Some researchers and policymakers, however, raise concerns over how much students learn in online courses, and there is limited evidence about the effectiveness of online credit recovery. This article presents findings from a multisite randomized study, conducted prior to the pandemic, to expand the field's understanding of online credit recovery's effectiveness. Within 24 high schools from a large urban district, the study randomly assigned 1,683 students who failed Algebra 1 or ninth grade English to a summer credit recovery class that either used an online curriculum with in-class teacher support or the school's business-as-usual teacher-directed class. The results suggest that online credit recovery had relatively insignificant effects on student course experiences and content knowledge, but significantly lower credit recovery rates for English. There was limited heterogeneity in effects across students and schools. Non-response on the study-administered student survey and test limit our confidence in the student experience and content knowledge results, but the findings are robust to different approaches to handling the missing data (multiple imputation or listwise deletion). We discuss how the findings add to the evidence base about online credit recovery and the implications for future research.

Before embarking on an individual participant data meta-analysis (IPDMA) project, researchers should consider the power of their planned IPDMA conditional on the studies promising their IPD and their characteristics. Such power estimates help inform whether the IPDMA project is worth the time and funding investment, before IPD are collected. Here, we suggest how to estimate the power of a planned IPDMA of randomised trials aiming to examine treatment-covariate interactions at the participant-level (i.e., treatment effect modifiers). We focus on a time-to-event (survival) outcome with a binary or continuous covariate, and propose an approximate analytic power calculation that conditions on the actual characteristics of trials, for example, in terms of sample sizes and covariate distributions. The proposed method has five steps: (i) extracting the following aggregate data for each group in each trial--the number of participants and events, the mean and SD for each continuous covariate, and the proportion of participants in each category for each binary covariate; (ii) specifying a minimally important interaction size; (iii) deriving an approximate estimate of Fisher's information matrix for each trial and the corresponding variance of the interaction estimate per trial, based on assuming an exponential survival distribution; (iv) deriving the estimated variance of the summary interaction estimate from the planned IPDMA, under a common-effect assumption, and (v) calculating the power of the IPDMA based on a two-sided Wald test. Stata and R code are provided and a real example provided for illustration. Further evaluation in real examples and simulations is needed.