Search

In this randomized control study, we evaluated science teachers' pedagogical practices via classroom observations following ongoing, intensive, and structured instructional support sessions. These sessions included virtual professional development (VPD) and virtual mentoring and coaching (VMC) that accompanied a literacy-infused science curriculum. Using a low-inference observational instrument, we explored the direct impact of VPD and VMC on fifth-grade science teachers' observed time allocation in a randomized controlled trial (RCT) validation study. The observations were collected three times during science instruction from 121 teachers in 68 schools from 35 public school districts in the U.S. state of Texas, during the 2017-2018 school year. Preliminary findings revealed pedagogical differences in time allocation among teachers between treatment and control classrooms. We identified improved instructional practices within treatment classrooms, which suggests the intervention had a positive effect by enhancing the quality of pedagogy as well as the content-area instruction in science.

Cheating in online exams without face-to-face proctoring has been a general concern for academic instructors during the crisis caused by COVID-19. The main goal of this work is to evaluate the cost of these dishonest practices by comparing the academic performance of webcam-proctored students and their unproctored peers in an online gradable test. With this aim in mind, we carried out a randomized field experiment using a simple video surveillance system through Google Meet during an online closed-book final exam of an Introduction to Microeconomics course. Given that all conditions except for webcam monitoring were identical, differences in between-subjects scores are attributed to academic dishonesty. After controlling for potential confounding factors, including gender, academic degree, instructor, previous score and whether students were repeaters or not, we found that those students who were proctored via webcam obtained statistically significant lower scores in the final exam than those who were not using this surveillance system with a low level of invasiveness. Inspection of the potential factors behind these differences in scores suggests that the poorer performance of proctored students is more related to academic dishonesty than to reasons involving anxiety or heterogeneity factors.

When planning a cluster randomized trial, evaluators often have access to an enumerated cohort representing the target population of clusters. Practicalities of conducting the trial, such as the need to oversample clusters with certain characteristics in order to improve trial economy or support inferences about subgroups of clusters, may preclude simple random sampling from the cohort into the trial, and thus interfere with the goal of producing generalizable inferences about the target population. We describe a nested trial design where the randomized clusters are embedded within a cohort of trial-eligible clusters from the target population and where clusters are selected for inclusion in the trial with known sampling probabilities that may depend on cluster characteristics (e.g., allowing clusters to be chosen to facilitate trial conduct or to examine hypotheses related to their characteristics). We develop and evaluate methods for analyzing data from this design to generalize causal inferences to the target population underlying the cohort. We present identification and estimation results for the expectation of the average potential outcome and for the average treatment effect, in the entire target population of clusters and in its non-randomized subset. In simulation studies, we show that all the estimators have low bias but markedly different precision. Cluster randomized trials where clusters are selected for inclusion with known sampling probabilities that depend on cluster characteristics, combined with efficient estimation methods, can precisely quantify treatment effects in the target population, while addressing objectives of trial conduct that require oversampling clusters on the basis of their characteristics.

Individual participant data (IPD) meta-analysis projects obtain, harmonise, and synthesise original data from multiple studies. Many IPD meta-analyses of randomised trials are initiated to identify treatment effect modifiers at the individual level, thus requiring statistical modelling of interactions between treatment effect and participant-level covariates. Using a two-stage approach, the interaction is estimated in each trial separately and combined in a meta-analysis. In practice, two complications often arise with continuous outcomes: examining non-linear relationships for continuous covariates and dealing with multiple time-points. We propose a two-stage multivariate IPD meta-analysis approach that summarises non-linear treatment-covariate interaction functions at multiple time-points for continuous outcomes. A set-up phase is required to identify a small set of time-points; relevant knot positions for a spline function, at identical locations in each trial; and a common reference group for each covariate. Crucially, the multivariate approach can include participants or trials with missing outcomes at some time-points. In the first stage, restricted cubic spline functions are fitted and their interaction with each discrete time-point is estimated in each trial separately. In the second stage, the parameter estimates defining these multiple interaction functions are jointly synthesised in a multivariate random-effects meta-analysis model accounting for within-trial and across-trial correlation. These meta-analysis estimates define the summary non-linear interactions at each time-point, which can be displayed graphically alongside confidence intervals. The approach is illustrated using an IPD meta-analysis examining effect modifiers for exercise interventions in osteoarthritis, which shows evidence of non-linear relationships and small gains in precision by analysing all time-points jointly.

Increasing integrity concerns in medical research have prompted the development of tools to detect untrustworthy studies. Existing tools primarily assess published aggregate data (AD), though scrutiny of individual participant data (IPD) is often required to detect trustworthiness issues. Thus, we developed the IPD Integrity Tool for detecting integrity issues in randomised trials with IPD available. This manuscript describes the development of this tool. We conducted a literature review to collate and map existing integrity items. These were discussed with an expert advisory group; agreed items were included in a standardised tool and automated where possible. We piloted this tool in two IPD meta-analyses (including 116 trials) and conducted preliminary validation checks on 13 datasets with and without known integrity issues. We identified 120 integrity items: 54 could be conducted using AD, 48 required IPD, and 18 were possible with AD, but more comprehensive with IPD. An initial reduced tool was developed through consensus involving 13 advisors, featuring 11 AD items across four domains, and 12 IPD items across eight domains. The tool was iteratively refined throughout piloting and validation. All studies with known integrity issues were accurately identified during validation. The final tool includes seven AD domains with 13 items and eight IPD domains with 18 items. The quality of evidence informing healthcare relies on trustworthy data. We describe the development of a tool to enable researchers, editors, and others to detect integrity issues using IPD. Detailed instructions for its application are published as a complementary manuscript in this issue.

Increasing concerns about the trustworthiness of research have prompted calls to scrutinise studies' Individual Participant Data (IPD), but guidance on how to do this was lacking. To address this, we developed the IPD Integrity Tool to screen randomised controlled trials (RCTs) for integrity issues. Development of the tool involved a literature review, consultation with an expert advisory group, piloting on two IPD meta-analyses (including 73 trials with IPD), preliminary validation on 13 datasets with and without known integrity issues, and evaluation to inform iterative refinements. The IPD Integrity Tool comprises 31 items (13 study-level, 18 IPD-specific). IPD-specific items are automated where possible, and are grouped into eight domains, including unusual data patterns, baseline characteristics, correlations, date violations, patterns of allocation, internal and external inconsistencies, and plausibility of data. Users rate each item as having either no issues, some/minor issue(s), or many/major issue(s) according to decision rules, and justification for each rating is recorded. Overall, the tool guides decision-making by determining whether a trial has no concerns, some concerns requiring further information, or major concerns warranting exclusion from evidence synthesis or publication. In our preliminary validation checks, the tool accurately identified all five studies with known integrity issues. The IPD Integrity Tool enables users to assess the integrity of RCTs via examination of IPD. The tool may be applied by evidence synthesists, editors and others to determine whether an RCT should be considered sufficiently trustworthy to contribute to the evidence base that informs policy and practice.

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two different approaches: (1) manually by human reviewers, and (2) automatically by the RobotReviewer. The manual assessment was based on two groups independently, with two additional rounds of verification. The agreement between RobotReviewer and humans was measured via the concordance rate and Cohen's kappa statistics, based on the comparison of binary classification of the risk of bias (low vs. high/unclear) as restricted by RobotReviewer. The concordance rates varied by domain, ranging from 63.07% to 83.32%. Cohen's kappa statistics showed a poor agreement between humans and RobotReviewer for allocation concealment (K = 0.25, 95% CI: 0.21-0.30), blinding of outcome assessors (K = 0.27, 95% CI: 0.23-0.31); While moderate for random sequence generation (K = 0.46, 95% CI: 0.41-0.50) and blinding of participants and personnel (K = 0.59, 95% CI: 0.55-0.64). The findings demonstrate that there were domain-specific differences in the level of agreement between RobotReviewer and humans. We suggest that it might be a useful auxiliary tool, but the specific manner of its integration as a complementary tool requires further discussion.

The National Board of Certification and Recertification for Nurse Anesthetists (NBCRNA) conducted a one-year research study comparing performance on the traditional continued professional certification assessment, administered at a test center or online with remote proctoring, to a longitudinal assessment that required answering quarterly questions online on demand. A randomized controlled trial of 1,000 certified registered nurse anesthetists (500 randomly assigned to the traditional assessment group and longitudinal assessment group) aimed to 1) compare assessment performance between groups, 2) compare perceptions and user experience between groups; and 3) describe participant feedback about usability of the longitudinal assessment platform. The mean scaled score for the traditional assessment group exceeded that of the longitudinal assessment group when scoring the first responses; however, upon scoring the longitudinal assessment group's most recent responses on repeat questions previously answered incorrectly, the mean scaled score was higher than the traditional assessment group. Both groups were satisfied with their experience, with slightly higher feedback ratings for the longitudinal assessment group who also found the platform easy to use and navigate. Overall results suggest the longitudinal assessment is a feasible, acceptable, and usable format to assess specialized knowledge for continued healthcare professional certification.

During the COVID-19 pandemic, the race to find an effective vaccine or treatment saw an 'extraordinary number' of clinical trials being conducted. While there were some key success stories, not all trials produced results that informed patient care. There was a significant amount of waste in clinical research during the pandemic which is said to have hampered an evidence-based response. Conducting trials which could have been predicted to fail to answer the research question (e.g. because they are not large enough to provide a definitive result) is not only a waste of resources but also a breach of research participants' trust and a violation of research ethics. The issues seen in COVID-19 clinical trials are symptomatic of a wider trial design crisis where many trials do not provide informative results. This paper examines the roles of key stakeholders in delivering ethical and informative trials and whether guidance published by 'The Good Clinical Trials Collaborative' could be used to align key stakeholder groups and enable a joined-up approach to improve clinical trial design.

The COVID-19 pandemic touched off an unprecedented search for vaccines and treatments. Without question, the development of vaccines to prevent COVID-19 was an enormous scientific accomplishment. Further, the RECOVERY and Solidarity trials identified effective treatments for COVID-19. But all was not success. The urgent need for COVID-19 prevention and treatment fueled an embrace of risks--to research participants and to the reliability of the science itself--as allegedly necessary costs to speed scientific progress. Scientists and (even) ethicists supported overturning longstanding norms protecting healthy volunteers in human challenge trials to speed vaccine development, but these trials led to no vaccines. Physicians, with the approval of research ethics committees, designed hundreds of unblinded, single-center clinical trials at high risk of bias to speed the identification of new treatments. But these clinical trials led to no treatments. The lesson for future pandemics is that the acceptance of greater risks to participants or science does not reliably lead to progress. We are better served by science that upholds the highest ethical and methodological standards.

Emergent technologies present platforms for educational researchers to conduct randomized controlled trials (RCTs) and collect rich data to study students' performance, behavior, learning processes, and outcomes in authentic learning environments. As educational research increasingly uses methods and data collection from such platforms, it is necessary to consider the most appropriate ways to analyze this data to draw causal inferences from RCTs. Here, we examine whether and how analysis results are impacted by accounting for multilevel variance in samples from RCTs with student-level randomization within one platform. We propose and demonstrate a method that leverages auxiliary non-experimental "remnant" data collected within a learning platform to inform analysis decisions. Specifically, we compare five commonly-applied analysis methods to estimate treatment effects while accounting for, or ignoring, class-level factors and observed measures of confidence and accuracy to identify best practices under real-world conditions. We find that methods that account for groups as either fixed effects or random effects consistently outperform those that ignore group-level factors, even though randomization was applied at the student level. However, we found no meaningful differences between the use of fixed or random effects as a means to account for groups. We conclude that analyses of online experiments should account for the naturally-nested structure of students within classes, despite the notion that student-level randomization may alleviate group-level differences. Further, we demonstrate how to use remnant data to identify appropriate methods for analyzing experiments. These findings provide practical guidelines for researchers conducting RCTs in similar educational technologies to make more informed decisions when approaching analyses.

Objective: To determine if treatment of co-occurring adult ADHD and Cannabis Use Disorder (CUD) with extended-release mixed amphetamine salts (MAS-ER) would be effective at improving ADHD symptoms and promoting abstinence. Method: A 12-week randomized, double-blind, two-arm pilot feasibility trial of adults with comorbid ADHD and CUD (n = 28) comparing MAS-ER (80 mg) to placebo. Main outcomes: ADHD: [greater than or equal to] 30% symptom reduction, measured by the Adult ADHD Investigator Symptom Rating Scale (AISRS). CUD: Abstinence during last 2 observed weeks of maintenance phase. Results: Overall, medication was well-tolerated. There was no significant difference in ADHD symptom reduction (MAS-ER: 83.3%; placebo: 71.4%; p = 0.65) or cannabis abstinence (MAS-ER: 15.4%; placebo: 0%; p = 0.27). MAS-ER group showed a significant decrease in weekly cannabis use days over time compared to placebo (p < 0.0001). Conclusions: MAS-ER was generally well-tolerated. The small sample size precluded a determination of MAS-ER's superiority reducing ADHD symptoms or promoting abstinence. Notably, MAS-ER significantly reduced weekly days of use over time.

Cluster randomized trials (CRTs) are commonly used to evaluate the causal effects of educational interventions, where the entire clusters (e.g., schools) are randomly assigned to treatment or control conditions. This study introduces statistical methods for designing and analyzing two-level (e.g., students nested within schools) and three-level (e.g., students nested within classrooms nested within schools) CRTs. Specifically, we utilize hierarchical linear models (HLMs) to account for the dependency of the intervention participants within the same clusters, estimating the average treatment effects (ATEs) of educational interventions and other effects of interest (e.g., moderator and mediator effects). We demonstrate methods and tools for sample size planning and statistical power analysis. Additionally, we discuss common challenges and potential solutions in the design and analysis phases, including the effects of omitting one level of clustering, non-compliance, heterogeneous variance, blocking, threats to external validity, and cost-effectiveness of the intervention. We conclude with some practical suggestions for CRT design and analysis, along with recommendations for further readings.

Background: Physical activity (PA) may benefit academic performance, but it is unclear what kind of classroom-based PA is optimal for learning. Aim: We studied the effects of physically active maths lessons on children's maths performance and maths-related effects, and whether gender and previous mathematical or motor skills modify these effects. Sample: A total of 22 volunteered teachers and their pupils with signed consent (N = 397, mean age: 9.3 years, 51% females) participated in a 5-month, teacher-led, multi-arm, cluster-randomized controlled trial. Methods: The intervention included a PAL group (20 min of physically active learning in each 45-min lesson), a breaks group (two 5-min PA breaks in each 45-min lesson) and a control group (traditional teaching). Maths performance was assessed with a tailored curriculum-based test. Maths-related enjoyment, self-perceptions and anxiety were measured with a self-reported questionnaire. The individual-level intervention effects were tested via covariate-adjusted linear mixed-effect models with school classes serving as random effects. Results: Changes in maths performance or self-perceptions did not differ between the intervention groups. Maths anxiety in learning situations increased in the PAL group (effect .28, 95% CI = .01--.56); there was no change in the other groups. Subgroup analyses suggested that maths anxiety increased in the PAL group among children in the two lowest tertiles of motor skills. It decreased in the highest tertile. Enjoyment decreased in the breaks group among pupils in the lowest motor skill tertile. Conclusions: Physically active maths lessons did not affect maths performance or self-perceptions but had divergent effects on maths anxiety and enjoyment, depending on motor skills.

This review aims to systematically summarize existing evidence to determine the effectiveness of cognitive-based interventions (CBIs) on psychological health and well-being among parents of children with developmental disabilities (DD). Six databases were searched to identify eligible randomized controlled trials (RCTs) from their inception to April 2023. The revised Cochrane Risk of Bias tool for RCTs was applied to assess the risk of bias and the certainty of evidence was evaluated using the Grading of Recommendation, Assessment, Development and Evaluation. Meta-analyses were conducted using a random-effects model. Twenty-five RCTs involving 1915 participants were identified. The results indicated that CBIs reduced parental stress levels (Hedges' g = - 0.69), depressive symptoms (g = - 0.95), anxiety levels (g = - 0.78), and parental distress (g = - 0.29), and improved parental well-being (g = 0.62) and parent-child relationships (g = 0.43) postintervention compared with the active/inactive control groups. Subgroup analysis of the effectiveness of interventions using mindfulness-based interventions and cognitive behavioural therapy showed positive effects. The favourable intervention duration and participant targets were also identified in this review. Furthermore, the effects of CBIs were impacted by the different types of DD among the children. This review highlighted the positive effects of CBIs on parental stress levels, depressive symptoms, anxiety levels, parental distress levels, parental well-being levels, and parent-child relationships. Future well-designed RCTs are needed to further investigate the effects of MBIs and CBT interventions on children with DD and their parents, as well as the factors and mechanisms of action affecting the efficacy of these interventions.

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

When studies use different scales to measure continuous outcomes, standardised mean differences (SMD) are required to meta-analyse the data. However, outcomes are often reported as endpoint or change from baseline scores. Combining corresponding SMDs can be problematic and available guidance advises against this practice. We aimed to examine the impact of combining the two types of SMD in meta-analyses of depression severity. We used individual participant data on pharmacological interventions (89 studies, 27,409 participants) and internet-delivered cognitive behavioural therapy (iCBT; 61 studies, 13,687 participants) for depression to compare endpoint and change from baseline SMDs at the study level. Next, we performed pairwise (PWMA) and network meta-analyses (NMA) using endpoint SMDs, change from baseline SMDs, or a mixture of the two. Study-specific SMDs calculated from endpoint and change from baseline data were largely similar, although for iCBT interventions 25% of the studies at 3 months were associated with important differences between study-specific SMDs (median 0.01, IQR -0.10, 0.13) especially in smaller trials with baseline imbalances. However, when pooled, the differences between endpoint and change SMDs were negligible. Pooling only the more favourable of the two SMDs did not materially affect meta-analyses, resulting in differences of pooled SMDs up to 0.05 and 0.13 in the pharmacological and iCBT datasets, respectively. Our findings have implications for meta-analyses in depression, where we showed that the choice between endpoint and change scores for estimating SMDs had immaterial impact on summary meta-analytic estimates. Future studies should replicate and extend our analyses to fields other than depression.

Meta-analyses are systematic summaries of research that use quantitative methods to find the mean effect size (standardized mean difference) for interventions. Critics of meta-analysis point out that such analyses can conflate the results of low- and high-quality studies, make improper comparisons and result in statistical noise. All these criticisms are valid for low-quality meta-analyses. However, high-quality meta-analyses correct all these problems. Critics of meta-analysis often suggest that selecting high-quality RCTs is a more valid methodology. However, education RCTs do not show consistent findings, even when all factors are controlled. Education is a social science, and variability is inevitable. Scholars who try to select the best RCTs will likely select RCTs that confirm their bias. High-quality meta-analyses offer a more transparent and rigorous model for determining best practices in education. While meta-analyses are not without limitations, they are the best tool for evaluating educational pedagogies and programs.

This article examines the implementation, participation rates, and potential determinants of participation in the digital addiction prevention program "ready4life." A two-arm cluster-randomized trial recruited German vocational students via class-based strategies. Intervention group received 16 weeks of in-app coaching; the control group received health behavior information, with coaching offered after 12 months. Potential determinants of participation were analyzed based on class and individual characteristics. Out of 525 contacted schools, 35 participated, enrolling 376 classes. Implementation during the pandemic required flexible adjustments, with 49.7% of introductions conducted in person, 43.1% digitally via online streaming, and 7.2% received a video link via email. Despite challenges, 72.3% of the vocational students downloaded the app, and 46.7% gave informed consent. Participation rates were highest among (associate) professionals, vocational grammar school classes, classes introduced by females, younger individuals, members of the project team, and classes introduced face-to-face. Female gender, lower social competencies, lifetime cannabis use, higher problematic internet use, and higher perceived stress were associated with higher individual participation. The study highlights the importance of proactive outreach and personalized interventions for addiction prevention programs in vocational schools. While reached students aligned with the aims of the app, tailored recruitment strategies could enhance engagement among under-represented groups.

Violence is a major public health problem globally, with the highest rates in low- and middle-income countries (LMICs) in the Americas and southern Africa. Parenting programmes in high-income countries can diminish risk for violence, by reducing risk factors such as child aggression and harsh parenting, and increasing protective factors such as child cognitive development and school readiness. However, there is critical need to identify low-cost programmes with replicable benefits that work in real-world LMICs contexts. A three-arm, randomised, single-blind trial evaluated effects of two low-cost, group-based parenting programmes recommended for LMICs (ACT: Raising Safe Kids; DBS: dialogic book-sharing) on child aggression (primary outcome), child development, parenting, maltreatment, and stress. Participants were 369 children with medium-high levels of aggression (mean age 3.1 years at baseline) in poor households. Interventions were implemented in city health and education services in southern Brazil. Maternal reports, filmed observations, child tasks, and hair cortisol were assessed at baseline, 1-month post-intervention, and 8-month follow-up. Intention-to-treat analyses compared each of ACT and DBS with a control group. Three hundred sixty-eight (99.7%) participants completed follow-up assessments 8 months after the interventions. There was no effect of ACT (standardised mean difference, SMD 0.11, 95% CI - 0.05, 0.27) or DBS (SMD 0.05, 95% CI - 0.11, 0.21) on the primary outcome of child aggression. ACT reduced harsh parenting behaviour post-intervention (SMD - 0.23; 95% CI - 0.46, 0.01), but not at follow-up. DBS improved book-sharing practices at both time points (e.g., maternal sensitivity at follow-up SMD 0.33; 95% CI 0.08, 0.57). There were no benefits of either programme for other parenting, child development, or stress outcomes. Two parenting programmes in Brazil had small effects on parenting practices but did not reduce child aggression or several other important risk/protective factors for violence. Effective early interventions that reduce violence in real-world LMIC settings are highly desirable but may be challenging to achieve.

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing--remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

Discussions of randomised controlled trials (RCTs) in education that do not show an impact regularly focus on the intervention and how it failed to impact on expected measures, with typologies identifying persistent critical points of failure. This paper uses one such RCT--the Integrating English programme--to exemplify the application of a new model to explain failure in RCTs. To do so, the paper develops a set of categories of context drawing on the wider social evaluation field: backdrop, design, operation and interpretation. Thus, the paper exposes critical weak points in the commission and interpretation, as well as the implementation, of an RCT. Our aim is to work towards more robust evaluations by demonstrating that it is not simply the programme design, implementation and evaluation that can contribute to a lack of impact; there can be more fundamental system issues at play.

The purpose of this study is to explore the views of autistic adults on randomised controlled trials, specifically on processes such as randomisation and blinding, to understand the barriers and facilitators for recruiting autistic people to randomised controlled trials involving medications. We conducted one-to-one interviews with 49 autistic adults. Interviews were audio-recorded and analysed thematically. The participants found randomised controlled trial processes acceptable and linked positive attitudes towards randomised controlled trial participation to autistic peoples' heightened sense of fairness and preference for evidence-driven knowledge. However, randomised controlled trial designs may be incompatible with a (1) preference for a controlled predictable world, (2) perceived vulnerability at physical and mental health levels and (3) history of misunderstanding and exclusion, crucially from healthcare professionals. Suggestions that emerged from our findings include efforts to co-produce research to nurture trust and adapting communication practices to improve access to trials. Autistic people are a highly motivated group to work with research teams to mitigate barriers to randomised controlled trial participation. We explored what psychosocial determinants play a role in the acceptability of randomised controlled trials to test interventions to improve quality of life and mental health in autistic adults. The study provides useful information that may help the design and conduct of more accessible trials with and for the autistic community.

Background: A volitional help sheet (VHS) is an intervention for promoting implementation intentions. This study was the first to test the effectiveness of a VHS for increasing university students' lecture attendance. Aims: To develop a VHS to increase university students' lecture attendance and test its effectiveness at increasing the proportion of lectures attended, and promoting the maintenance of lecture attendance, over an 11-week teaching semester. Sample and Method: One hundred and seventy-eight undergraduate students enrolled in a psychology degree programme were allocated at random to a VHS or active control condition. Prior to intervention, measures of goal intention to attend lectures and trait conscientiousness were collected using self-report, online questionnaires. Over the following 11-week teaching semester, attendance at synchronous (live) online lectures was measured. Results: The VHS condition attended a greater proportion of lectures and maintained their lecture attendance for longer than did the active control condition. These effects were not sensitive to underlying goal intentions, although the sample means on the measures of goal intention were approaching ceiling. Trait conscientiousness increased the effects of the VHS on the proportion of lectures attended. Conclusions: VHSs constitute useful interventions for increasing and maintaining university students' lecture attendance.

Implementation of evidence-based practices for autistic students must be informed by robust research literature. Randomized-controlled trials (RCTs) are often considered a "gold standard" methodology for determining the effectiveness of interventions. However, the complex nature of schools presents challenges for implementing successful RCTs. We review common challenges that researchers may face when implementing RCTs of interventions for autistic students in school settings. Additionally, we provide recommendations to researchers for avoiding and addressing such challenges, based on our own experience conducting a school-based RCT. Limitations and directions for future research are also discussed.

Many autistic children access some form of early intervention, but little is known about the value for money of different programs. We completed a scoping review of full economic evaluations of early interventions for autistic children and/or their families. We identified nine studies and reviewed their methods and quality. Most studies involved behavioral interventions. Two were trial-based, and the others used various modelling methods. Clinical measures were often used to infer dependency levels and quality-adjusted life-years. No family-based or negative outcomes were included. Authors acknowledged uncertain treatment effects. We conclude that economic evaluations in this field are sparse, methods vary, and quality is sometimes poor. Economic research is needed alongside longer-term clinical trials, and outcome measurement in this population requires further exploration.

Background: Although existing studies have shown that both repetitive transcranial magnetic stimulation (rTMS) and music therapy have advantages in the treatment of non-fluent aphasia, the efficacy of the combination of these two methods remains to be investigated. Aims: To investigate the clinical efficacy of low-frequency rTMS combined with music therapy on language function and depression in patients with non-fluent aphasia after stroke. Methods & Procedures: A single-blind parallel randomised controlled trial was conducted. Sixty patients (mean duration = 93.78 days) with non-fluent aphasia after stroke were randomly divided into a traditional therapy group (n = 20), a music therapy group (n = 20) and a combined therapy group (n = 20, 1 Hz). The language function and depression were evaluated before and 3 weeks after treatment with the Chinese version of the Western Aphasia Battery scale, Boston Diagnostic Aphasia Examination scale and Stroke Aphasic Depression Questionnaire Hospital Version scale. Outcomes & Results: The combined therapy group was significantly better in all outcomes than the traditional therapy group and was significantly better in depression than the music therapy group. The music therapy group was significantly better in repetition and depression than the traditional therapy group. Language improvement was positively correlated with depression improvement. For adverse events, only two patients in the combined therapy group showed slight dizziness during rTMS treatment and their symptoms improved after rest. Conclusions & Implications: Our preliminary randomised controlled study indicates that low-frequency rTMS combined with music therapy is feasible and safe in improving language function and depression in non-fluent aphasia patients after stroke.

Risk of bias (RoB) assessment is essential to the systematic review methodology. The new version of the Cochrane RoB tool for randomized trials (RoB 2) was published in 2019 to address limitations identified since the first version of the tool was published in 2008 and to increase the reliability of assessments. This study analyzed the frequency of usage of the RoB 2 and the adequacy of reporting the RoB 2 assessments in non-Cochrane reviews published in 2020. This meta-research study included non-Cochrane systematic reviews of interventions published in 2020. For the reviews that used the RoB 2 tool, we analyzed the reporting of the RoB 2 assessment. Among 3880 included reviews, the Cochrane RoB 1 tool was the most frequently used (N = 2228; 57.4%), followed by the Cochrane RoB 2 tool (N = 267; 6.9%). From 267 reviews that reported using the RoB 2 tool, 213 (79.8%) actually used it. In 26 (12.2%) reviews, erroneous statements were used to indicate the RoB 2 assessment. Only 20 (9.4%) reviews presented a complete RoB 2 assessment with a detailed table of answers to all signaling questions. The judgment of risk of bias by the RoB 2 tool was not justified by a comment in 158 (74.2%) reviews. Only in 33 (14.5%) of reviews the judgment in all domains was justified in the accompanying comment. In most reviews (81.7%), the RoB was inadequately assessed at the study level. In conclusion, the majority of non-Cochrane reviews published in 2020 still used the Cochrane RoB 1 tool. Many reviews used the RoB 2 tool inadequately. Further studies about the uptake and the use of the RoB 2 tool are needed.

Single-case experimental designs (SCEDs) have a long history in clinical and educational disciplines. One underdeveloped area in advancing SCED design and analysis is understanding the process of how internal validity threats and operational concerns are avoided or mitigated. Two strategies to ameliorate such issues in SCED involve replication and randomization. Although replication and randomization are indispensable tools in improving the internal validity of SCEDs, little attention has been paid to (a) why this is the case; or (b) the ways in which these design features are not immune from internal validity threats and operational concerns. In the current paper, we describe the use of directed acyclic graphs (DAGs) to better understand, discover, and mitigate internal validity threats and operational concerns in SCEDs. DAGs are a tool for visualizing causal relations among variables and can help researchers identify both causal and noncausal relations among their variables according to specific algorithms. We introduce the use of DAGs in SCEDs to prompt applied researchers to conceptualize internal validity threats and operational concerns, even when an SCED includes replication and randomization in the design structure. We discuss the general principles of causal inference in conventional "group" designs and in SCEDs, the unique factors impacting SCEDs, and how DAGs can be incorporated into SCEDs. We also discuss the limitations of DAGs applied to SCEDs, as well as future directions for this area of work.

McKinney (2018) has argued that for the scholarship of teaching and learning (SoTL) to advance within a discipline, the integration of SoTL must be closely examined and opportunities for growth in SoTL must be recognized and discussed. To that end, this paper reflects on the degree to which SoTL is integrated into communication sciences and disorders (CSD) by examining a variety of topics: perspectives and theories historically valued by our discipline, existing supports for SoTL at various levels (i.e., individual teacher-scholars, departments, institutions, and the CSD discipline as a whole), the application of SoTL findings in teaching and learning. Four specific recommendations are made because of this examination and reflection.

Randomized A/B tests within online learning platforms represent an exciting direction in learning sciences. With minimal assumptions, they allow causal effect estimation without confounding bias and exact statistical inference even in small samples. However, often experimental samples and/or treatment effects are small, A/B tests are underpowered, and effect estimates are overly imprecise. Recent methodological advances have shown that power and statistical precision can be substantially boosted by coupling design-based causal estimation to machine-learning models of rich log data from historical users who were not in the experiment. Estimates using these techniques remain unbiased and inference remains exact without any additional assumptions. This paper reviews those methods and applies them to a new dataset including over 250 randomized A/B comparisons conducted within ASSISTments, an online learning platform. We compare results across experiments using four novel deep-learning models of auxiliary data and show that incorporating auxiliary data into causal estimates is roughly equivalent to increasing the sample size by 20% on average, or as much as 50-80% in some cases, relative to t-tests, and by about 10% on average, or as much as 30-50%, compared to cutting-edge machine learning unbiased estimates that use only data from the experiments. We show that the gains can be even larger for estimating subgroup effects, hold even when the remnant is unrepresentative of the A/B test sample, and extend to post-stratification population effects estimators.

Randomized controlled trials (RCTs) are an increasingly common research design for evaluating the effectiveness of community college (CC) interventions. However, when planning an RCT evaluation of a CC intervention, there is limited empirical information about what sized effects an intervention might reasonably achieve, which can lead to under- or over-powered studies. Relatedly, when interpreting results from an evaluation of a CC intervention, there is limited empirical information to contextualize the magnitude of an effect estimate relative to what sized effects have been observed in past evaluations. We provide empirical benchmarks to help with the planning and interpretation of community college evaluations. To do so, we present findings across well-executed RCTs of 39 CC interventions that are part of a unique dataset known as The Higher Education Randomized Controlled Trials (THE-RCT). The analyses include 21,163-65,604 students (depending on outcome and semester) enrolled in 44 institutions. Outcomes include enrollment, credits earned, and credential attainment. Effect size distributions are presented by outcome and semester. For example, across the interventions examined, the mean effect on cumulative credits earned after three semesters is 1.14 credits. Effects around 0.16 credits are at the 25th percentile of the distribution. Effects around 1.69 credits are at the 75th percentile of the distribution. This work begins to provide empirical benchmarks for planning and interpreting effects of CC evaluations. A public database with effect sizes is available to researchers (https:// www.mdrc.org/the-rct-empirical-benchmarks).

Although experimental evaluations have been labeled the "gold standard" of evidence for policy (U.S. Department of Education, 2003), evaluations without an analysis of costs are not sufficient for policymaking (Monk, 1995; Ross et al., 2007). Funding organizations now require cost-effectiveness data in most evaluations of effects. Yet, there is little guidance on how to integrate research on costs into efficacy or effectiveness evaluations. As a result, research proposals and papers are disjointed in the treatment of costs, implementation, and effects, and studies often miss opportunities to integrate what is learned from the cost component into what is learned about effectiveness. To address this issue, this paper uses common evaluation frameworks to provide guidance for integrating research on costs into the design of field experiments building on the ingredients method (Levin et al., 2018). The goal is to improve study design, resulting in more cohesive, efficient, and higher-quality evaluations.

The purpose of this study was to examine the effects of MoBeGo, a mobile self-monitoring app, on the initial and sustained academic engagement and disruptive behavior of third- to eighth-grade students with challenging behavior. Student-teacher pairs (N = 57) were randomly assigned to the treatment (MoBeGo) or control (business-as-usual) condition. We conducted systematic direct observation of students' behavior throughout prebaseline, baseline, intervention, and postintervention conditions of the study. Multivariate multilevel models revealed differential improvement for the MoBeGo group in student outcomes (less disruptive behavior; more academic engagement) from baseline to intervention, as well as successful postintervention effects for disruptive behavior. Limitations, future directions, and implications for practice are discussed.

We describe an activity that introduces school-aged children to clinical trials, that presents the terminology associated with randomized controlled trials, and that reveals how the findings from clinical trials are applicable to everyone everywhere.

Underrepresentation of historically marginalized populations in clinical trials continues to threaten the validity of health intervention research. Evidence supports the merits of intercept and other proactive forms of recruitment for achieving more equitable representation. However, researchers also report lower retention and adherence to protocols among these populations, particularly in longitudinal studies. Few studies have compared recruitment methods for longitudinal randomized trials testing health interventions, with even fewer having done so for trials involving ecological momentary assessment (EMA). As intervention research integrates EMA and other data collection approaches requiring substantial participant effort, it is critical to better understand the effectiveness and implications of strategies to improve the representativeness of health research. This secondary data analysis compared outcomes of proactive and reactive recruitment strategies (mobile lab intercepts and internet/flyer advertising, respectively) in study inclusion, task completion, and retention within a 14-day randomized controlled trial that used EMA to evaluate cigarette package health messages. Proactive recruitment resulted in higher proportions of participants with low income and education, limited health literacy, and of diverse racial/ethnic makeup. However, this recruitment method also resulted in lower task completion, especially in the second week of the trial period, and lower retention, although group differences were not explained by participant sociodemographic characteristics targeted by inclusion efforts. We conclude that proactive recruitment via intercepts is an effective strategy for health intervention research that aims to include stakeholders from historically marginalized groups but that researchers and funders must recognize these methods require additional resources, considerations, and capacity to address non-trivial challenges to successful participation.

Randomized controlled trials (RCTs) in education are common as the design allows for an unbiased estimate of the overall impact of a program. As more RCTs are completed, researchers are also noting that an overall average impact may mask substantial variation across sites or groups of individuals. Mixed methods can provide insight and help in unpacking some of the reasons for these variations in impact. This article contributes to the field of mixed methods research by integrating mixed methods into a recently developed conceptual framework for understanding variations in impact. We model the use of this approach within the context of an RCT for online courses that found differences in impact across courses.

This paper seeks to raise awareness among educational researchers and practitioners of some significant weaknesses and internal contradictions of randomised control trials (RCTs). Although critiques throughout the years from education scholars have pointed to the detrimental effects of this experimental approach on education practice and values, RCTs are considered the "gold standard" for assessing the impact of education policies and interventions. By drawing on the approach of immanent critique, we elucidate substantial argumentative gaps between the assumptions and applications -- that is, between the theory and reality -- of RCTs in empirical research. This kind of analytic exercise complements existing critiques from outside the experimental discourse based on moral and epistemic principles. The present paper, in contrast, contributes to the literature by highlighting internal limitations and contradictions that can be seen by probing the logic espoused by those who are proponents of RCTs. In fleshing out our argument, we seek to encourage more informed and critical engagement by educators, policymakers, and researchers, among other stakeholders, when they are confronted with proposals for education programmes and reforms supported by findings from RCTs.