Search

This article compares and contrasts two versions of the Education Endowment Foundation's (EEF) Teaching and Learning Toolkit ('Toolkit'), a web-based summary of international evidence on teaching 3-18 year-olds. The Toolkit has localised versions in six different languages in Australia, Cameroon, Chile, Jordan and Spain. The initial Toolkit, created in 2011 with funding from the Sutton Trust and updated since then with funding from EEF, drew upon over 250 meta-analyses across 30 areas of education research. An updated version, drawing on a database of over 2500 single studies from these meta-analyses was launched in Autumn 2021. This change was motivated by increased interest in evidence-use in education, and a desire to engage in more rigorous synthesis of primary studies. The article presents the rationale for these changes, outlines the methods adopted to populate and analyse the Toolkit database and presents results from this analysis. Findings indicate that although the broad picture of the relative benefits of the different approaches is similar, a more fine-grained analysis is possible. This deeper synthesis can provide more specific guidance about what has been successful in the different areas of the Toolkit in research studies and offers opportunities for further refinement and improvement. This increased specificity, however, comes at the cost of greater complexity in the findings and the implications for policy and practice, and it increases the challenge of ensuring findings are both accurate and accessible. A final section reflects on the challenges of summarising evidence from research to inform decision-making in education.

In education, multisite trials involve randomization of pupils into intervention and comparison groups within schools. Most analytical models in multisite educational trials ignore that the impact of an intervention may be school dependent. This study investigates the impact of statistical models on the uncertainty associated with an effect size using comparable outcomes and covariates from ten multisite educational trials funded by the UK's Education Endowment Foundation. Ordinary least squares (OLS) models often assume that the pupil's outcomes within schools are independent, which is not always true. Multilevel models address this limitation by incorporating heterogeneity between schools to account for intra-school dependency. This inflates the confidence interval of an effect size obtained from the multilevel models than from an OLS model. For a multisite trial, the heterogeneity between schools also includes the differences in the expected impact of intervention between schools. Ignoring this additional school-by-intervention variation in a multisite trial could affect both its interpretation and conclusions. A robust approach to estimate the confidence intervals for effect size from multisite trials is by treating effect size as a parameter with its distribution. This paper is important for evaluating evidence from multisite trials by accounting for all sources of variability.

Educational stakeholders are keen to know the magnitude and importance of different interventions. However, the way evidence is communicated to support understanding of the effectiveness of an intervention is controversial. Typically studies in education have used the standardised mean difference as a measure of the impact of interventions. This measure, commonly known as the effect size, is problematic, in terms of how it is interpreted and understood. In this study, we propose a "gain index" as an alternative metric for quantifying and communicating the effectiveness of an intervention. This is estimated as the difference in the percentage of children who make positive gains between the intervention and control groups. Analysis of four randomized controlled trials in education supports the expectation that most children make progress due to normal school activities, which is independent of the intervention. This study elaborates a method to illustrate how trials with a positive gain index and with a higher percentage of pupils with positive gain in the intervention group can be used to communicate which trials are effective in improving educational outcomes.

Meta-analysis is the synthesis of findings from research projects, which enables an estimate of the average or pooled effect across various studies. This study presents findings from the intention to treat analysis for a series of educational evaluations in England using a two-stage meta-analysis with standardised outcome data and individual participant data meta-analyses. The research estimates the overall impact of educational trials on pupils eligible for Free School Meals (FSM) and the attainment gap in literacy and mathematics performance between FSM and non-FSM pupils based on analysis of 88 trials and data from over half a million pupils. For the meta-analyses, frequentist and Bayesian multilevel models were used to estimate the individual and pooled effect size across categories of explanatory variables such as age groups (key stages in England) and aspects of the type of interventions (one-to-one, small group, whole class). Results indicated that the overall impact of interventions on the literacy outcomes of FSM pupils was positive, with a pooled effect size of 0.06 (0.03, 0.08). However, for mathematics, no overall effect on FSM pupils was observed. Analysis of the attainment gap indicated that literacy outcomes for FSM pupils were improved by interventions marginally more than for non-FSM pupils (pooled attainment gap 0.01 (-0.01, 0.04)). The risk of bias assessment showed that estimates were consistent across different methodological approaches. Overall, evidence from this study can be used to identify, test and scale educational interventions in schools to improve educational outcomes for disadvantaged pupils.

Educational researchers advocate the use of an effect size and its confidence interval to assess the effectiveness of interventions instead of relying on a p-value, which has been blamed for lack of reproducibility of research findings and the misuse of statistics. The aim of this study is to provide a framework, which can provide direct evidence of whether an intervention works for the study participants in an educational trial as the first step before generalizing evidence to the wider population. A hierarchical Bayesian model was applied to ten cluster and multisite educational trials funded by the Education Endowment Foundation in England, to estimate the effect size and associated credible intervals. The use of posterior probability is proposed as an alternative to p-values as a simple and easily interpretable metric of whether an intervention worked or not. The probability of at least one month's progression or any other appropriate threshold is proposed to use in education outcomes instead of using a threshold of zero to determine a positive impact. The results show that the probability of at least one month's progress ranges from 0.09 for one trial, GraphoGame Rime, to 0.94 for another, the Improving Numeracy and Literacy trial.

AbstractThe effectiveness of educational interventions has traditionally been evaluated using effect size measures which focus on a single feature of the distribution of the outcomes under intervention and control conditions: a (standardized) mean difference. Recently there has been increased interest in methods which assess the information contained in the full distributions of outcomes under intervention and control, providing measures of separation from these distributions which do not depend on arbitrary cutoffs, hence which are “threshold-free”. We investigate the statistical relationship between several concepts of this type, and discuss how they can be used to estimate alternative effect size metrics as well as their uncertainties in the context of multilevel models as commonly used for the analysis of educational data. A specific aim of this paper is to investigate how the recently proposed “gain index” relates to other measures of separation including the Area under the Curve (AUC) and the overlapping index. A simulation study, using data with an educationally motivated structure, is presented to compare the different methodologies. [ABSTRACT FROM AUTHOR]

Background: Poor performance in the 5‐chair stand test (5‐CST) indicates reduced lower limb muscle strength. The 5‐CST has been recommended for use in the initial assessment of sarcopenia, the accelerated loss of muscle strength and mass. In order to facilitate the use of the 5‐CST in sarcopenia assessment, our aims were to (i) describe the prevalence and factors associated with poor performance in the 5‐CST, (ii) examine the relationship between the 5‐CST and gait speed, and (iii) propose a protocol for using the 5‐CST. Methods: The population‐based study Cognitive Function and Ageing Study II recruited people aged 65 years and over from defined geographical localities in Cambridgeshire, Newcastle, and Nottingham. The study collected data for assessment of functional ability during home visits, including the 5‐CST and gait speed. We used multinomial logistic regression to assess the associations between factors including the SARC‐F questionnaire and the category of 5‐CST performance: fast (<12 s), intermediate (12–15 s), slow (>15 s), or unable, with slow/unable classed as poor performance. We reviewed previous studies on the protocol used to carry out the 5‐CST. Results: A total of 7190 participants aged 65+ from the three diverse localities of Cognitive Function and Ageing Study II were included (54.1% female). The proportion of those with poor performance in the 5‐CST increased with age, from 34.3% at age 65–69 to 89.7% at age 90+. Factors independently associated with poor performance included positive responses to the SARC‐F questionnaire, physical inactivity, depression, impaired cognition, and multimorbidity (all P < 0.005). Most people with poor performance also had slow gait speed (57.8%) or were unable to complete the gait speed test (18.4%). We found variation in the 5‐CST protocol used, for example, timing until a participant stood up for the fifth time or until they sat down afterwards. Conclusions: Poor performance in the 5‐CST is increasingly common with age and is associated with a cluster of other factors that characterize risk for poor ageing such as physical inactivity, impaired cognition, and multimorbidity. We recommend a low threshold for performing the 5‐CST in clinical settings and provide a protocol for its use. [ABSTRACT FROM AUTHOR]