Search

This document summarises key findings from the quantitative strands of a review of the Education Endowment Foundation (EEF) evaluations that had reported from the establishment of EEF in 2011 up to January 2019. The quantitative strands summarised include meta-analyses of effect sizes reported for attainment outcomes and descriptive analyses of cost-effectiveness and attrition. Additionally, descriptive univariate analyses of the explanatory variables used in the review are summarised. Complete findings can be found in the main report. The review conducted qualitative interviews to explore perceptions on effective scale-up and developed and piloted an IPE quality measure, and these are reported separately. [For "Review of EEF Projects. Evaluation Report," see ED625480. For "Review of EEF Projects. Technical Annex," see ED625484. For "Review: Scale-Up of EEF Efficacy Trials to Effectiveness Trials," see ED625479. For "Review: EEF Implementation and Process Evaluation (IPE) Quality Pilot," see ED625478.]

This report presents findings from exploratory, descriptive meta-analyses of effect sizes reported by the first 82 EEF evaluations that used a randomised controlled trial (RCT) or clustered RCT impact evaluation design published up to January 2019. The review used a theoretical framework derived from literature with five overarching themes to group explanatory variables: (1) Intervention; (2) Implementation & fidelity; (3) Theory & evidence; (4) Context; and (5) Evaluation design. Meta-analyses of effect sizes reported for intention-to-treat (ITT) analyses of primary attainment outcomes, ITT analyses of secondary attainment outcomes and free school meals (FSM) subsample analyses of primary or secondary attainment outcomes are reported. Effect sizes reported for psychological outcomes were also examined but not included in the meta-analyses because of the distinct and diverse nature of these outcomes. Also presented are findings from trial/evaluation-level descriptive analyses of cost effectiveness of interventions and overall pupil-level attrition. [For "Review of EEF Projects. Summary of Key Findings," see ED625481. For "Review of EEF Projects. Technical Annex," see ED625484. For "Review: Scale-Up of EEF Efficacy Trials to Effectiveness Trials," see ED625479. For "Review: EEF Implementation and Process Evaluation (IPE) Quality Pilot," see ED625478.]

The aim of this review was to support the Education Endowment Foundation (EEF), other funders, and developers and deliverers scaling-up interventions by undertaking, for the seven effectiveness trials that had been completed by 2019: (1) a quantitative comparison of effect sizes at the efficiency and effectiveness stages; and (2) a qualitative analysis of interviews with developers and deliverers who have been engaged in scale-up from an EEF efficacy trial to an effectiveness trial, to identify any models, supporting structures, and programme features that either support or hinder intervention scale-up to large numbers of schools. The findings indicate that the there is scope for a more systematic and evidence-based approach to supporting developers to scale-up their interventions that includes greater clarity around what is necessary to reduce or change to implement at scale, builds on evidence on effective approaches to training trainers and CPD for direct implementers, and highlights the need for robust quality assurance of training and implementation. Funders may also wish to consider in more depth how an 'effectiveness trial' should be defined, including whether it is appropriate to change the target group. [For "Review of EEF Projects. Evaluation Report," see ED625480. For "Review: EEF Implementation and Process Evaluation (IPE) Quality Pilot," see ED625478.]

The Education Endowment Foundation (EEF)'s mission is to break the link between family income and educational achievement. This is achieved through summarising the best available evidence in plain language, generating new evidence of 'what works' to improve teaching and learning, and supporting teachers and school leaders to use research evidence to maximise the benefit for young people. Evidence of what works to raise attainment is primarily generated through randomised controlled trials commissioned by the EEF. These trials include a quantitative impact evaluation to estimate the effect on pupils' attainment (and, in some cases, other outcomes) and an implementation and process evaluation (IPE). This report presents an IPE quality measure and pilots it using data coded in the Review of EEF Projects (Demack et al., 2021) from the 79 EEF trial reports that incorporated IPEs and had been published up to January 2019. [For "Review of EEF Projects. Evaluation Report," see ED625480. For "Review: Scale-Up of EEF Efficacy Trials to Effectiveness Trials," see ED625479.]

ThinkForward is a coaching programme, developed by Impetus Private Equity Foundation, which is designed to support secondary school pupils. ThinkForward is targeted at pupils who have been identified as being at high risk of not being in education, employment or training (NEET) following the completion of compulsory education. Coaches are assigned to schools and work with selected pupils as they progress through Key Stage 4 (GCSEs), with the aim of supporting them to make a successful transition into adulthood. The programme provides targeted support tailored to pupils' needs through one-to-one and group work. The programme usually works with young people aged 14-19, though this pilot focused only on the intervention up to age 16, prior to pupils leaving school. The pilot involved Year 10 and 11 pupils in four London secondary schools, beginning in January 2014. The Y11 pupils received up to six months of ThinkForward until their GCSEs in summer 2014. The Y10 pupils received up to 18 months of ThinkForward up until their GCSEs in summer 2015. The pilot was evaluated using a range of qualitative methods and a small Randomised Controlled Trial (RCT). The aims of the evaluation were to determine whether a future large-scale evaluation was possible, and to gain an initial estimate of the programme's impact on GCSE scores, the likelihood of continuing into post-compulsory education, and decreasing pupil absences. Randomisation took place at both school and pupil level. Within the two randomly assigned intervention schools, eligible pupils were randomly allocated either to an intervention group or a within-school control group. Across the four schools, there were 181 pupils in the Y11 cohort, 40 of whom received the intervention, and 160 pupils in the Y10 cohort, 37 of whom received the intervention. Key conclusions include: (1) The approach is not ready for a large school level randomised controlled trial; (2) Before a further trial, it would be necessary to solve data collection issues and to develop an approach that allows the intervention to be targeted at pupils at risk of becoming NEET, while conducting a rigorous randomised controlled trial; (3) Future trials should not use pupil-level randomisation because it is likely that the intervention has 'spillover' effects to other pupils; (4) The trial did not find evidence that the programme had an impact on GCSE scores, absences, or attitudes. This was a small pilot RCT so these findings have low security; and (5) Teachers reported benefits, particularly during the later months of the pilot, where they observed impacts on the intervention pupils' behaviour.

This report evaluates a developmental project designed by School 21 and the University of Cambridge to improve Year 7 students' oracy skills. The project involved developing an Oracy Skills Framework, which sets out the physical, linguistic, cognitive, and social-emotional oracy skills required by students for education and life. Other components which were informed by this framework are: (1) a dedicated Year 7 oracy curriculum comprising weekly oracy lessons; (2) oracy in every lesson; (3) building a whole school oracy culture; and (4) an Oracy Assessment Toolkit. These components were piloted and further developed with Year 7 students within School 21 from September 2013 to July 2014. During the final stages of the project the components were brought together to create an 'Oracy Curriculum, Culture and Assessment Toolkit' that can be adopted by other schools and a website was created, Voice 21 (http://voice21.org/), containing guidance and resources for schools using the Toolkit. This report focuses on an evaluation of: (1) The approaches and materials which formed the Oracy Curriculum, Culture and Assessment Toolkit, including an indicative impact finding on the impact on Year 7 pupils in School 21; and (2) What further development of the Oracy Curriculum, Culture and Assessment Toolkit is needed and would enable a more robust evaluation of its impact.

The Summer Active Reading Programme aimed to improve reading skills and particularly comprehension by raising children's engagement in, and enjoyment of, reading at the transition from primary school to secondary school. The programme was delivered by Booktrust, an independent charity that aims to change lives through engaging people with reading. Participating pupils were gifted four book packs and invited to attend two summer events led by Booktrust staff at their new secondary school. The first book pack was gifted towards the end of the child's final term at primary school, the second and third packs at the summer events and the final pack in the first term of secondary school. Volunteers, recruited by Booktrust, gifted the book packs and supported activities, including one to one reading, at the summer events. The trial examined the impact of the programme on 205 pupils from 10 schools in the north of England who had been identified as unlikely to achieve Level 4a or above by the end of Key Stage 2. Pupils who were not likely to gain at least Level 2 were not included in the trial. The study was funded by the Education Endowment Foundation as one of 24 projects in a themed round on literacy catch-up at the primary-secondary transition. Projects funded within this round aimed to identify effective ways to support pupils not achieving Level 4 in English at the end of Key Stage 2. The project was one of four funded with a particular focus on reading for pleasure. Among the key conclusions were: (1) On average, pupils who participated in the programme made slightly more progress in reading comprehension than similar pupils who did not; The evaluation was the first randomised controlled trial of a book-gifting programme in England that included a primary outcome measure of attainment, but problems with recruiting schools reduced the sensitivity of the evaluation and its potential to identify impact; and (3) A positive impact on enjoyment of reading was detected for pupils not in receipt of free school meals. However, in terms of attainment this group improved slightly less than those eligible for free school meals.

The "TextNow" Transition Programme, developed by national charity Unitas, is a reading for pleasure literacy programme with one to one sessions with a volunteer coach occurring 20 minutes a day each weekday. This occurs at two time points: in Year Six, the final year of primary school for five weeks in the second half of the summer term; and at the beginning of Year 7, the first year of secondary school, for ten weeks. The role of the coach is to support the three "TextNow" core principals of 'choose, enjoy and understand'. Coaches are usually: teachers; teaching assistants; non-teaching staff such as technicians or administrative staff; parents; university students; community volunteers; or older year group pupils (usually Years 10-13). A TextNow programme coordinator in each school recruits and supports the coaches and manages the administration of the programme. Coordinators are usually teachers, teaching assistants or school librarians. Schools are free to timetable the sessions whenever suits them best, for example through withdrawal from lessons, during registration or tutor periods, or in breaks, at lunchtime or after school. The programme also includes access to My Choice!, an online resource that has news, quizzes and competitions to engage young people in activities to do with reading, as well as giving them access to a book store. Participants are given credits to choose books from MyChoice! for summer holiday reading at the beginning of the project. They are able to earn further credits for their attendance and engagement with the coaching sessions during the five-week programme in Year 6 and the ten-week programme in Year 7: these can be used to 'buy' books for reading at home and to build their own personal library. This trial is the first time that the TextNow Transition Programme has been delivered. It was adapted from the Unitas TextNow programme that targets young people in their middle teenage years. TextNow was originally designed to meet the needs of young people in the youth justice system that were not in education, training or employment (NEET) and struggling with poor reading skills. Gradually TextNow was made available to the education sector, initially focusing on alternative education providers, but increasingly being used in mainstream secondary schools from 2010, when it was provided as a ten-week programme. Key conclusions include: (1) The trial has not provided any evidence that the TextNow Transition Programme improved reading comprehension or attitudes towards reading for pleasure over the transition from primary to secondary school; (2) On average, pupils who participated in the programme made slightly less progress than similar pupils who did not. However, this finding was not statistically significant, meaning that it could have occurred by chance; (3) The programme was found to have a differential effect for pupils eligible for free school meals compared to their peers. A small positive (but not significant) effect was found for pupils eligible for free school meals, while a negative (and statistically significant) effect was detected for pupils not eligible for free school meals. It is unclear why this differential effect was found; (4) Higher attendance at the 20-minute daily coaching sessions was found to have a positive impact on reading comprehension. However, this was only found to be statistically significant for attendance at sessions in secondary schools. Attendance of the coaching sessions was not found to have an impact on the secondary outcomes (liking reading and motivation to read); and (5) The programme appeared to be more effective when coaches were highly trained, enthusiastic and committed, and when secondary schools worked closely with feeder primaries to coordinate all elements of the programme. The evaluation results indicated that the Unitas TextNow Transition programme was having no significant effects on pupil outcomes i.e., either their reading ability or attitudes towards reading for pleasure.