Thinking, Doing, Talking Science: Evaluation report and executive summary

205 schools randomised (8,966 pupils); 7,806 Year 5 pupils in the primary analysis (3,843 intervention, 3,963 control)

Year 5 (age 9-10) pupils in English primary schools, September 2016 to June 2017; recruitment deliberately skewed towards above-average-FSM schools.

EFFECTIVENESS trial — the same programme as the 2015 efficacy trial, run at five times the scale by INDEPENDENT evaluators (American Institutes for Research). Two-arm cluster RCT with a waitlist control; 233 Year 5 teachers in 102 intervention schools received the training. Three deliberate changes on scale-up, all in the direction of realism: CPD cut from five days to four; the two funded in-school preparation days were removed; and delivery moved to a TRAIN-THE-TRAINER model in which pairs of trainers taught by the programme authors delivered the CPD (all of them for the first time) rather than the authors teaching teachers directly. Trainers, not the developers, also did school recruitment. CRITICALLY FOR MEASURE-TYPE ANALYSIS, THE OUTCOME TEST DID NOT CHANGE: AIR obtained permission to use the same study-delivered science assessment built by the IEE team for the efficacy trial, drawn from the same 2001 Russell and McGuigan item bank, raw score range 0-41. What did change is who handled it — administration moved to NatCen and scoring to independent scorers, whereas in the efficacy trial participating teachers administered it. 11% of pupils were missing outcome or baseline data. EEF security: HIGH. Cost about GBP 29 per pupil per year over three years.

The efficacy result did not survive. Science attainment ES = +0.01 (95% CI -0.08 to 0.10), p=0.791, n=7,806 — zero months' progress, against +0.22 and three months in the developer-run efficacy trial. FSM pupils +0.05 (-0.07 to 0.18), p=0.386, n=2,000 — one month, not significant. Secondary outcomes did move: interest in science +0.12 (0.02 to 0.22), p=0.015, and self-efficacy for science +0.09; teachers in treatment schools were 19 percentage points more likely to report confidence in adapting teaching to pupils' interests and 30 points more likely to report confidence in challenging high achievers. So the programme changed practice and attitudes and did not change learning. Because the outcome instrument was IDENTICAL across the two trials, the collapse from +0.22 to +0.01 CANNOT be attributed to measure type — it is pure efficacy-to-effectiveness decay driven by scale, a diluted training chain, and independent administration.

Low. Large cluster RCT, independent evaluation, balanced prior attainment, 11% missingness.

This is the replication that failed the 2015 efficacy trial (Hanley, Slavin & Elliott). It was itself replicated by the 2025 York Trials Unit trial (180 schools, g=0.02) after the developers strengthened the train-the-trainer model — the null reproduced. EEF's scale-up synthesis (Demack et al. 2021) lists TDTS efficacy 0.22 vs effectiveness 0.01 among seven interventions whose mean effect fell from 0.25 to 0.01.