Using Digital Technology to Improve Learning

Four recommendations drawn from two commissioned evidence reviews plus practitioner consultation; the quantitative content is a tabulation of 8 EEF-funded evaluations (ABRA, Accelerated Reader, Affordable Maths Tuition, GraphoGame Rime, Learner Response System, Mathematical Reasoning efficacy and effectiveness, MathsFlip, Texting Parents), each individually a school-randomised trial of several hundred to several thousand English pupils.

English primary and secondary school pupils, roughly Key Stage 1 to Key Stage 4 (ages 5-16) - the only source in this cluster inside the archive's age band from start to finish, and the only one drawn from a funder that commissions independent evaluation as a matter of policy. The trials cover reading, phonics, mathematics and parental engagement.

WHY THIS IS IN AN ONLINE-AS-SUBSTITUTE CLUSTER: the report is mostly about technology as a SUPPLEMENT to classroom teaching, which is a different question - but it states the substitution answer directly and it is the only independent-evaluator base rate available. Its operative sentence is "To date, technology has been most effective when it is used to supplement or enhance teaching, rather than to replace it", and it observes that in the studies with the largest recorded impacts the technology supplies ADDITIONAL resources and ADDITIONAL learning time - i.e. the gains come from more instruction, not from a better delivery medium. WHAT THE RECOMMENDATIONS ARE: (1) consider how technology is going to improve teaching and learning before introducing it; (2) technology can be used to improve the quality of explanations and modelling; (3) technology offers ways to improve the impact of pupil practice; (4) technology can play a role in improving assessment and feedback. On their own these are expert consensus over evidence reviews - grade D content. The report is graded C on the strength of Table 1, whose rows are independently evaluated RCTs, not on the recommendations. LIMITS OF TABLE 1 AS A BASE RATE: it is explicitly a curated selection ("the most relevant results published by December 2018 that had an EEF security rating of at least 3 padlocks"), not the full EEF ed-tech portfolio, so it is not a clean denominator and is if anything selected toward the interesting. The unit is EEF "months' additional progress", a coarse conversion from standardised effect sizes that the archive dislikes; the underlying evaluation reports carry the effect sizes and confidence intervals. THE TWO ROWS THAT BEAR DIRECTLY ON SUBSTITUTION: Affordable Maths Tuition delivered one-to-one maths tuition over the internet from trained maths graduates in India and Sri Lanka - a genuine attempt to substitute remote delivery for an in-person tutor - and returned no evidence of impact on Key Stage 2 mathematics. And the ABRA trial ran the medium-versus-content test the rest of the literature never runs - the same literacy content was delivered online and on paper (matched activity for activity, same stories, vocabulary, questions, letter sounds) against a business-as-usual control; both versions beat the control and neither beat the other, which locates the active ingredient in the content and not the screen. DISPLACEMENT is flagged twice and explicitly, which is rare - the GraphoGame Rime null is annotated with the observation that the control group was receiving other literacy support (small-group and one-to-one), so the comparison was against an active alternative rather than nothing; and the report warns that recommending an online quiz site can cause pupils to cut back other revision. INDEPENDENCE: the EEF funds and commissions the trials but contracts independent evaluators (NFER, universities) to run and report them; developers supply the product and training. Independent under the archive's rule.

Across eight independently evaluated EEF-funded ed-tech trials in English schools the modal result is a few weeks' progress and the median is about one month - specifically +2 (ABRA online), +3 (Accelerated Reader), 0 (Affordable Maths Tuition), -1 (GraphoGame Rime), 0 (Learner Response System), +1 effectiveness / +3 efficacy (Mathematical Reasoning), +1 (MathsFlip), +1 (Texting Parents). That distribution is the closest thing available to an independent-evaluator base rate for buying ed-tech, and it is roughly "a month, sometimes nothing, occasionally negative". Two rows carry the substitution question. Affordable Maths Tuition - online one-to-one maths tutoring delivered remotely by graduates in India and Sri Lanka - found no evidence of impact on Key Stage 2 mathematics. And ABRA compared the SAME literacy content delivered online and on paper: both beat business-as-usual, neither beat the other. The report's own conclusion is the cluster's conclusion in miniature: technology has worked as a supplement or enhancement, not as a replacement, and the best judgment of a new technology is not "has it had an impact?" but "has it had a greater impact than the alternative?"

Low for the tabulated trials - each row is a randomised evaluation with a business-as-usual or active comparison, so heritable differences cannot differ systematically between arms. The four recommendations themselves are consensus judgments and carry no causal identification of their own.

A guidance report is not a finding and cannot itself be replicated. What it carries that IS replicable is Table 1 - eight EEF-funded evaluations of ed-tech products in English schools, each an independently commissioned and independently conducted RCT with a published evaluation report, filtered to those with an EEF security rating of at least three padlocks out of five and published by December 2018. The one internal replication in the table runs the archive's own way - Mathematical Reasoning returned +3 months in its efficacy trial and +1 month when scaled up in an effectiveness trial.