Intelligent tutoring systems and learning outcomes: A meta-analysis

107 effect sizes from 73 reports, 14,321 participants; distribution of the 107 effect sizes M = 0.43, SD = 0.40

All education levels from elementary school to postsecondary, mixed subjects (mathematics and language/literacy both g = 0.35); ITS used as principal instruction, as a supplement, as an integral component of a course, or as a homework aid.

The meta-analysis organised around COMPARISON CONDITION rather than around the technology, which is what makes it useful here: it prices the same software against five different things a student could otherwise have been doing, and the answer changes sign. Pool is not restricted to high-quality experimental designs - it includes experimental, quasi-experimental and pre-experimental studies, which is the standing criticism of it (Leite et al. 2025) and the reason the grade stays C. Attrition was reported in only 21.5% of studies (33.6% explicitly reported none, 44.9% said nothing). Publication bias was assessed by fail-safe N only: the number of null studies needed to overturn the pooled effect exceeds Rosenthal's 5k+10 criterion, so the authors conclude bias is not a significant threat - no funnel plot, trim-and-fill or PET-PEESE. NOT READ IN FULL - APA paywall, no OA copy located (the author's SFU copy, ResearchGate, Academia.edu and Ovid all refused). Numbers below come from the paper's own ERIC/PsycNET abstract plus Kulik & Fletcher (2016, p. 45) and Leite et al. (2025), two independent readers of its tables. THE 95% CONFIDENCE INTERVALS FOR THE COMPARISON- CONDITION CONTRASTS COULD NOT BE RETRIEVED; the significance statements below are the paper's own.

Across 107 comparisons the average ITS effect is g = 0.43, but the number decomposes entirely into what the control group was doing. ITS beat teacher-led LARGE-GROUP instruction by g = 0.42, beat non-ITS computer-based instruction by g = 0.57, and beat textbooks and workbooks by g = 0.35. Against the two conditions that give a student individual or near-individual human attention, the advantage disappears: g = 0.05 vs small-group instruction and g = -0.11 vs individualized human tutoring, neither significant. This is one of the two sources (with VanLehn 2011) behind the claim that intelligent tutoring software is "no worse than a human tutor" - and read precisely, the claim is that a non-significant point estimate favours the human by a tenth of a standard deviation, not that the software matches a tutor. Unusually for this literature, the measure-type moderator was essentially flat here (standardized g = 0.41 vs researcher-developed g = 0.42), which is the sharpest disagreement with Kulik & Fletcher (2016), whose same-moderator gap was 0.13 vs 0.73.

Medium - the pool mixes randomized, quasi-experimental and pre-experimental designs without restricting on design quality, so selection into ITS conditions is not excluded; the human-tutoring contrasts in particular come mostly from small within-course randomizations where the confound risk is low but the samples are self-selected.

The "ITS is statistically indistinguishable from a human tutor" claim is replicated by VanLehn (2011), who found step-based ITS d = 0.76 against human tutoring d = 0.79 - the two sources the claim rests on. It is contradicted in direction by Steenbergen-Hu & Cooper (2014), who found ITS 0.25 SD BELOW human tutoring in college settings, though that gap is also not the two sigma of folklore. The ITS-vs-large-group contrast (g = 0.42) is corroborated by Kulik & Fletcher (2016, weighted g = 0.50 against conventional instruction) but not by the two large-scale K-12 literatures (Steenbergen-Hu & Cooper 2013 ~0.05; Cheung & Slavin 2013 large randomized +0.06). The measure-type null reported here (standardized 0.41 vs researcher-made 0.42) is itself unreplicated and is the outlier finding in this literature.