Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review

50 controlled evaluations (63 independent comparisons) drawn from 550 candidate reports; 35 published, 15 dissertations/technical reports

Mixed K-12 and postsecondary (23 elementary/high-school evaluations, 27 postsecondary); mathematics 18, other subjects 32; 39 US, 11 other. Systems include Cognitive Tutor (15 evaluations), AutoTutor, ALEKS, Wayang Outpost and others.

THE archive's canonical measure-inflation case for ed-tech, because the same 50 studies produce a famous number and a boring one depending only on which test was used. Primary estimator is Glass's ES, unweighted, with evaluation study as the unit and 90% Winsorization of outliers; Hedges's g and weighted random-effects estimates are reported as supplements and are systematically lower (weighted g = 0.50, 95% CI 0.40-0.59). Inclusion rules are strict in ways that matter and are themselves selective: control groups had to receive CONVENTIONAL instruction (studies with human-tutor, CAI-tutor or no-instruction controls were excluded); pretest gaps >= 0.5 SD excluded; overaligned outcome measures excluded; and - critically - "the treatment had to be implemented without major failures", which removes exactly the implementation failures that characterise ordinary school deployment. Only 13 of 50 evaluations randomized; 32 used intact groups. Publication bias was handled only as a published/unpublished moderator (0.67 vs 0.46); no funnel plot, trim-and-fill or PET-PEESE was performed. Fletcher is at the Institute for Defense Analyses and Kulik at Michigan; neither is a vendor, but the paper is written from inside the ITS research community and its discussion argues that studies finding nulls "were weak in design or execution".

The median ITS effect across 50 evaluations is 0.66 SD - the number everyone quotes - and it is almost entirely an artefact of outcome-measure alignment. Test type is the strongest moderator in the paper (r = -.63): local, researcher-developed tests give 0.73 (k = 38), a mix gives 0.45 (k = 3), and independent standardized tests give 0.13 (k = 9; 0.09 under Hedges g homogeneity analysis). A 5-6x gap. Within Cognitive Tutor alone the median is 0.86 on local tests and 0.16 on standardized. The same deflation shows up on every other quality axis: studies with <= 80 participants 0.78 vs > 250 participants 0.30; conventional controls 0.60 vs controls given ITS-derived materials 0.18; strong implementations 0.44 vs weak implementations -0.01. The paper's own K-12 mathematics subgroup on standardized measures only is 0.10 - the same null the large RCT literature reports.

Medium - only 13 of 50 evaluations randomized (mean 0.55) and 32 used intact groups (mean 0.67); pretest gaps up to 0.5 SD were tolerated. Not the main threat here; measure alignment is.

The headline 0.66 is contradicted by Steenbergen-Hu & Cooper (2013), who found ~0.05 on K-12 mathematics; Kulik & Fletcher devote a discussion section to the disagreement and attribute it to definition and control-group standards. It is ALSO contradicted from inside this paper: the same 50 studies give 0.13 on standardized tests, and the paper's own K-12 mathematics subgroup gives 0.10 on standardized-only measures - a number identical to the large randomized literature (Cheung & Slavin 2013 large randomized +0.06; Pane 2014 ~0 to +0.21). The measure-type gap itself is heavily replicated (Cohen/Kulik/Kulik 1982: 0.84 vs 0.27; Cheung & Slavin 2016; Xu 2019 reading-comprehension ITS: researcher-developed measures 0.73 SD larger).