Do intelligent tutoring systems benefit K-12 students? A meta-analysis and evaluation of heterogeneity of treatment effects in the U.S.

18 studies, 77 effect sizes, 11 different ITS; US K-12 schools only

US elementary, middle and high schools; mathematics (22 effect sizes) and reading/writing (55 effect sizes). Restricted to the US so that policy and schooling context are held roughly fixed.

Added to the archive because it is the only meta-analysis of this literature restricted to US K-12 school studies, and because it codes the two moderators that decide the answer - outcome measure type and measurement timing - in the same model. Multivariate multilevel meta-regression with random effects of studies and fixed effects of ITS name, plus a MetaForest (random-forest) moderator-importance analysis over 15 coded moderators (cross-validated R-squared 0.25). Internal validity is assessed against What Works Clearinghouse standards: only 8 of 18 studies reported differential attrition at all (5 meet WWC standards without reservations, 3 with reservations, and the status of the other 10 is unknown), which is a fair description of the state of this literature rather than a defect of the review. Publication bias examined by funnel plot only - the authors report a cluster of precise estimates between 0 and 0.5 and spread outside the funnel that they attribute to genuine heterogeneity; no trim-and-fill or PET-PEESE. Grade C: mixed-quality pool, small k, and preprint status.

Across 18 US K-12 studies and 77 effect sizes, intelligent tutoring systems move learning g = 0.271 (SE 0.011, p = .001) - smaller than most prior ITS meta-analyses and, the authors note, the first estimate restricted to US K-12 studies screened against WWC standards. Two moderators do the work the archive cares about. Outcome measure: standardized tests g = 0.200 (95% CI 0.166 to 0.234, k = 61) versus researcher-developed tests g = 0.483 (95% CI -0.154 to 1.120, k = 16, p = .098) - a 2.4x gap, in the same direction as every other meta in this cluster except Ma et al. (2014). Measurement timing: immediately after the intervention g = 0.460 versus at the end of the school year g = 0.155, both significant - the effect loses two-thirds of its size in the months between the posttest and the year-end test. Duration runs the same way: up to two weeks 0.334, two weeks to three months 0.449, three to six months 0.225, more than six months 0.110 and not significant. After-school use produced essentially nothing (g = 0.014).

Medium - the pool is mixed randomized and quasi-experimental, and only 8 of 18 studies reported differential attrition, so differential dropout between arms is unassessed in more than half the pool.

This is the study that adjudicates the cluster's central disagreement, and it lands between the camps while confirming the mechanism both sides argued about. Its standardized-measure estimate (g = 0.200) is close to Kulik & Fletcher's standardized-measure figure (0.13) and to Cheung & Slavin's large randomized figure (+0.06 to +0.12); its researcher-developed-measure estimate (0.483) reproduces the alignment inflation Kulik & Fletcher, Cohen/Kulik/Kulik (1982) and Xu et al. (2019) all report, and directly contradicts Ma et al. (2014), who found no gap. Its end-of-school-year estimate (0.155) versus immediate-posttest estimate (0.460) reproduces the archive's standard fadeout signature. NOT peer-reviewed: presented at AERA 2024 and posted to arXiv in November 2025; as of July 2026 no journal version was located.