Can algorithm-based feedback help students to write better? A meta-analysis exploring surface- and deep-level outcomes

33 studies; 49 comparisons at posttest

Secondary-school and university students. No primary-school studies.

Meta-analysis of automated/algorithm-based writing feedback (automated writing evaluation, grammar checkers and similar), keeping surface-level outcomes separate from deep-level ones and - crucially, and unusually for this literature - reporting MAINTENANCE effects as well as posttest effects. That single design choice produces the most important number in the whole software-and-conventions literature. Numbers taken from the published abstract and reported summary, not the full text.

Overall g = 0.36. Surface-level outcomes (grammar, spelling, mechanics): g = 0.31 at POSTTEST but g = -0.02 at MAINTENANCE. Deep-level outcomes ran the other way: 0.31 at posttest rising to 0.54 at maintenance. Eighteen of 49 comparisons were negative at posttest. The signature - surface gains that evaporate once the tool is withdrawn, deep gains that consolidate - is what performance support looks like rather than skill acquisition, and it is exactly the pattern a conventions topic has to worry about: a child using a grammar checker produces more conventional text while using it and is no more conventional afterwards.

Low. Pooled experimental and quasi-experimental comparisons in which students are assigned to feedback conditions.