Evaluating the Effect of Coaching on SAT Scores: A Meta-analysis

SAT coaching studies with and without comparison groups

US high-school students preparing for the SAT

The paper that introduced the DerSimonian-Laird random-effects estimator, applied here to the coaching literature. Its methodological contribution to this archive is a single decomposition: it separates studies by whether they had a control group and shows that this design feature, not the coaching, drives most of the reported variation in effect size. Graded C - the estimate inherits the quality of the pooled primaries.

Uncontrolled studies - no comparison group, just before-and-after - show mean gains of about 40 points verbal and 50 points maths. Studies WITH comparison groups show average coaching effects of about 15 points on each section. The apparent effect is four to five times larger when nobody measured what would have happened anyway. Typical commercial-course effects fall in the 9-25 verbal and 15-25 maths range. This is the cleanest available statement of why the test-preparation industry's own numbers and the research literature's numbers differ so wildly: the industry quotes the uncontrolled figure, which mostly measures ordinary retest gain and maturation.

Medium in the pooled primaries; the controlled subset is the less contaminated cell and is also the one with the smaller effect, which is the direction selection predicts.

The controlled-versus-uncontrolled gap is reproduced by Becker (1990) on an overlapping study set and is consistent with Powers & Rock's finding that uncoached retesters gain 12-16 points on their own.