The Evidence on Teaching

Systematic reviews of the effects of preparatory courses on university entrance examinations in high school-age students

Montgomery P, Lilly J · 2011

grade Cmeta-analysisindependentmixed
Sample
10 experimental/controlled studies published 1970-2000; treatment-control contrasts pooled for SAT-V and SAT-M
Population
US high-school-age students preparing for the SAT
Design
A Campbell/Cochrane-style systematic review, so the search and screening are more transparent than in the older metas - but it pools the same weak primary literature, and its conclusion diverges sharply from every other synthesis. Recorded at C and recorded deliberately as the DISSENTING high estimate. The archive's job is not to pick the number it prefers: the honest reading is that this literature's upper bound is contested, and a review that finds roughly double what Becker found on overlapping studies is evidence about the fragility of the pooling, not only about coaching.
Key findings
Treatment groups gained 23.5 points on SAT-Verbal and 32.7 points on SAT-Math over controls - which on a scale with SD near 110 is roughly 0.21 and 0.30 SD, about double Becker's 0.09 and 0.16. The authors describe the combined figure as "nearly triple that which was previously assumed" and draw a policy conclusion - that universities should reconsider the weight given to SAT scores because coaching is not equally accessible. They also concede the state of the field: they "call for researchers to increase the production of high-quality data in this field to ensure accurate estimates of coaching's effects." Even at this upper bound, the gains are a fifth to a third of a standard deviation, not the transformation commercial prep advertises.
Genetic confound
Medium. Selection into coaching is the dominant threat in the pooled primaries and is not removed by systematic-review methodology; a review can only be as unconfounded as its inputs.
Replication notes
Directly at odds with Becker (1990) and Powers & Rock (1999) on magnitude while agreeing on direction. Recorded as `mixed` because the disagreement is real and unresolved rather than because either side has been refuted.

Effects

OutcomeMetricValueMeasureTimingVsHorizonClass
SAT-Verbal gain, coached vs controlscaled score points23.5 points (~0.21 SD)standardizedpost-coaching administrationbusiness-as-usualend-of-treatmentdomain-skill
SAT-Math gain, coached vs controlscaled score points32.7 points (~0.30 SD)standardizedpost-coaching administrationbusiness-as-usualend-of-treatmentdomain-skill

Cited by