Effects of Coaching on SAT I: Reasoning Test Scores
Powers DE, Rock DA · 1999
grade Cquasi-experimentdeveloper-involvedreplicated
Sample
4,200+ examinees from a nationally representative survey of 1995-96 SAT takers; about 500 had attended formal outside coaching programmes
Population
US 11th and 12th graders taking the SAT, national stratified random sample
Design
Observational with a serious attack on the selection problem: several independent analytical methods were applied to the same data (propensity matching, instrumental variables, ANCOVA, repeated measures, Belson and Heckman models), which is the right response when no single method is credible on its own. The estimates differed by method, which is honest and is itself the finding. The independence flag matters - the survey was College Board sponsored and the authors are ETS, so the organisation with an interest in the SAT being uncoachable produced the estimate. That cuts the opposite way from most conflicts in this archive: it is a reason to treat the LOW estimate with caution, not the high one.
Key findings
Coaching moves SAT scores by a small amount. Across five estimation models the effect ranged 6-12 points on SAT-V and 13-26 points on SAT-M (21-34 combined), with preferred estimates of about 8 verbal and 18 maths - roughly 0.07 and 0.16 SD on a scale with SD near 110. The single most useful comparison in the paper is the counterfactual: UNCOACHED juniors retesting as seniors gained about 12 verbal and 16 maths on their own, so a coached student's headline "gain" is mostly the gain they would have had anyway. Commercial companies at the time advertised 100-140 combined points. The authors' summary is the quotable one: all estimates "suggested that the effects of coaching are far less than is claimed by major commercial test preparation companies", and the revised SAT is no more coachable than its predecessor.
Genetic confound
Medium and unavoidable. Students who buy coaching differ systematically - in family income, parental education, and academic push - from those who do not, and every one of those correlates with heritable ability. The multi-method design bounds the bias rather than removing it; the true effect is plausibly at or below the low end of the reported range.
Replication notes
The direction and rough magnitude are reproduced by Becker's meta-analysis (1990) and by Briggs's independent NELS:88 analysis; the disagreement in the literature is about the upper bound (see Montgomery & Lilly 2011), not about whether commercial claims of 100+ point gains hold up.
Effects
| Outcome | Metric | Value | Measure | Timing | Vs | Horizon | Class |
|---|---|---|---|---|---|---|---|
| SAT-V score gain from formal coaching | scaled score points (200-800) | 6-12 across five models; preferred ~8 points (~0.07 SD) | standardized | post-coaching SAT administration | business-as-usual | end-of-treatment | domain-skill |
| SAT-M score gain from formal coaching | scaled score points (200-800) | 13-26 across five models; preferred ~18 points (~0.16 SD) | standardized | post-coaching SAT administration | business-as-usual | end-of-treatment | domain-skill |
| Gain of UNCOACHED students retesting junior to senior year | scaled score points | ~12 verbal, ~16 maths with no coaching at all | standardized | retest one year later | none | under-1yr | domain-skill |
| Coaching effect vs commercial advertising claims | comparison | measured 21-34 combined against advertised 100-140 combined | standardized | not applicable | none | not-applicable | domain-skill |
Cited by
- Test preparation — does coaching raise scores, and does a raised score mean raised ability?mixedconf: mediumgc: medium