Time and method in coaching for the SAT.
Messick S, Jungeblut A · 1981
grade Cmeta-analysisdeveloper-involvedmixed
Sample
coaching studies published 1953-1980; hours-effect regressions on 19 SAT-V and 14 SAT-M studies
Population
US high-school students preparing for the SAT
Design
Review and quantitative re-analysis by ETS researchers - the developer-involved flag is real, and the direction of the paper's conclusion (coaching is expensive and inefficient) is the direction ETS preferred. Graded C. The most-cited part, the regression of effect on contact time, was subsequently criticised by DerSimonian & Laird and could not be replicated by the Kulik group on a slightly different study pool, so the archive records the dose-response claim as contested rather than established.
Key findings
The dose-response result, and the sentence it produced. Spearman rank correlations between programme contact hours and coaching effect were 0.77 for SAT-V (19 studies) and 0.71 for SAT-M (14 studies), and the relation is LOG-LINEAR: arithmetic score gains require geometric increases in contact time, with returns flattening beyond roughly 45 hours and flattening fastest on verbal. From controlled studies they put the mean coaching effect at about 14 points verbal and 15 maths, with a typical first 20 hours worth about 0.2 SD (15-19 points) on SAT-M. Their conclusion is the one worth keeping: the contact time needed to produce gains much greater than 20 to 30 points per section "rapidly approaches that of full-time schooling." If you want a large score gain, the cheapest way to get it is to go to school.
Genetic confound
Medium. The pooled studies are mostly observational with self-selected coached groups; the hours-effect regression is across studies, so any study-level correlate of duration (including student ability) can drive it.
Replication notes
The hours relation was criticised methodologically by DerSimonian & Laird (1983) and NOT replicated by Kulik, Kulik & Bangert-Drowns on a different study pool. Hausknecht et al. (2007) later found contact time to be a significant positive predictor in a much larger retest meta-analysis, so the underlying pattern has some support even though this specific regression does not.
Effects
| Outcome | Metric | Value | Measure | Timing | Vs | Horizon | Class |
|---|---|---|---|---|---|---|---|
| Correlation between programme contact hours and coaching effect, SAT-V | Spearman rank correlation | 0.77 across 19 studies | standardized | not applicable | business-as-usual | end-of-treatment | domain-skill |
| Correlation between programme contact hours and coaching effect, SAT-M | Spearman rank correlation | 0.71 across 14 studies | standardized | not applicable | business-as-usual | end-of-treatment | domain-skill |
| Mean coaching effect from controlled studies | scaled score points | ~14 points verbal, ~15 points maths | standardized | post-coaching administration | business-as-usual | end-of-treatment | domain-skill |
| Contact time required for gains much greater than 20-30 points per section | time | "rapidly approaches that of full-time schooling" | standardized | not applicable | business-as-usual | end-of-treatment | domain-skill |
Cited by
- Test preparation — does coaching raise scores, and does a raised score mean raised ability?mixedconf: mediumgc: medium