Score gains on g-loaded tests: No g
te Nijenhuis J, van Vianen AEM, van der Flier H · 2007
grade Cmeta-analysisindependentreplicated
Sample
64 test-retest studies, N = 26,990
Population
Pooled test-retest, coaching and training samples on cognitive-ability batteries; mixed ages including school-age
Design
Meta-analysis using the method of correlated vectors - correlating each subtest's g-loading with the score gain observed on that subtest. Graded C because the method is contested (it is sensitive to artifact corrections and to the number of subtests) and because it pools primary studies of mixed quality. The direction of the result, however, has been reproduced across independent datasets and is corroborated by an entirely different design (Lievens et al. 2007, which tests criterion validity directly rather than g-loadings).
Key findings
The correlation between how strongly a subtest loads on g and how much scores rise on that subtest with practice, coaching or training is approximately -1.00 after artifact corrections. The subtests that gain most are the ones that measure the construct least. The authors' word for the gains is "hollow", and they report that practice and coaching actively REDUCE a test's g-loadedness - the test measures less of what it was built to measure after preparation than before. For this archive this is the mechanism behind the whole test-preparation verdict: a coached score is not a proportionally smaller version of a real gain, it is a gain in a different thing.
Genetic confound
Low relevance in the usual sense - the finding is about what changes rather than about who differs. It does bear on the archive's premise - it is a direct measurement-level demonstration that training moves test performance without moving g, matching Ritchie, Bates & Deary (2015) on schooling.
Replication notes
The method of correlated vectors has been criticised (notably by Ashton & Lee and by Dolan), so the exact magnitude should not be over-read; the near-perfect negative direction has been reproduced across several independent gain datasets and is convergent with criterion-validity evidence obtained by a different route.
Effects
| Outcome | Metric | Value | Measure | Timing | Vs | Horizon | Class |
|---|---|---|---|---|---|---|---|
| Relation between subtest g-loading and score gain | correlation (method of correlated vectors, artifact-corrected) | r ~ -1.00 | standardized | retest after practice, coaching or training | none | not-applicable | g |
| Effect of practice and coaching on a test's g-loadedness | direction | reduces the test's g-loadedness | standardized | retest | none | not-applicable | g |
Cited by
- Test preparation — does coaching raise scores, and does a raised score mean raised ability?mixedconf: mediumgc: medium