The Evidence on Teaching

The Flynn effect and U.S. policies: the impact of rising IQ scores on American society via mental retardation diagnoses.

Kanaya T, Scullin MH, Ceci SJ · 2003

grade Clongitudinalindependentreplicated
Sample
longitudinal IQ records from 9 sites around the United States
Population
US school-age children in the borderline and mild intellectual-disability range, retested across a test renorming
Design
Longitudinal records study exploiting a naturally occurring event - the periodic renorming of IQ tests - as the manipulation. Assignment to old-norm or new-norm testing is a function of when the child happened to be retested, not of the child, which is what gives the comparison its force. Not randomised, and the sample is the low tail rather than the general population, so the 5.6-point figure should not be read as the effect at every ability level.
Key findings
The same child, on the same construct, scores an average of 5.6 IQ POINTS LOWER when retested on a renormed version of a test - and was correspondingly more likely to be classified as intellectually disabled than a peer retested on the same old-norm test. Nothing about the children changed; the reference group did. This is the cleanest empirical demonstration that a norm-referenced score is a statement about a comparison group and not about a person, and the consequences here were not academic - the authors note the effect reaches eligibility for education services, social security, the military, and death-penalty eligibility. For an archive that will read percentiles off commercially normed batteries, the implication is that the AGE of the norms is part of the score.
Genetic confound
Not applicable in the usual direction. The finding is that a score moved without any change in the child, which is evidence AGAINST reading the score as a fixed trait measure.
Replication notes
The Flynn effect itself is one of the most replicated findings in psychometrics; the specific classification consequence has been reproduced in subsequent policy analyses by the same group and others.

Effects

OutcomeMetricValueMeasureTimingVsHorizonClass
IQ score change on retest with a renormed testscore points-5.6 points on average, borderline and mild ID rangestandardizedretest across a renormingactive-alternativeunclearg
Probability of classification as intellectually disabledcomparisonhigher when retested on a renormed test than on the same old-norm teststandardizedretestactive-alternativeunclearg

Cited by