Limitations in the Use of Achievement Tests as Measures of Educators' Productivity

synthesis of score-inflation studies, principally a third-grade district study (Koretz, Linn, Dunbar & Shepard 1991) and the Kentucky KIRIS evaluation (Koretz & Barron 1998)

US K-12 students under high-stakes testing regimes

A synthesis rather than new estimation, so the grade reflects the underlying studies rather than this paper's own design. The two underlying studies are what carry it, and both use the right instrument - an independent audit measure administered to the same students.

Two demonstrations that generalise past accountability policy and straight onto any repeated test. First, the SAWTOOTH: in a district that changed tests, scores rose sharply on each newly adopted test for several years; when the OLD and now-unfamiliar test was re-administered in 1990, performance fell back to roughly the first-year baseline. Years of apparent learning were specific to the form. Second, KENTUCKY: KIRIS maths gains ran three to four times the state's NAEP gains, and Kentucky's NAEP gains were no better than other states'; high-school KIRIS maths and reading gains were not reflected in the ACT AT ALL; KIRIS science gains were accompanied by ACT science gains only one-fifth as large. Schools with the biggest KIRIS gains also showed the biggest discrepancies between new and reused items - the signature of narrow coaching on past forms. The general rule the archive should carry: score gains on the test you practised for routinely fail to appear on any independent measure of the same domain.

Not applicable. Comparisons are within the same student populations across measures.

The sawtooth and audit-divergence findings have been reproduced in Chicago (Jacob 2005), Texas (Klein et al. 2000) and across the NAEP-versus-state-test literature; this is a settled empirical regularity.