Methodology Review: Evaluating Person Fit

none - methodology review of person-fit statistics

Item-response-theory and classical-test-theory testing generally

Methodology review, grade D. It is included for one reason - it establishes that the formal machinery for asking "is this response pattern consistent with any ability level?" exists, and it establishes the limits of that machinery honestly, including that most person-fit statistics do not behave as their theory predicts on short tests.

Person-fit methods can detect response patterns that are inconsistent with the measurement model - the formal version of "this score pattern does not look like a person's knowledge." But the caveats are the point. Sound person-fit methods exist for the Rasch model; for other IRT models the empirical and theoretical distributions of the statistics DIFFER on short and moderate-length tests, which describes most subtests on a school achievement battery. Detection rate depends on the type of misfitting pattern, on test length, and on the examinee's trait level. The usefulness of the whole apparatus "depends on the application" - an unusually candid closing assessment that the archive should not paper over.

Not applicable - statistical methodology.

A review of methods rather than a finding; the limits it reports have been restated in the later person-fit literature.