The Evidence on Teaching

Rates of below-chance performance in forced-choice symptom validity tests.

Greve KW, Binder LM, Bianchini KJ · 2009

grade Clongitudinalindependentreplicated
Sample
1,032 examinees, three symptom-validity tests (PDRT, TOMM, WMT)
Population
Private-practice forensic neuropsychology referrals - alleged mild TBI, moderate-severe TBI, alleged toxic exposure, chronic pain. Adults, not children, and a maximally incentivised sample.
Design
Consecutive-case series reporting base rates, not a designed comparison. Its value here is one number that is otherwise unobtainable: how often a genuinely below-chance result occurs, even in the population most motivated to produce one. Two boundaries on transfer: these are adults with financial incentive to underperform, and the tests are two-alternative forced choice, where chance is 50% and the guessing distribution is far wider relative to the scale than on a four-option achievement subtest.
Key findings
Significantly below-chance performance is RARE even where it is most expected. Rates differ by instrument - PDRT and WMT are equivalent and both yield below-chance results more often than the TOMM - and the harder sections of each test yield more below-chance results than the easier ones, which is what a difficulty-graded guessing model predicts. Using multiple validity tests yields below-chance results more often than any single test, so a single subtest can only ever raise the question, never settle it.
Genetic confound
Not applicable - base rates of a response pattern, no ability comparison and no causal claim about a child.
Replication notes
Three instruments within one sample give an internal cross-check, and the general finding that below-chance responding is rare relative to overall validity-test failure is consistent across the performance-validity literature.

Effects

OutcomeMetricValueMeasureTimingVsHorizonClass
Below-chance performance on forced-choice validity testsbase rate, three instrumentsrare in absolute terms; PDRT = WMT > TOMMstandardizedsingle administrationnonenot-applicabledomain-skill
Yield of harder vs easier test sectionscomparisonseemingly harder sections yield more below-chance resultsstandardizedsingle administrationnonenot-applicabledomain-skill
Single vs multiple validity testscomparisonmultiple tests more likely to detect a below-chance result than one test alonestandardizedsingle administrationnonenot-applicabledomain-skill

Cited by