A critical review of the refutation text literature: Methodological confounds, theoretical problems, and possible solutions

Critical review of four decades of refutation-text research

The refutation-text research literature itself (science, health, psychology and history misconceptions; predominantly undergraduate samples).

An adversarial review, not an evidence source - it produces no effect size and grades D on the design hierarchy, but it is the correct instrument for the question "does conceptual-change instruction move the concept or move the test?" because that is exactly what it interrogates. Read at abstract depth (ERIC record); full text paywalled at Taylor & Francis, with only a Figshare metadata deposit open.

Argues that the ability of a refutation text to change a learner's mind "needs to be qualified and modulated", and names the confounds. METHODOLOGICAL: (a) most studies use a SINGLE topic or very few, so the effect is confounded with the particular misconception chosen and cannot be separated from item idiosyncrasy; (b) the role of TESTING itself in producing the apparent conceptual change is not controlled - pretest, immediate post-test and delayed post-test on the same bespoke items is a retrieval-practice schedule as much as a measurement schedule; (c) DURABILITY OF CHANGE BEYOND AN IMMEDIATE POST-TEST is largely unestablished. THEORETICAL: how attentional measures (reading time, eye fixations) are interpreted as evidence of conceptual change is unjustified; which knowledge domains even admit of refutation is unspecified; there is no agreed criterion for what makes a text refutational; and the assumptions about how conceptual change occurs are unexamined. Taken with the Schroeder & Kucera 2022 moderator table - where the only significant moderator is publication type and dissertations show g = 0.11 n.s. - this is the strongest available reason to treat the refutation-text effect as a real but small and possibly measurement-entangled phenomenon rather than as evidence that a naive conception has been replaced.

Not applicable.

No replication concept applies. Its complaint about single-topic designs is independently visible in Schroeder & Kucera 2022 (44 comparisons, most one topic each) and its durability complaint is independently visible in Pacaci et al. 2024, which ran no retention analysis at all across 218 studies.