Improving Middle-School Students' Knowledge and Comprehension in Social Studies: a Replication

1,487 eighth-grade students in 85 US history classes randomly assigned to PACT or business-as-usual

Eighth-graders in Texas middle schools taking US history

Direct, larger replication of Vaughn et al. (2013) by the same team, with the same within-teacher randomisation logic - students in both arms received the same US history content from the same teachers, so the contrast isolates the PACT components rather than content coverage. Outcomes are the same battery: researcher-developed ASK content-acquisition and content-reading-comprehension subtests plus an independent standardized broad reading comprehension test. Follow-up assessments at 4 and 8 weeks make this one of the few sources in this cluster with any retention data. Recorded at SECONDARY read depth: the effect sizes here are taken from the developers' own detailed summary of the trial in Roberts et al. (2023, Journal of Educational Psychology) and from the published abstract, not from the article itself, which is paywalled - so the moderator tables and attrition detail have not been checked.

The replication is the pivot point of the whole PACT record. Content acquisition replicated and grew: g = 0.32 at posttest, maintained at 0.29 after 4 weeks and 0.26 after 8 weeks - genuine retention of taught history knowledge, on a researcher-built measure. But NEITHER content-area reading comprehension NOR broad reading comprehension showed statistically significant group differences, i.e. the two comprehension effects found in the original 2013 trial both failed to replicate at five times the sample size. Fidelity to three components (comprehension canopy, essential words, TBL knowledge application) mediated the treatment effect, which is the usual dose-response signature but is a post-randomisation contrast and cannot be read causally.

Low. Classes were randomly assigned within teacher, so ability-relevant alleles are balanced in expectation between arms.

This IS the replication of Vaughn et al. (2013), and the outcome is a partial failure: the knowledge effect held, the comprehension effects did not. The pattern then repeated at 11th grade (Swanson et al. 2015) and at scale (Roberts et al. 2023). All by the developer team; no independent group has evaluated PACT.