Reading Like a Historian: A Document-Based History Curriculum Intervention in Urban High Schools

236 eleventh graders enrolled at baseline in 10 classrooms across 5 schools; 182 students in the analysed sample after listwise deletion (23% loss)

11th graders in US History, five public high schools in San Francisco Unified School District

The anchor evaluation of Reading Like a Historian, and much weaker than its reputation. NOT randomised: the author states plainly that neither students nor teachers could be assigned to condition. Treatment teachers were the ones who could attend a four-day summer training; teachers who were interested but could not attend became controls - i.e. condition is confounded with teacher availability and motivation. Each of five schools contributed one treatment and one control classroom. Six-month intervention; treatment teachers agreed to use RLH lessons for at least 50% of instructional time and in fact used them 42-72% (M = 58.3%). Clustering was NOT modelled: the author concedes an HLM "would have been preferable, but was not possible given our small sample of teachers", so 182 students nested in 10 classrooms were analysed as independent observations via MANCOVA with school as a fixed factor. That inflates significance substantially; the effective unit of analysis is 5 vs 5 teachers. The treatment package also bundles the curriculum with four days of PD, two follow-up workshops, and TWICE-WEEKLY observation by the intervention's designer, who reports occasionally stepping in to assist teachers with classroom management and answering questions about lessons; control classrooms received no outside presence at all. Fidelity was low by the study's own rubric - average scores fell below the baseline of 2 for four of five teachers, and only three teachers ever facilitated whole-class text-based discussion, "extremely rare" when it happened. Three of the four outcomes were built by the research team. Group equivalence at pre-test was reasonable (no differences on 10th-grade CST, Historical Thinking pre-test, or Gates-MacGinitie) except that the treatment group had significantly more females.

Treatment classes scored higher on all four post-tests, but the paper reports only partial eta-squared and adjusted means, never effect sizes. Computed from the paper's Table 5 (observed means/SDs), Cohen's d is roughly 0.49 historical thinking, 0.56 transfer, 0.29 factual knowledge, and 0.30 reading comprehension; using the covariate-adjusted means instead gives 0.39 / 0.44 / 0.23 / 0.29. The famous and most-repeated claim - that document work raises GENERAL reading comprehension - rests on a single d~0.3 on the Gates-MacGinitie in one unrandomised five-school study, and the author's own explanation is time-on-task, not disciplinary transfer: RLH students "read daily" while control classrooms could spend time on films, posters, role-plays and games. Follow-up decomposition found effects on only two of four historical reading strategies - sourcing and close reading - and NO effect on contextualization or corroboration, which the author attributes to the near-absence of whole-class discussion. Struggling readers (bottom quartile on GMRT, n = 42 across both arms) gained on historical thinking and factual knowledge but NOT on transfer, and their reading-comprehension advantage was not significant. No treatment x school, x gender, x race/ethnicity, x ELL or x incoming-reading interactions.

Medium-high. Assignment was by teacher availability, not randomisation, so treatment and control student populations are not exchangeable by design. The author checked equivalence on prior achievement (10th-grade state history test, GMRT) and found none, which mitigates but does not eliminate the concern - equivalence on measured pre-tests does not establish equivalence on unmeasured heritable traits, and the assignment mechanism (which teacher could give up four days of summer) plausibly selects on school and classroom composition.

No independent replication of this trial exists as of mid-2026. Every subsequent RLH study from the Stanford History Education Group is descriptive (curriculum-use surveys, teacher case studies) rather than a second outcome evaluation. An independent effectiveness study of new RLH materials is under way at AIR under US Dept of Education grant S422B2300049, with no results published yet. Under the Replication rule an unreplicated developer-led positive caps the verdict at mixed.