Examining the Reliability of Running Records: Attaining Generalizable Results

generalizability study - 10 teachers x 10 first-grade children x 2 leveled passages

US first-grade children assessed with running records by their own teachers

A generalizability study, which decomposes score variance into its sources rather than estimating an effect - the right design for asking how many observations a reliable placement decision needs. Small (10 teachers, 10 children), single grade, and the passages are commercially leveled texts. Graded C.

The problem with a running record is not the scorer, it is the sample. RATERS contributed less than 1% of score variance - teachers agree with each other almost perfectly about what they heard. But passage-to-passage and occasion-to-occasion variance is large, and the authors conclude that a MINIMUM OF THREE PASSAGES is needed for a reliable estimate. A single running record, which is how reading level is assigned in practice everywhere, does not reliably place a child. A follow-up generalizability study on naturalistic lessons put the requirement higher still, at roughly eight to ten reads for reliable accuracy and self-correction measurement.

Not applicable - a variance decomposition of an instrument.

Replicated in direction and strengthened in magnitude by the 2021 Reading Psychology generalizability study on naturalistic lessons, which found that even more observations are needed.