School’s Out: The Role of Summers in Understanding Achievement Disparities

roughly 200 million test scores from about 18 million students in 50 states, grades 1-8, school years 2008-2016

US school children taking the NWEA MAP assessment

American Educational Research Journal 58(2), 239-282; full text read from the June 2019 EdWorkingPaper version. Descriptive multilevel growth models on NWEA MAP - a CONTINUOUSLY ADAPTIVE, IRT-scaled, vertically linked assessment, which is what makes it immune to the change-of-test-form artifact that von Hippel and Hamrock identify as fatal to the classic summer-loss literature. The authors also reconstructed actual district calendars, recovering about 77% of district-year start and end dates, rather than interpolating test dates to assumed school-year boundaries - the second artifact in the dispute. Grade B rather than higher because it is descriptive: it measures when learning happens, not the causal effect of any intervention.

SUMMER LOSS IS REAL AND SUBSTANTIAL ONCE THE MEASUREMENT ARTIFACTS ARE REMOVED. 'On average, 19% of students' pathways from their 1st to 8th grade test-score occur during summers. For some students, this number is closer to 30 percent.' Mean summer change is NEGATIVE at every grade in both subjects: ELA falls 6.6 RIT points after first grade, 3.9 after second, 3.4 after third, easing to 0.9 by the start of grade 8, while school-year gains fall from 23.7 points in first grade to 4.4 by eighth. Expressed as a share: 'the average student loses between 17 and 28 percent of their school year ELA gains (a 9-month period) during the following summer (a 3-month period). In math, the relative losses are a little larger: between 25 and 34 percent.' And 52% of students lose ground in all five consecutive years in ELA. BUT THE LOSS IS NOT A GAP STORY: race, ethnicity and socio-economic status together explain only about 4% of the variance in summer learning loss. Summer is when a great deal of forgetting happens, and it is nearly universal rather than concentrated among the disadvantaged.

Low for the mean-loss finding, which is a within-child seasonal comparison and does not require comparing families at all. The demographic decomposition is a different matter, and the paper's own answer - that demographics explain about 4% of the variance - is the useful one: whatever produces individual differences in summer loss, it is largely not the observable family characteristics that the summer-gap literature assumed.

Replicated on the same data family by Kuhfeld, Condron and Downey, who find negative summer learning rates for every racial and ethnic group across 2.5 million children, and by Kuhfeld's Phi Delta Kappan analysis of 3.4 million students. It corroborates the mean-loss half of Cooper's 1996 meta-analysis while contradicting its gap half. Against von Hippel's much smaller headline - about two weeks of skill lost in the first summer - the disagreement is largely about denominator: absolute skill units versus a share of a shrinking school-year gain. Both are looking at adaptive tests.