Learning from Summer: Effects of Voluntary Summer Learning Programs on Low-Income Urban Youth

about 5,600 applicants randomised across five urban districts; 3,192 in the analytic sample

Low-income urban elementary students entering fourth grade, in five US districts

RAND research report RR-1557-WF; the summary and methods chapters were read in full. LOTTERY ADMISSION to free, voluntary, district-run five-week summer programmes across two consecutive summers (2013 and 2014), with outcomes tracked to spring 2015 - the only randomised trial of summer programmes at this scale. Independence is recorded as developer-involved because the Wallace Foundation both commissioned the evaluation and funded the programmes being evaluated, though RAND is methodologically independent and the report's headline findings are null. Note the honest self-correction in a footnote: an earlier publication reported an intent-to-treat estimate of 0.09 which was revised to 0.08 after additional data became available.

One small causal effect, and it faded. 'The first summer of programming resulted in a modest near-term benefit in mathematics measured in fall 2013... The standardized average effect of offering the program was 0.08 and was statistically significant' on a study-administered standardized assessment. Then: 'NO CAUSAL EVIDENCE THAT TWO SUMMERS OF PROGRAMMING PROVIDED BENEFITS - we found no significant effects of offering two summers of programming for treatment students in mathematics, language arts, social-emotional competencies, or school-year behaviors.' The report's own section heading is 'Modest Near-Term Benefit in Mathematics, Dissipated by the Next Fall', and the summary sentence is 'we generally conclude that we have no evidence of causal impacts except for the near-term mathematics estimate after the first summer'. The much-quoted larger figures - 0.13 to 0.16 SD in maths and 0.08 to 0.09 in ELA - are all HIGH-ATTENDER comparisons, which the authors label correlational and exploratory and about which they write that 'selection bias remains a possibility'. Roughly 60% of attending students were high attenders and about 35% hit the time-on-task threshold, and nearly half of treatment students did not attend at all in the second summer, which dilutes the intent-to-treat estimate.

Low for the randomised estimates - lottery admission balances family and child characteristics - and high for the high-attender analyses, which the authors flag themselves. The gap between the two is the useful measurement: 0.08 SD randomised against 0.11-0.16 SD for the self-selected high attenders.

Unreplicated as a trial. Its randomised magnitude is consistent with the design-screened meta-analytic estimates - Lynch and colleagues' +0.10 SD for summer mathematics programmes and Kidron and Lindsay's non-significant 0.16 SD for summer-school literacy - and inconsistent with Cooper's 2000 summer-school meta-analysis, which reported 0.26 SD for literacy and mathematics from a corpus that Kidron and Lindsay describe as 'mostly studies with less rigorous study design'.