The Evidence on Teaching

Head Start Impact Study: Final Report

Puma, M., Bell, S., Cook, R., Heid, C., et al. (US DHHS/ACF) · 2010

grade Arctindependentunreplicatednumbers spot-checked
Sample
4,667 randomized (2,559 age-3 cohort; 2,108 age-4 cohort)
Population
Head Start-eligible children nationally.
Design
The only large national RCT of Head Start — congressionally mandated, designed on the recommendation of the independent Advisory Committee on Head Start Research and Evaluation, sponsored by ACF (the agency that administers Head Start; the ACF project officer is on the author line) and executed under contract by Westat with Abt, AIR, Urban Institute and others. Two independently randomised age cohorts (entering at 3 and at 4), each followed to the end of 1st grade; the 3-year-old cohort has an extra measurement wave (end of the age-4 year), during which many treatment children were in a SECOND Head Start year. No IQ test was administered — PPVT (receptive vocabulary) is the most g-loaded proxy available. Roughly 10,000 impact comparisons were tested across two cohorts, four waves and many outcomes, so the report grades evidence in three tiers: STRONG (p<0.05 and survives Benjamini-Hochberg), MODERATE (p<0.05 only), SUGGESTIVE (p<0.10, may or may not survive). Nearly every effect size in the summary exhibits is reported at the p<=0.10 threshold, so several "impacts" are suggestive-only. No-shows ~15-20% and crossovers ~14-17%; both ITT and IV-adjusted participation impacts are reported, and everything below is ITT.
Key findings
The definitive test of Head Start as actually delivered, and the cohort distinction is load-bearing. 4-YEAR-OLD COHORT: six small language/literacy gains at the end of the Head Start year (0.09-0.25 on direct assessments, 0.31 on parent report), never any math or pre-writing or promotion impact in any year, exactly zero cognitive impacts at the end of kindergarten, and at 1st grade only a suggestive PPVT +0.09; social-emotional zero in the Head Start year and kindergarten, then mixed and partly unfavourable at 1st grade. 3-YEAR-OLD COHORT: a broader end-of-year effect including math (Applied Problems +0.15) and PPVT +0.18, of which only two measures survive to the age-4 year and none survive to 1st grade except a suggestive Oral Comprehension +0.08; it is also the only cohort with social-emotional impacts, all of them parent-reported, and it carries the study's single unfavourable cognitive impact (teacher-rated math ability -0.19 at kindergarten). Across both cohorts, the durable findings are health-access and parenting, not learning: dental care +15pp (4yo) and +17pp (3yo) at the end of the programme year, gone by 1st grade, and health insurance coverage +4pp still present for the 4-year-old cohort at 1st grade. The counterfactual is an active alternative, not nothing, and both groups remain far below national norms throughout.
Counterfactual
NOT nothing, and the report says so explicitly: Head Start was tested "against a mixture of alternative care settings rather than a condition of no services". 4-year-old controls (Exhibit 1, spring 2003): 39.7% parent care only, 35.3% other centre-based care, 13.8% crossed into Head Start anyway, 11.2% other care. 3-year-old controls: 38.4% parent care, 25.2% other centre-based, 17.8% Head Start crossover. Controls who were in non-parental care were in it 4-5 MORE hours per week than the Head Start group. In the 3-year-old cohort's second year the control group was given access to Head Start, so the age-4 contrast is badly contaminated: ~50% of controls in centre-based care were in Head Start by spring 2004.
Genetic confound
RCT. The effect ordering is diagnostic: largest on parent-report, smallest on the most g-loaded measure.
Replication notes
No second national RCT of Head Start exists, so `unreplicated` refers to the design, not to the fadeout pattern, which is one of the most replicated regularities in the literature. The significant end-of-Head-Start-year cognitive effect sizes in this study run 0.09-0.31 (4yo cohort) and 0.10-0.35 (3yo cohort), centring near 0.15-0.20 — roughly a quarter to a third of the ~0.60 SD claimed for Perry/Abecedarian, which is the intensity gap that blocks extrapolation from the boutique trials. The ~0.14 SD "average" formerly recorded here is an external synthesis figure, not a number this report states.

Effects

OutcomeMetricValueMeasureTimingVsHorizonClass
Language/literacy, 4-year-old cohort, END OF HEAD START YEAR (Exhibit 3a)ITT SDsix significant direct-assessment impacts — PPVT (receptive vocabulary) +0.09, WJ III Letter-Word Identification +0.22, WJ III Spelling +0.15, WJ III Pre-Academic Skills +0.19, Color Identification +0.16, Letter Naming +0.25 — plus parent-reported Emergent Literacy Scale +0.31. LARGEST on the parent-reported measure, SMALLEST on the most g-loaded test.mixedspring 2003, end of the Head Start yearactive-alternativeend-of-treatmentdomain-skill
Math, pre-writing, grade promotion and teacher-rated school performance, 4-year-old cohort, ALL YEARSITT SDzero — "No significant impacts were found for math skills, pre-writing, children's promotion, or teacher report of children's school accomplishments or abilities in any year." The 4-year-old cohort never showed a math impact at any wave.mixedall four wavesactive-alternativeover-2yrdomain-skill
All cognitive outcomes, 4-year-old cohort, END OF KINDERGARTENITT SDzero — no significant cognitive impacts at all, one year after the programme endedstandardizedspring 2004, end of kindergartenactive-alternative1-2yrdomain-skill
Cognitive, 4-year-old cohort, END OF 1ST GRADEITT SDPPVT +0.09, SUGGESTIVE ONLY (p<0.10) — the single cognitive impact at this wave, on the same measure that was smallest at the end of treatment. Everything else null.standardizedspring 2005, end of 1st gradeactive-alternativeover-2yrdomain-skill
Cognitive, 3-YEAR-OLD cohort, END OF HEAD START YEAR (Exhibit 3b)ITT SDbroader than the 4-year-old cohort — PPVT +0.18, WJ III Letter-Word +0.26, CTOPPP Elision +0.10, Letter Naming +0.24, WJ III Pre-Academic Skills +0.22, pre-writing (McCarthy Draw-a-Design) +0.14, parent-reported Emergent Literacy +0.35, AND a math impact the older cohort never got — WJ III Applied Problems +0.15.mixedspring 2003, end of the age-3 Head Start yearactive-alternativeend-of-treatmentdomain-skill
Cognitive, 3-year-old cohort, END OF AGE-4 YEAR (second programme year)ITT SDonly two of eight survive one more year — parent-reported Emergent Literacy +0.16 and CTOPPP Elision +0.15. Math, PPVT, Letter-Word, Letter Naming and Pre-Academic Skills all gone. Note the contrast is degraded here because controls gained Head Start access in year 2.mixedspring 2004, end of the age-4 yearactive-alternativeend-of-treatmentdomain-skill
Cognitive, 3-year-old cohort, END OF KINDERGARTEN — including the study's only unfavourable cognitive impactITT SDSpanish Letter-Word Identification (Batería) +0.26 favourable; teacher-rated MATH ABILITY -0.19, UNFAVOURABLE — described by the report as "the only unfavourable cognitive impact found for either cohort as a whole in any year". It is not corroborated by the three direct math assessments, and the control group's kindergartens had significantly higher math proficiency.mixedspring 2005, end of kindergartenactive-alternative1-2yrdomain-skill
Cognitive, 3-year-old cohort, END OF 1ST GRADEITT SDWJ III Oral Comprehension +0.08, SUGGESTIVE ONLY. Nothing else.standardizedspring 2006, end of 1st gradeactive-alternativeover-2yrdomain-skill
Social-emotional, 4-year-old cohortITT SDzero on every measure in the Head Start year AND kindergarten. At the END OF 1ST GRADE the findings turn mixed and partly UNFAVOURABLE — teacher-reported socially reticent (shy/hesitant) behaviour higher in the Head Start group (moderate evidence) and more problematic student-teacher interactions (suggestive), against parent-reported less withdrawn behaviour (suggestive).mixedall four wavesactive-alternativeover-2yrnon-cognitive
Social-emotional, 3-year-old cohortITT SDthe one domain where the younger cohort clearly moves, but ENTIRELY on parent report: strong evidence of less hyperactive behaviour and fewer total problem behaviours at the end of the Head Start year; suggestive better social skills/approaches to learning at age 4 and kindergarten plus moderate evidence of less hyperactivity at kindergarten; at 1st grade only parent-child closeness (moderate) and overall parent-child relationship (suggestive) remain. NO impacts on any teacher-reported social-emotional measure in kindergarten or 1st grade.self-reportall four wavesactive-alternativeover-2yrnon-cognitive
Received dental care (parent-reported)ITT percentage points / SD4-year-old cohort +15pp (ES 0.31) at the end of the Head Start year; 3-year-old cohort +17pp (ES 0.33) at the end of the Head Start year and +10pp (ES 0.20) at the end of the age-4 year. Raw rates 73% vs 56% (4yo) and 68% vs 52% (3yo). Gone by 1st grade for both cohorts. Strong evidence (survives Benjamini-Hochberg) — the most robust short-run effect in the study, though the parent-reported Emergent Literacy effect sizes (0.31/0.35) are equally large.self-reportend of the Head Start year (and age-4 year for the 3yo cohort)active-alternativeend-of-treatmenthealth
Health insurance coverage and general health status (parent-reported)ITT percentage points / SD4-year-old cohort — health insurance +4pp (ES 0.11) at kindergarten (suggestive) and again at 1ST GRADE (ES 0.11, moderate evidence); overall health excellent/good +5pp (ES 0.13) at kindergarten (suggestive). 3-year-old cohort — health status ES 0.11 at the Head Start year, insurance ES 0.14 at kindergarten, care for injury ES 0.10 at age 4, nothing at 1st grade. Health insurance for the 4yo cohort is the only favourable impact of ANY kind that both cohorts' 1st-grade waves retained.self-reportkindergarten and 1st gradeactive-alternativeover-2yrhealth
Parenting practicesITT SD4-year-old cohort — essentially nothing, one isolated impact (less use of time-out, ES -0.17, Head Start year). 3-year-old cohort — spanking -0.14 (Head Start year) and -0.09 (kindergarten), read to child +0.15, family cultural enrichment +0.18, authoritarian parenting -0.14 (age 4) and -0.11 (1st grade), time-out -0.13 (kindergarten) and -0.11 (1st grade). Parenting is the domain with the most durable 3-year-old-cohort impacts, and it is entirely parent self-report.self-reportall four wavesactive-alternativeover-2yrnon-cognitive
Absolute standing against national normspercentile / percentthe gap is not closed. 4-year-old cohort PPVT in 2003: control group 27th percentile, Head Start group 31st. 3-year-old cohort: 29th vs 32nd. By the end of kindergarten 55% of the 4-year-old Head Start group and 58% of its controls could recognise all their letters, against 95% nationally (ECLS-K).standardizedHead Start year and end of kindergartennone1-2yrdomain-skill
Subgroup variation (exploratory, secondary — not confirmatory)ITT, differential impactssome subgroups retain cognitive impacts to 1st grade — 3-year-olds from high-risk households (five academic assessments at 1st grade), 3-year-olds in NON-URBAN settings (six language/literacy plus one math measure at 1st grade), 3-year-olds of parents with no depressive symptoms, and 4-year-olds of parents with mild depressive symptoms. Others are unfavourable — White 4-year-olds show unfavourable teacher-reported social-emotional impacts at 1st grade, and 3-year-olds of parents with MODERATE depressive symptoms show sustained negative impacts across cognitive, social-emotional and health domains including lower parent-reported grade promotion. The report labels all of this secondary and exploratory against ~10,000 comparisons.mixedall four wavesactive-alternativeover-2yrdomain-skill

Cited by