The Evidence on Teaching

Effects of Dual-Language Immersion Programs on Student Achievement: Evidence From Lottery Data

Steele JL, Slater RO, Zamarro G, Miller T, Li J, Burkhauser S, Bacon M · 2017

grade Bnatural-experimentindependentunreplicated
Sample
seven cohorts of dual-language immersion lottery applicants in a large urban district (Portland, Oregon)
Population
Kindergarten applicants to oversubscribed dual-language immersion programmes in Portland Public Schools (Spanish, Japanese, Mandarin, Russian, Vietnamese partner languages), followed through grade 8; both English-proficient students and English learners
Design
The identification is a genuine admissions LOTTERY among oversubscribed immersion programmes, which is what makes this the only grade-B evidence in the immersion literature: parents self-select into APPLYING, but not into winning. Intent-to-treat estimates are the headline; compliance is imperfect in both directions (lottery losers sometimes enrol later, winners sometimes decline), so treatment-on-treated effects are larger. CRITICAL SCOPE LIMIT: every achievement outcome is a state test administered IN ENGLISH. The study does not measure partner-language proficiency at all, so it cannot say how well immersion students learn the second language - the outcome most people buy immersion for.
Key findings
Positive intent-to-treat effects on ENGLISH READING of 13% to 22% of a standard deviation at grades 5 and 8 - the authors gloss this as 7 to 9 months of learning. Little benefit in mathematics and science, but also no detriment, which is the finding that answers the usual objection (that teaching half the day in another language costs content learning). By grades 6 and 7, lottery winners who had been classified as English learners were 3 to 4 percentage points less likely to remain classified as EL, and that effect was stronger when the student's native language matched the programme's partner language.
Genetic confound
Low, by design. Winners and losers are drawn from the same pool of applicant families, so the expected distribution of heritable ability and of family motivation is identical across arms. This is the whole reason the study is worth more than the rest of the literature combined: everywhere else, immersion families are the confound.
Replication notes
No second immersion lottery of comparable scale has been published, so this is a single unreplicated grade-B result carrying an entire literature. Its English-reading finding is broadly consistent in direction with the bilingual-education meta-analyses, which are much weaker designs.

Effects

OutcomeMetricValueMeasureTimingVsHorizonClass
ENGLISH reading achievement, grade 5 (state test)ITT effect size0.13-0.22 SD across grades 5 and 8 (glossed as 7-9 months of learning)standardizedgrade 5business-as-usualover-2yrdomain-skill
ENGLISH reading achievement, grade 8 (state test)ITT effect size0.13-0.22 SD across grades 5 and 8standardizedgrade 8business-as-usualover-2yrdomain-skill
Mathematics and science achievement (state tests, in English)ITT effect sizelittle benefit and no detrimentstandardizedgrades 5 and 8business-as-usualover-2yrdomain-skill
Probability of remaining classified as an English learner, grades 6-7ITT effect, percentage points3-4 percentage points lower for lottery winners; stronger when home language matches the partner languageadministrativegrades 6-7business-as-usualover-2yrattainment
Partner-language (L2) proficiencynot measurednot measured by this studyunknownnot applicableunclearnot-applicableunclear

Cited by