Examining the effects of a highly rated science curriculum unit on diverse students: Results from a planning grant

10 schools (5 intervention, 5 comparison); Cohort 1 n=1,896 students, Cohort 2 n=2,280; 4,176 grade-8 students total

Grade 8 science, Montgomery County Public Schools, Maryland — a large, ethnically diverse and high-performing suburban district (no ethnic majority; 28% free/reduced lunch, 6% English learners).

Schools were sampled from five matched demographic/achievement profile strata, two per stratum, and one of each pair was randomly assigned to the intervention — so it is a cluster randomised design, but with only TEN clusters. The intervention is Chemistry That Applies, a Michigan Department of Education unit rated acceptable by AAAS Project 2061 and described by WWC as focusing on "guided inquiry" with hands-on, student-centred material: students work in groups, explore chemical reactions, collect data, use evidence-based arguments, keep science notebooks, and respond to scripted "Think and Write" prompts. 18 of the 24 lessons were taught, over about seven weeks, after two days of teacher PD. The comparison arm was NOT a straw man — comparison teachers used the regular Montgomery County materials addressing the same benchmarks, drawn from traditional textbooks, Prentice Hall, NSF-funded reform-based materials and teacher-designed materials, and all teachers in both arms had exposure to "reform-based" strategies. The outcome, the Conservation of Matter Assessment (COMA, 10 items, alpha=0.71), was built for the study and aligned to the benchmarks the unit targets. Differential student attrition was 13% in Cohort 1 (high) and 3% in Cohort 2.

WWC-recomputed effects on the COMA were d=0.32 (Cohort 1) and d=0.25 (Cohort 2), domain average d=0.29 (+11 percentile points) — but the WWC correction for clustering makes the domain average NOT statistically significant, because the design has only ten clusters. The published paper reports significant posttest advantages, and the subgroup story it is famous for (largest gains among Hispanic students, low-income students and English learners; every subgroup except currently-enrolled ESOL students beating its comparison peers) survives only within that uncorrected analysis. This is the cleanest example in the science tranche of a "guided inquiry beats business-as-usual" result that shrinks to non-significance once cluster randomisation is analysed correctly.

Low by design (paired random assignment of schools within demographic strata), but ten clusters cannot balance much, and the high Cohort 1 differential attrition is a residual exposure; WWC confirmed baseline equivalence for Cohort 1 before including it.

A follow-on effectiveness study by the same group (Lynch, Taymans, Watson, Ochsendorf & Pyke, 2007) exists but was not obtained; WWC's Chemistry That Applies report rests on the Pyke/Lynch cohort analyses and rates the intervention "potentially positive effects" with a small extent of evidence.