The Impact of Project-Based Learning on AP Exam Performance

74 schools randomised (86 consenting teachers) across five districts; Year One analytic sample 74 teachers in 68 schools with 3,645 students, of whom 42 teachers in 42 schools and 1,952 students were in AP Environmental Science

High-school Advanced Placement students (ages roughly 16-18) in five large, predominantly urban US districts with open-enrollment AP policies; 43% from lower-income households; four of the five districts serve majority Black and Hispanic students. 2016-17 and 2017-18.

The most valuable feature of this trial for a science archive is the OUTCOME: the College Board AP examination, an externally written and externally scored high-stakes test that no one in the study controls. Knowledge in Action (KIA) is full-year project-based learning — students work through project units with the teacher as facilitator — designed by University of Washington researchers with local teachers to cover the AP U.S. Government (APGOV) and AP ENVIRONMENTAL SCIENCE (APES) frameworks. Schools were randomised within five districts using Morgan and Rubin rerandomisation, with a staggered rollout so control schools got KIA in year two. THREE CAVEATS MUST TRAVEL WITH THIS. First, six schools (all treatment) attrited before the end of September of Year One, giving 8% overall attrition with a SIXTEEN percentage point differential — Year Two attrition was worse still (22 percentage point differential, exceeding WWC thresholds). Second, the headline estimate depends entirely on covariate adjustment: the unadjusted qualifying-score effect is -0.005 (SE 0.21), the primary covariate-adjusted model gives 0.264, and an alternative specification including all covariates with baseline-equivalence effect sizes above 0.05 gives 0.16 and is not significant. Third, the evaluators are Lucas Education Research grantees evaluating a Lucas-funded curriculum, and the developers are not on this author list but are the source of the programme. Numbers below are taken from the companion two-year report (Saavedra et al. 2021, ERIC ED616435, https://files.eric.ed.gov/fulltext/ED616435.pdf) because the EEPA article itself was paywalled; the trial is the same.

Year One, all students (n=3,645): the proportion earning a qualifying AP score (3+) was 4 percentage points higher with KIA — 35.1% versus a counterfactual 31.1% — ES=0.264, p=0.016. Among exam-takers (n=2,963) the difference was 7.6 points (44.8% vs 37.2%), ES=0.457, p=0.002, but this is contaminated by course-specific exam-taking effects. On continuous AP scores in four districts: total ES=0.192 (p=0.009), multiple-choice 0.188, free-response 0.181. THE SCIENCE-SPECIFIC ESTIMATE: the APES subgroup ES=0.304 on qualifying score, p=0.048 — significant but only just, and the study was not powered for subgroups. Effects were similar for lower- and higher-income students. A second year of teacher experience with the curriculum added little; the effect on AP exam performance occurred primarily in teachers' FIRST KIA year, when coaching support was intensive. This is the only estimate in the science tranche on a fully independent, externally scored achievement test at scale, and it lands at roughly 0.19-0.30 SD — with high differential attrition and covariate-dependence attached.

Medium. Randomisation was at school level within districts, but 16-22 percentage point differential school attrition plus the collapse of the estimate to zero without covariate adjustment means selection cannot be dismissed; the authors checked for and did not find student sorting into KIA classrooms.

The consistency across two years, two courses, five districts and both income groups is the study's own internal replication. No independent (non-Lucas-funded) replication of Knowledge in Action was located.