Measuring the impact of an instructional laboratory on the learning of introductory physics

2 courses; ~1,100 lecture students, of whom 211/571 (physics 1) and 169/530 (physics 2) took the optional lab

Undergraduates at a single elite private US research university; calculus-based introductory mechanics and electromagnetism.

The prototype of the Holmes et al. 2017 design, at one institution and two courses. Optional lab enrolment; selection handled by normalising each student's performance on lab-related exam questions against their performance on non-lab-related questions from the same exam. Grading was done BLIND to lab enrolment, which is worth noting — the outcome scorer did not know the treatment. The labs studied were not 'cookbook': they included pre-lab preparation, prediction and reflection prompts, and were deliberately designed to support conceptual learning, so this is a null on a reasonably steelmanned lab, not a caricature. Single institution and highly selective intake is the reason for grade C rather than B.

OFF-SCOPE (undergraduate) but recorded because it is the origin of the 'labs add nothing' result and because it contains the archive's favourite object lesson. Raw comparison: students who took the optional lab scored significantly HIGHER on the exam, t(236.33)=4.86, p<.001 — the naive correlational finding that would be reported as 'labs work'. Once each student's lab-related score is normalised against their own non-lab-related score, the entire advantage disappears: the difference is within 1% with an uncertainty of 2%. The whole apparent benefit of labs was selection. Grading was blind to lab enrolment, and the labs in question were thoughtfully designed with pre-lab preparation and reflection prompts, not cookbook exercises. The authors conclude this 'raises questions as to the effectiveness of labs at supporting mastery of physics content'.

Medium — self-selection into an optional lab, addressed by the within-student normalisation rather than by design.

Replicated and substantially extended by Holmes, Olsen, Thomas & Wieman 2017 (9 courses, 3 institutions, 2,617 students), which reproduced the null with far greater precision and across much more varied student populations.