Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning

327 volunteers randomised (112 to live, 215 to online) out of roughly 1,600 enrolled; 15 live-assigned students requested reassignment for timetable clashes and were dropped, leaving 97 in the live arm; 296 students (91 live, 205 online) had complete exam data. Subgroup cells are small - about 35 Hispanic students in total.

POST-SECONDARY. Undergraduates in a large Principles of Microeconomics course (1,600-2,600 students per semester, one instructor) at a large, selective, doctorate-granting US university, Spring 2007. Well outside the archive's 4-18 age band, and at an institution where enormous lecture courses and electronic delivery were already the norm - the authors flag both as external validity limits.

WHAT "ONLINE" MEANS HERE: ASYNCHRONOUS on-demand video of the SAME live lecture, watched on the same course website. It is not live synchronous video, not self-paced software, and not a different course. Both arms shared the identical rich web environment (online quizzes, past exams), the same instructor, the same TA and office-hours access, and the same machine-graded multiple-choice exams taken at the same time in the evening; grading was exams only. The paper is therefore explicit that this is NOT lecture-vs-internet in the abstract - the whole course was already a live/online hybrid, and the experiment varies only how the lecture portion is delivered. DESIGN: the instructor emailed all enrolled students; 327 volunteered in exchange for a half-letter-grade boost (the only incentive the IRB permitted) and were randomised. Live-assigned students had online lecture access removed from their accounts and were the only people admitted to the 190-seat lecture room, with graduate students on the door recording attendance. THREATS THE PAPER DOCUMENTS: (1) volunteers only - 327 of ~1,600 - and volunteers differ modestly from non-volunteers (higher university GPA, 18 SAT points lower, less likely to have a college-graduate mother), so this is a LATE for a self-selected group and external validity is the authors' own stated main worry; (2) non-compliance - only 32% of "live-only" students attended at least 90% of live lectures and 36% attended fewer than 20%, and some could watch online via friends' accounts, which the authors argue biases toward zero; (3) baseline imbalance - the online arm started with higher university GPA (3.32 vs 3.14, p<.05), which disappeared by course end; (4) small samples for the subgroup claims. POWER: the design can detect about 2 points on a 100-point scale. ATTRITION: 6% of the live arm and 4.6% of the online arm withdrew. FUNDING: NSF and the US Department of Education via CALDER; no vendor. READ NOTE: read in full as NBER Working Paper 16089 (June 2010); the Journal of Labor Economics version (31(4), 763-784) is paywalled and the tables below are the working paper's.

Students randomised to watch the lecture live rather than online scored 1.44 points higher on the 0-100 average of three exams (SE 1.21) - roughly +0.15 SD favouring live, and not statistically distinguishable from zero. The honest headline is therefore "modest evidence that live dominates internet", which is what the authors wrote, and its main value in 2010 was as the first randomised contradiction of the US Department of Education meta-analysis then being read as showing online is better. The subgroup results are what get quoted - live instruction was ahead by 4.05 points for below-median prior achievers (p<.01), 3.48 points for male students (p<.05) and 11.28 points for Hispanic students (p<.01) - and they are the least trustworthy part of the paper - the Hispanic cell is about 35 students, an 11-point gap is over a full standard deviation, and the authors report trying "a variety of stratifications". Take the pattern (harm concentrated on weaker students) as consistent with Bettinger and Kofoed, and take the magnitudes as noise.

Minimal (randomised), with the caveat that randomisation was among self-selected volunteers, so the estimate is a local effect for students willing to trade format choice for extra credit, not for the course population.

The direction has held up in stronger designs - Bettinger et al. (2017) find -0.25 to -0.33 SD at DeVry with an instrumental-variables design on 2.3 million course records, and Kofoed et al. (2024) find -0.215 SD in a preregistered randomised trial at West Point. What has NOT been replicated is anything in this paper's subgroup table - the +11.3-point Hispanic estimate rests on roughly 35 students and the authors themselves report having tried several stratifications, so it should be treated as hypothesis-generating rather than as a finding. The overall contrast here is also statistically indistinguishable from zero, so calling this a "negative result" overstates what it shows on its own.