Upgrading Education with Technology: Insights from Experimental Research
Escueta, M., Nickow, A. J., Oreopoulos, P., & Quan, V. · 2020
grade Creviewindependentmixednumbers spot-checked
Sample
Roughly 100 RCTs and regression-discontinuity studies, sorted into four categories - 11 RCTs and 4 RDDs on access to technology, 29 RCTs on computer-assisted learning in developed countries, a large set of behavioural-nudge trials, and a small set of online-course trials
Population
Primarily developed countries (US-heavy), all education stages from early childhood through postsecondary and adult; developing-country evidence drawn on for comparison.
Design
The field's best organising review, and its value is taxonomic rather than statistical: it refuses to average across "ed-tech" and instead prices four different things separately, reaching opposite conclusions for each. Inclusion is restricted to RCTs and regression-discontinuity designs, which is a real quality floor and the reason it is not graded D. But the synthesis is NARRATIVE VOTE COUNTING, not meta-analysis: there is no pooled estimate, no weighting, no coding of outcome-measure type, and no publication-bias analysis, so the CAL section's "only eight of 29 report no effect" is a count of statistical significance across studies of wildly different size and measure alignment. That is precisely the inferential move Cheung & Slavin (2013) and Kulik & Fletcher (2016) show to be unsafe in this literature. Funded by the Spencer Foundation with J-PAL North America support; no vendor involvement. VERSION READ: NBER Working Paper 23744 (August 2017), "Education Technology: An Evidence-Based Review", read in full (102 pp.). The published Journal of Economic Literature 58(4) 897-996 (December 2020) version is a revision under the title recorded above and was not read; author order also differs between the two versions (NBER: Escueta, Quan, Nickow, Oreopoulos).
Key findings
The review's organising result is that "ed-tech" is not one thing and the four things it is have different signs. (1) Giving students hardware reliably increases computer ownership, usage and computer skills and does approximately nothing to achievement - the only qualifying K-12 RCT found no significant effect on homework time, grades, standardized test scores or attendance, and the Romanian and Dutch subsidy RDDs found NEGATIVE achievement effects. (2) Computer-assisted learning software, especially with a personalization/adaptive component and especially in mathematics, is where the positive RCT results are - 20 of 29 developed-country CAL RCTs positive, with standouts at 0.18 SD (a light-touch online maths homework programme) and 0.63/0.56 SD (an intensive software maths curriculum in Texas). (3) Technology-enabled behavioural nudges (text and email campaigns to students and parents) produce generally positive but smaller effects, at very low cost, across the whole education life cycle. (4) Fully online courses produce small negative learning effects relative to face-to-face; blended courses are at parity. The headline the archive keeps is the contrast between (1) and (2): the hardware does nothing, the well-designed adaptive software sometimes does a lot, and the difference is the software, not the screen.
Genetic confound
Low - by construction the review admits only randomized and regression-discontinuity designs, so genetic/selection confounding between arms is minimal. The residual threat is selection into the studies and into the settings, not within them.
Replication notes
The access-to-technology conclusion (hardware alone does almost nothing to achievement) is replicated everywhere it has been tested and is the most secure finding in the review. The computer-assisted-learning optimism is NOT replicated by the meta-analytic literature on the same studies: this review's vote count (20 of 29 CAL RCTs positive) sits against Cheung & Slavin (2013) large randomized +0.06 on standardized mathematics tests, Steenbergen-Hu & Cooper (2013) ~0.05 for K-12 ITS, and the Dynarski/Campuzano federal RCTs (+0.03) that this review itself sets aside as hard to interpret. The online-vs-face-to-face results (small negative for fully online, parity for blended) are consistent across the handful of postsecondary RCTs but rest on few studies.
DOI / URL
10.1257/jel.20191507
Effects
| Outcome | Metric | Value | Measure | Timing | Vs | Horizon | Class |
|---|---|---|---|---|---|---|---|
| Access to technology (free computers/laptops), K-12 achievement | ES | no significant impact - positive or negative - on homework time, grades, standardized test scores or attendance in the only qualifying K-12 RCT (6th-10th graders, 15 California middle and high schools). Household computer subsidies in Romania and school computer subsidies in the Netherlands produced NEGATIVE achievement effects in RDD studies. US internet subsidies and English connection-speed variation found no substantial impact either way. | standardized | end of programme year | business-as-usual | end-of-treatment | domain-skill |
| Access to technology, proximate/computer-skill outcomes | ES | consistently effective at their proximate targets - the California laptop RCT raised computer ownership 55 percentage points, computer use 2.5 hours/week and home internet connection 25 percentage points; the Romanian subsidy raised computer skills and cognitive test scores even while lowering achievement | mixed | end of programme year | business-as-usual | end-of-treatment | near-transfer |
| Displacement of study time by home computers | description | the Romanian household-computer RDD attributes its negative achievement effect partly to students spending more time playing games; the California K-12 RCT measured homework time explicitly and found no change. This is the archive's displacement row for hardware access, and the review notes it is measured in only a handful of studies. | self-report survey | during programme | business-as-usual | end-of-treatment | behaviour |
| Access to technology at POSTSECONDARY level | SD | +0.137 SD on a constructed academic performance index (course success rate, taking courses for a grade, taking transfer courses, graduation) from a laptop distribution RCT at a Northern California community college; effects stronger for minority students. A follow-up found NO impact on earnings seven years later. | administrative | through college and 7 years post | business-as-usual | over-2yr | attainment |
| Computer-assisted learning (CAL) software, developed countries | vote count + exemplar effect sizes | 29 RCTs; 20 positive, 8 null, 1 negative; 15 of the 20 positives were mathematics. Exemplars - an online mathematics homework feedback programme at 0.18 SD on less than 30-40 min/week, and an intensive software-based mathematics curriculum in Texas at 0.63 SD (grade 7) and 0.56 SD (grade 8). NO pooled estimate, no measure-type coding, no publication-bias analysis - the count is of statistical significance, not of magnitude. | mixed | end of course or school year | business-as-usual | end-of-treatment | domain-skill |
| Technology-enabled behavioural nudges (texts, emails to students and parents) | description | generally positive across every stage of the education life cycle, but generally SMALLER than the best CAL effects - and extremely cheap, which is where the cost-effectiveness case sits. Several large nudge RCTs in the review found no effects at all (automated texting to high-schoolers, college application nudges). | administrative | varies | business-as-usual | under-1yr | attainment |
| Fully online courses vs face-to-face (postsecondary) | description | small NEGATIVE learning effects for fully online relative to face-to-face; the review declines to back substantial shifts toward fully online courses on this evidence | standardized | end of course | active-alternative | end-of-treatment | domain-skill |
| Blended (part online, part face-to-face) courses vs face-to-face | description | no significant underperformance in any qualifying study - the review's practical conclusion is that switching from fully in-person to blended can cut cost without cutting quality | standardized | end of course | active-alternative | end-of-treatment | domain-skill |