Students, Computers and Learning: Making the Connection

About 510,000 students in 65 countries and economies (PISA 2012), of whom roughly 20 OECD countries took the computer-based digital reading and mathematics assessments; trend analyses compare PISA 2003 with PISA 2012.

15-year-olds in 65 PISA-participating countries and economies, 2012 (with 2003 comparison cohorts for the trend analyses).

RECORDED PRIMARILY AS A BOUNDARY MARKER. This is the most-quoted evidence in the popular debate about screens in school, and the archive should hold its real grade rather than leave the number circulating ungraded. The analyses are CROSS-SECTIONAL CORRELATIONS at two levels: (a) across countries, relating national PISA scores and score trends to national ICT investment, with partial correlations that adjust for per-capita GDP and prior PISA performance; (b) within countries, relating individual students' self-reported computer use to their scores, adjusting for the PISA index of economic, social and cultural status at student and school level. No randomization, no lottery, no discontinuity, no instrument, no panel. Grade D is the correct row of METHODOLOGY.md's hierarchy and this is not a close call. To OECD's credit the report says so itself, at length and unprompted (Box 6.1): "in non-experimental, cross-sectional data such as those gathered through PISA, even sophisticated statistical techniques cannot isolate the cause-and-effect relationship"; it names reverse causality (systems that are worried about results buy more computers) and non-random allocation of computers to students, schools and tracks as the specific threats. THE U-SHAPE IS THE PART TO BE MOST CAREFUL WITH. A hill-shaped relationship between computer use and achievement — rare users below moderate users, heavy users far below both — is exactly the pattern that confounding by student ability and school selection would produce with no causal effect whatsoever: the lowest-attaining students are disproportionately in remedial and vocational settings where software is used most intensively, are most likely to be assigned drill software, and are most likely to report heavy leisure screen use; meanwhile the highest-attaining students in the best-resourced schools sit in the middle of the use distribution. Every one of those mechanisms generates the observed hill without any harm from screens. The report is nonetheless honest enough to state the trade-off explicitly, and it is one of the few sources in this cluster that names DISPLACEMENT as the mechanism to look for: "computer use in classrooms and at home can displace other activities that are conducive to learning", and it insists that the opportunity cost of the money belongs in the accounting. It names it; it does not measure it.

Countries that invested heavily in school computers showed no appreciable improvement in reading, mathematics or science. After adjusting for per-capita GDP and for PISA 2003 performance, the cross-country partial correlation between computers per student and PISA 2012 mathematics is -0.26 (reading -0.23, digital reading -0.51), and between the index of ICT USE at school and mathematics -0.65 (reading -0.50); systems that reduced their student-computer ratio most between 2003 and 2012 tended to see mathematics performance fall (R2 = 0.27). Within countries, the relationship is hill-shaped: moderate users outscore both rare users and heavy users on print reading, digital reading and the quality of online navigation, after adjusting for socio-economic status. Heavy leisure Internet use also tracks disengagement — students online more than six hours a weekday are twice as likely as moderate users to report feeling lonely at school (14% vs 7%) and much more likely to have arrived late for school in the past fortnight (45% vs 32% for those online under an hour). The caveat that most changes how this should be read: none of it is causal, the U-shape is precisely what selection on student ability and school type would produce with zero true effect, and OECD says so itself. The report's own conclusion is about pedagogy, not devices: "technology can amplify great teaching but great technology cannot replace poor teaching."

High. Every relationship reported is an association between what students and systems chose to do and how students score. Ability, family background and school selection drive both sides of the within-country gradients, and the report explicitly declines to interpret any of it causally.

These are descriptive analyses of one cross-sectional survey wave (PISA 2012), plus a 2003-to-2012 trend comparison, so "replication" in the trial sense does not really apply and no replication attempt is recorded here. We have not checked whether the specific correlations have been independently recomputed on later PISA waves, so this is `unclear` rather than `unreplicated` or `not-applicable`. What HAS been tested independently is the underlying causal question this report is used to answer, and there the record is broadly consistent with the null: randomized and quasi-experimental evaluations of school computer provision (Angrist & Lavy, Goolsbee & Guryan, Leuven et al., Fairlie & Robinson) find little or nothing on achievement. The report's celebrated U-shaped curve, by contrast, has no causal corroboration at all and should not be treated as if it did.