Does teaching programming actually teach programming — and does the method matter?

Verdict

moderate-support, and it is the honest justification for computer science in school — the one this subject should be sold on instead of the transfer claim it usually leans on.

Three things are true at once and the verdict has to hold all of them.

1. The domain-skill claim survives. One genuine cluster RCT moved coding skill g = 0.47 against business-as-usual; a second moved it d = 0.68; a pupil-randomised English trial moved it d ≈ 0.67 over a full academic year. Four meta-analyses converge on roughly g = 0.5-0.8. Under this archive's premise that is exactly what should happen: nobody programs without being taught, so mean instructional effects on a taught skill ought to be real and large.
2. Nothing reaches grade A. There is no preregistered multi-site RCT, no independently replicated RCT and no lottery design anywhere in this literature. The single best trial is developer-led, uses developer-built instruments, and was downgraded by the What Works Clearinghouse for attrition.
3. The instructional-approach question comes back much weaker than the does-it-work question. If someone asks which method to use, the honest answer from this literature is: it does not much matter — teach it seriously and measure it with something you did not build.

What the evidence shows

The measurement problem is not a caveat; it is the headline. Computing education's only serious language-independent instrument line — FCS1, then SCS1, then SCS1Rv2 — has needed psychometric repair at every generation, and the version the community actually adopted reported an alpha around 0.59, below the 0.65 threshold its own authors set. That means the overwhelming majority of the effect sizes in this file rest on instruments the intervention's own designers wrote. Apply the archive's standing correction — researcher-designed measures inflate roughly 2× against independent standardized ones — and the honest translation of "g = 0.81 for programming instruction" is something near 0.4, which is approximately the one randomised number this field has.

Every time the measuring instrument and the intervention stop sharing an author, the effect shrinks. Blocks versus text, the one contrast metaed by people who built neither environment: null with publication bias. The CAL trial's independently scored outcomes: null. The federal What Works Clearinghouse reviewed that trial and reviewed only the literacy and computational-thinking domains — the coding result, which carries the entire positive case, sits outside the independent review.

The subgoal literature contains its own replication failure, in its founding paper. Morrison, Margulieux & Guzdial set out to test given against generated subgoal labels and report that the given-label gains did not replicate as expected in the introductory CS task, despite replicating in mathematics and science. The honest reading of the whole line is that subgoal labelling is a real but small and fragile effect on near-term formative performance, whose best-documented benefit is reducing catastrophic failure rather than raising the mean — exam variance fell and the rate of failing or missing exams roughly halved.

Parsons problems are the one clean, useful, internally replicated lever, and it is a lever on time. Same post-test, same one-week retention, roughly half the practice time. The authors flag a ceiling effect on their post-tests, which is exactly the condition under which a no-difference result is least informative, so treat "same learning" as "no detectable loss" rather than as equivalence.

Two base rates deflate the field's own framing. Worldwide CS1 pass rates are 67.7% and failure rates 28%, against 42-50% for US college algebra — programming is not uniquely brutal. And CS grades are not bimodal in 94% of the 778 distributions where anyone tested it. The second finding comes with an experiment: professors randomly primed with the bimodality folklore, and professors who believed in innate CS aptitude, were more likely to label ambiguous histograms bimodal. The field's most durable belief about who can learn to program is a confirmation loop.

The strongest attainment evidence is not about instruction at all. A staggered-rollout instrumental-variable design on 635,771 Maryland students finds that taking a high-quality high-school CS course raises the probability of declaring a CS major by 10.2 points, of earning a CS degree by 5.5 points, and annual earnings at 24 by about 8%. Two honest caveats: the first stage is modest (F 13.0-16.6) and the authors concede it may bias the estimate upward, and the gains come partly from reallocation within STEM — CS pulled students out of other STEM (10-13 points) and engineering (5-9 points), not out of non-STEM.

Hereditarian-lens assessment

Risk: low. The anchor evidence is randomised at pupil, class or school level, and the Maryland attainment result is identified off the timing of a school's course offering — none of which any family chose.

The lens does real work in one place, though, and it cuts against a hereditarian folk belief rather than for it. The "geek gene" — the claim that programming aptitude splits people into two kinds — is the strongest innatist claim in this field, and it fails on its own terms: only 5.8% of 778 CS grade distributions are multimodal, and the belief predicts the perception of bimodality under experimental priming. Be precise about what that does and does not establish. A unimodal grade distribution is not evidence against heritable variation in programming aptitude; individual differences in a heritable trait produce a normal distribution, not a bimodal one. What fails is the specific two-populations claim, which was never what heritability predicts. The archive's premise is untouched, and the folk belief that borrows its prestige is not.

No genetically informative study exists anywhere in this literature — no twin, adoption or sibling design on programming skill or on response to programming instruction. That is a clean gap, and it is the same gap the archive flags for trainability in motor skill.

Boundaries & what critics say

• The design distribution is bad. One real cluster RCT (developer-led, WWC-with-reservations), one credible natural experiment (which measures earnings, not skill), one randomised-block school study that discarded its randomisation to report dose-response associations, and otherwise small unregistered lab experiments and single-institution section comparisons. The meta-analyses inherit exactly this.
• The robotics meta-analyses should be read as hypothesis-generating. Sun & Zhou's moderator table — biggest effects at the shortest durations, the smallest samples, and dependent on the choice of measurement scale — is a portrait of small-study bias with the labels rearranged. Alonso-García's k = 10 open-access-only search returns SMD 0.93 with a prediction interval touching zero.
• The most awkward result in the subject is that unplugged activities beat screen-based ones in the early-childhood meta-analysis. If the mechanism is learning to program a computer, the arm that does not involve programming a computer should not win.
• Vendor involvement is real and should be named. Code.org's own research lead co-authors an evaluation of a Code.org unit; the CAL curriculum, its app, its coding assessment and its computational-thinking assessment were all built by one group; the TIPP&SEE strategy, curriculum and assessment likewise. None of this makes the findings false. It means every positive estimate here carries the archive's standard developer discount, and the nulls — including the developer-led ones — carry none.
• Half the instructional evidence is undergraduate. Pair programming, Parsons problems and subgoal labelling are all CS1 findings. Whether they transfer to 11-year-olds is untested.
• A whole question is unanswered. The block-to-text transition — the design decision every school CS programme actually faces — has one recent systematic review, and it could not be obtained.

Practical guidance

• Justify computer science on programming, employment and earnings, not on thinking. That is what the evidence supports, and the Maryland natural experiment gives it a defensible economic case.
• Budget about 0.4-0.5 SD on a coding assessment from a twelve-week course, and treat any figure above about 0.5 from this literature as a measurement artefact until shown otherwise.
• Do not agonise over blocks versus text. The pooled contrast is a null with detected publication bias. Pick one and teach it.
• Use Parsons problems for practice volume. Same measured learning in roughly half the time is a real lever, and it is the one instructional finding here with internal replication.
• Use subgoal-labelled worked examples if your problem is the failing tail, not if your problem is the mean. That is what the variance and withdrawal results actually show.
• Demand an assessment the vendor did not write. In this literature that single question is worth more than the entire curriculum-comparison debate.

Open questions

• There is no preregistered, multi-site, independently evaluated RCT of a school computing curriculum anywhere. The field's best evidence is one developer-led cluster trial.
• No credible design exists for coding bootcamps — the searchable literature is alumni surveys and interviews.
• England's 2014 compulsory computing curriculum has never been exploited as a natural experiment, despite being an unusually clean national shock with pre- and post-administrative data.
• Nobody has measured retention of programming skill beyond one week except at end of treatment. Every headline here is acquisition-phase.
• No genetically informative design exists on programming aptitude or on response to programming instruction.