The Struggle to Pass Algebra: Online vs. Face-to-Face Credit Recovery for At-Risk Urban Students

1,224 ninth graders (613 assigned online, 611 face-to-face) across 17 Chicago Public Schools high schools, summers 2011 and 2012. 63 school staff participated (34 face-to-face algebra teachers, 30 in-class mentors); Aventa supplied six online teachers.

US ninth graders, roughly 14-15 years old, in Chicago Public Schools, who had failed second-semester Algebra I in their first year of high school - a predominantly low-income, majority Black and Hispanic urban at-risk population. This is the only randomised study in the online-substitution cluster that sits squarely inside the archive's 4-18 age band, and it is also the population on which online credit recovery is actually deployed at national scale.

WHAT "ONLINE" MEANS HERE: Aventa Learning's (later FuelEducation) self-paced, mastery-gated Algebra IB credit-recovery courseware, taken IN a school computer lab during summer school. Instruction came from a remote certified online teacher communicating mostly ASYNCHRONOUSLY (messages and class message boards - the paper notes there was no requirement for synchronous teacher-to-student or student-to-student contact), plus an on-site in-class mentor whose defined role was monitoring attendance and progress and handling technology, NOT teaching maths. Only 53% of mentors were certified in mathematics; their daily logs show 28% of time on instruction-adjacent activity, with additional time on technology support. To progress, students had to pass each quiz; mentors converted Aventa's system scores into final course grades. COMPARISON ARM: the district's standard face-to-face summer Algebra IB class, taught by a CPS-certified maths teacher (all were maths-certified; average 14 years' experience) with substantial discretion over content and pacing. So `baseline: active-alternative` throughout - both arms got a summer algebra course of the same length, and the contrast is a genuinely equal-resourced one. This is the cluster's cleanest randomised test of taking the teacher out of the room. DESIGN: student-level random assignment within school and summer. Outcomes: (a) administrative credit recovery - the study's designated primary outcome; (b) a 28-item algebra posttest administered on paper by the study team, Rasch-scaled, covering pre-algebra, Algebra IA and Algebra IB content, deliberately NOT counted toward the course grade and not aligned to either condition's curriculum; (c) student survey scales; (d) second-year outcomes from district records (PLAN mathematics and its algebra subtest, subsequent maths credits, on-track indicator). ATTRITION - the main reason this is graded B rather than A: 32% of students are missing on the survey and study posttest (68% response, identical across arms), with about 7 percentage points of differential missingness; the extant administrative outcomes have about 2 points of differential missingness, so the primary credit-recovery estimate rests on near-complete data and the learning estimate does not. GRADING IS NOT COMMON ACROSS ARMS - mentors and face-to-face teachers set final grades by different rules, so the credit-recovery gap mixes learning with grading regime; the independently administered posttest moving the same direction is what makes the result credible. FUNDING/INDEPENDENCE: IES grant R305A110149 to AIR; conducted by AIR and the University of Chicago Consortium on School Research in partnership with CPS. Aventa supplied courseware and online teachers but did not run or author the evaluation - independent under the archive's rule.

Students randomly assigned to recover their algebra credit online passed at 66% against 78% face-to-face - a 12 percentage-point gap, d = -0.35 - and scored lower on an independently administered algebra posttest (d = -0.19, consistent across pre-algebra, Algebra IA and Algebra IB item sets). They also reported the course as harder (d = +0.51) and much less clear about what was required to succeed (d = -0.64), and came out liking and trusting their own maths less (d = -0.18). The caveat that most changes how to read it - by the end of the second year of high school none of this was visible - no significant difference on the PLAN mathematics test or its algebra subtest, on passing subsequent maths courses, on cumulative maths credits (2.39 vs 2.51, p = .09) or on being on track for graduation (26% vs 23%). Both groups were performing very poorly in absolute terms (38% and 40% accuracy on the posttest; roughly half of each group earned any later maths credit), which is the uncomfortable frame for the whole result - the face-to-face alternative was not good either, and the online course was worse than a weak comparison. Adding a maths-certified mentor to the online room did not help (68% vs 65% credit recovery).

Minimal (randomised) - students were randomly assigned to condition within school and summer session, so heritable differences cannot differ between arms. The residual threat is differential non-response on the posttest, not selection into treatment.

Directly replicated in design by Rickles, Heppen and colleagues (2023, Journal of Research on Educational Effectiveness 17(3); ERIC ED647309), a multisite RCT in 24 Los Angeles high schools randomising 1,683 students who had failed Algebra 1 or ninth-grade English. The algebra results did NOT reproduce - credit recovery -6 pp (ns) and the algebra posttest +0.15 SD (ns, i.e. nominally favouring online) - while English 9 credit recovery was -23 pp, an effect size of about -0.58, statistically significant. The 2023 authors state plainly that these two studies are the only experimental tests of online credit recovery in existence and that they cannot say whether the divergence is due to provider or instructional model. The difference in model is the most instructive thing about the pair and belongs in any synthesis: in Chicago the teaching role was genuinely REMOVED from the room (a remote online teacher plus an on-site "mentor" with no instructional expectation, only 53% of whom were maths-certified), whereas in Los Angeles there was NO online instructor and the in-class teacher was expected to provide instructional support. The harm tracks how much of the teacher was removed, not whether the curriculum was on a screen.