Research on the Use of Khan Academy in Schools: Implementation Report

Nine sites, 20 schools, more than 70 teachers, about 2,000 students in each of two study years (grades 5-10). The quantitative sub-analyses are far smaller - Site 1 fifth grade n = 435 and sixth grade n = 415 for the achievement association; Site 9 grade-level cells of n = 51 to 100; the attitude analysis n = 856 / 838.

US grades 5-10 mathematics, SY 2011-12 and SY 2012-13. Nine VOLUNTEER sites spanning public, charter and independent schools; one public elementary district (Site 1) contributed 8 schools, 50+ teachers and over 1,000 students, and the other eight sites ranged from a single teacher to five or six. A majority of sites served low-income communities and several were using Khan Academy specifically with their lowest-attaining students.

SRI SAYS IN ITS OWN WORDS THAT THE QUESTION CANNOT BE ANSWERED FROM THIS STUDY, and that is the reason to keep the record: "SRI conducted an implementation study rather than an evaluation of Khan Academy's impact. An experimental test of an intervention's impact (a randomized control trial) would have required a clearly specified treatment, including a protocol for its enactment. Because neither the Khan Academy resources and tools nor the way in which they were used in classrooms was stable across the various study sites and across the 24 months of this work, it was too soon to attempt a rigorous evaluation." Three structural reasons an impact estimate was impossible: (1) the product changed continuously DURING the study - Khan Academy was actively iterating its content, coach dashboards, learning-flow redesign and Common Core mapping in response to feedback from the study schools, so there was no fixed treatment; (2) use models varied wildly across the nine sites and changed within them; (3) at all but one site Khan Academy was SUPPLEMENTARY, so "the effects of Khan Academy cannot be separated from those contributed by other elements of the math curriculum." All sites were volunteers, and there was no comparison group of any kind. THE QUANTITATIVE ANALYSES THAT DO EXIST ARE CORRELATIONAL BY DESIGN: fall prior achievement and fall attitude measures were used to predict spring outcomes, students were then split into better-than-predicted and worse-than-predicted groups, and the two groups' Khan Academy usage was compared. That is a within-student association between usage and residual achievement, and the direction of causation is unidentified - as SRI states, students doing better may simply have been able to complete more problem sets. SRI's own caption: "These analyses are exploratory, and the results preliminary; they cannot be used to make definitive claims about the effectiveness of Khan Academy resources. Other plausible explanations could account for these associations that the analyses did not consider." THE ONE LARGE EFFECT SIZE IN THE REPORT (+0.61 and +1.03) IS A HISTORICAL-COHORT COMPARISON at a single school with a bundled intervention and must never be quoted as a Khan Academy effect - see the effects rows. INDEPENDENCE: SRI International ran and authored the study, so `independent` under this archive's rule; funding came from the Bill & Melinda Gates Foundation, and Khan Academy was an active co-development partner throughout, which is a formative-evaluation structure rather than an impact-evaluation one. SOURCE TYPE: the controlled vocabulary has no slug for "implementation study"; `longitudinal` is the least wrong, since the quantitative content is two years of within-site prior-achievement-controlled prediction.

The single most widely deployed piece of instructional software in the world was studied for two years across 20 schools by an independent research organization, and the study produced NO causally interpretable estimate of its effect on achievement - because, SRI says, the product was changing while it was being studied, use models were incompatible across sites, and Khan Academy was one supplement among many at all but one site. What the study did produce is positive correlational associations at two sites: at Site 1, fifth graders whose spring California Standards Test scores exceeded prediction had spent about 12 more hours on Khan Academy across the year and completed 26 more problem sets (+39%) than fifth graders who under-performed prediction; sixth graders about 3 more hours and 20 more problem sets (+22%). Similar associations appeared for self-reported math anxiety, math self-concept and academic efficacy (10-20% more problem sets), significant for self-concept and efficacy. Direction of causation unidentified. Teacher sentiment was strongly positive (89% planned to continue; 65% said it improved their ability to help struggling students catch up), and student engagement was high. The +0.61 and +1.03 effect sizes that circulate from this report are a before/after comparison of successive cohorts at ONE charter high school which introduced Khan Academy as part of an entire new math programme - they measure the programme, not the software, and have no control group.

High. No randomization, no comparison group, volunteer sites; the achievement analyses correlate usage with residual achievement after prior-achievement adjustment, which does not address the possibility that more able or better-supported students both use the software more and outperform prediction for reasons unrelated to it.

There is no causal estimate in this study to replicate, which is the finding. A decade on, the most widely deployed piece of instructional software in the world still has no well-designed independent impact evaluation on file in this archive. That absence, not any number, is what this record exists to state.