Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment
Parker, M. C., Guzdial, M., & Engleman, S. · 2016
grade Dreplicationindependentmixed
Sample
Not confirmed
Population
CS1 students.
Design
A psychometric replication: the SCS1 was built as an isomorphic version of the FCS1 and released for community use, with item-response-theory and classical reliability analysis. Read at secondary depth — the alpha figure below comes from secondary sources rather than the paper body and should be spot-checked before this source is marked verified.
Key findings
The SCS1 returned a Cronbach alpha of about 0.59, which the authors describe as slightly below acceptable against the 0.65 threshold they use, and their IRT analysis concluded that items need improvement in difficulty calibration and discrimination. The field's flagship validated instrument is, on its own authors' account, not quite reliable enough — which is the strongest single reason to distrust effect sizes in this literature.
Genetic confound
Not applicable — instrument validation.
Replication notes
A direct replication attempt on the FCS1 that produced a working instrument with reliability below the authors' own acceptability threshold. A third-generation revision (SCS1Rv2) was reported in 2023 as a reasonable predictor of course outcomes — twelve years after the FCS1 first appeared, the field still lacks a settled reliable language-independent measure.
DOI / URL
10.1145/2960310.2960316
Effects
| Outcome | Metric | Value | Measure | Timing | Vs | Horizon | Class |
|---|---|---|---|---|---|---|---|
| SCS1 internal consistency | Cronbach alpha | about 0.59, below the 0.65 acceptability threshold the authors use | validated-instrument | not applicable | none | not-applicable | domain-skill |
Cited by
- Does teaching programming actually teach programming — and does the method matter?moderate supportconf: mediumgc: low