The Impact of the Measures of Academic Progress (MAP) Program on Student Reading Achievement
Cordray D, Pion G, Brandt C, Molefe A, Toby M · 2012
grade Brctindependentreplicatednumbers spot-checked
Sample
32 schools / 3,720 students (grade 4 n=1,914; grade 5 n=1,806) / 172 teachers, 5 Illinois districts
Population
US grade 4 and grade 5 students in 32 Illinois elementary schools; Illinois Standards Achievement Test (ISAT) reading and NWEA MAP reading+language composite outcomes, measured at the end of Year 2 (2009/10).
Design
Cluster RCT of NWEA's MAP interim-assessment programme plus its associated teacher training, using a cross-over design: half the schools implemented MAP in grade 4 and served as the grade-5 control, half the reverse. Both the state test AND MAP's own composite were used as outcomes, which removes the usual 'the test wasn't aligned' defence. Well powered: achieved MDES 0.16 (G4 ISAT), 0.18 (G4 MAP), 0.14 (G5 ISAT), 0.22 (G5 MAP) against a planned 0.20. Grade 4 was the confirmatory question, grade 5 exploratory. IES report (NCEE 2013-4000), no DOI; ERIC ED537982.
Key findings
The market-leading interim-assessment data system produced nothing on the state test (+0.05 G4, -0.05 G5) and nothing on its own composite (+0.07 G4, +0.01 G5), in a trial powered to detect 0.14-0.22. Observed differentiated instruction did not change, though grade-5 teachers said on a survey that it had. Giving teachers more data is not an intervention. Independence caveat: NWEA delivered the training and supplied the MAP data, but the evaluation was run by REL Midwest/Vanderbilt under an IES contract with an explicit no-financial-interest disclosure, and NWEA's objection to the findings is printed in a footnote rather than accommodated.
Genetic confound
Minimal — school-level randomization.
Replication notes
Same failure mode as the CASL trial: data reached teachers, practice did not change, achievement did not move. NOT verified against an actual replication of THIS trial in the full-text pass — the enum records the archive's prior judgement, not something the report establishes.
Effects
| Outcome | Metric | Value | Measure | Timing | Vs | Horizon | Class |
|---|---|---|---|---|---|---|---|
| State reading achievement (ISAT), grade 4 | effect size | +0.05 (impact 1.29 scale points, SE 1.570, p = .412) — not significant | standardized | end of Year 2 (spring 2010) | business-as-usual | end-of-treatment | domain-skill |
| MAP's own reading + language composite, grade 4 | effect size | +0.07 (impact 0.96, SE 0.891, p = .280) — not significant; the product did not move even its own measure | standardized | end of Year 2 (spring 2010) | business-as-usual | end-of-treatment | domain-skill |
| State reading achievement (ISAT), grade 5 | effect size | -0.05 (impact -1.48, SE 1.366, p = .280) — not significant, and negative in sign | standardized | end of Year 2 (spring 2010) | business-as-usual | end-of-treatment | domain-skill |
| MAP's own reading + language composite, grade 5 | effect size | +0.01 (impact 0.15, SE 1.037, p = .889) — not significant | standardized | end of Year 2 (spring 2010) | business-as-usual | end-of-treatment | domain-skill |
| Teacher use of differentiated instruction | achieved relative strength index (ARSI) | best cross-measure estimate 0.227 (grade 4) and 0.340 (grade 5) — "small differences". Classroom observations and teacher logs showed small non-significant differences; the grade-5 teacher SURVEY composite was significantly higher for MAP teachers (p < 0.001, ARSI 0.894) while the grade-4 survey composite was not. So self-report moved in one grade, observed practice did not. | mixed | end of Year 2 (spring 2010) | business-as-usual | end-of-treatment | behaviour |
| Programme implementation fidelity | dose | moderate — teachers implemented on average 0.66 of the 12 MAP components in both grades; participation by component ranged 36% (web resources) to 90% (using MAP resources to plan lessons) | self-report | across Years 1-2 | none | end-of-treatment | behaviour |
Cited by
- Feedback and formative assessmentmixedconf: highgc: low