Experimental Evaluation of the Tools of the Mind Pre-K Curriculum: Technical Report
Farran DC, Wilson SJ, Meador D, Norvell J, Nesbitt K · 2015
grade Arctindependentfailed
Sample
Cohort 1 - 60 pre-K classrooms (32 Tools, 28 control) in 45 schools across 5 districts in Tennessee and North Carolina, 866 children with a pretest; Cohort 2 - 20 classrooms, 12 schools, 263 children. Followed to the end of first grade.
Population
United States; pre-K children around age 4, followed through kindergarten and first grade.
Design
The independent evaluation that settled Tools of the Mind, and it is the design the archive asks for - randomisation at SCHOOL level blocked by district, low and balanced attrition (816 to 810 to 778 across three years), business-as-usual control, and standardized direct child assessments (Woodcock-Johnson III achievement plus Peg Tapping, Head-Toes-Knees-Shoulders, DCCS, Copy Design and span tasks for executive function). The developers were sub-awarded to deliver training and coaching, so implementation was developer-supported while evaluation was independent. When Cohort 1 came back null the developers were shown the results and intensified coaching for Cohort 2.
Key findings
At the end of pre-K there was no significant effect on any of six self-regulation or executive-function measures or on the EF composite - DCCS 0.01 (p = .94), Copy Design 0.15, Peg Tapping 0.16, HTKS 0.03 - nor on any achievement measure. Every statistically significant effect that did appear FAVOURED THE CONTROL GROUP - end-of-pre-K oral comprehension, end-of-kindergarten letter-word identification, quantitative concepts and the achievement composite, end-of-kindergarten backward digit span, end-of-first-grade Copy Design and the self-regulation composite, and end-of-first-grade spelling. Cohort 2, run after the developers intensified coaching, produced the same result - no difference in achievement, self-regulation or teacher ratings at the end of the pre-K year. Implementation fidelity varied but was NOT associated with greater gains in achievement or self-regulation.
Genetic confound
Low. School-level randomisation blocked by district.
Replication notes
This is the failed replication of Barnett 2008 and Diamond et al. 2007. Reinforced by Solomon et al. 2018 in Canada with an active control and prospective registration, and published in peer-reviewed form as Nesbitt and Farran 2021.
Effects
| Outcome | Metric | Value | Measure | Timing | Vs | Horizon | Class |
|---|---|---|---|---|---|---|---|
| Executive function and self-regulation at end of pre-K | effect size | no significant effect on any of six measures or the composite (DCCS 0.01, Copy Design 0.15, Peg Tapping 0.16, HTKS 0.03) | standardized | end of pre-K | business-as-usual | end-of-treatment | near-transfer |
| Achievement at end of pre-K (Woodcock-Johnson III) | effect size | null throughout; letter-word -0.11, applied problems 0.03, picture vocabulary -0.05 | standardized | end of pre-K | business-as-usual | end-of-treatment | domain-skill |
| Statistically significant effects across all waves | direction | every one favoured the CONTROL group, through end of first grade | standardized | end of pre-K, kindergarten and first grade | business-as-usual | over-2yr | domain-skill |
| Cohort 2 after intensified developer coaching | result | same as cohort 1 - no difference in achievement, self-regulation or teacher ratings | standardized | end of pre-K | business-as-usual | end-of-treatment | near-transfer |
| Implementation fidelity and child outcomes | association | fidelity was not associated with greater gains | standardized | end of pre-K | business-as-usual | end-of-treatment | near-transfer |
Cited by
- Can executive function be trained, and does it transfer to learning?no effectconf: highgc: low