Replication of Short-Term Experimental Impacts of Reading Recovery's Investing in Innovation Fund (i3) Scale-Up with Regression Discontinuity

This paper presents evidence confirming the validity of the RD design in the Reading Recovery study by examining the ability of the RD design to replicate the 1st grade results observed in the original i3 RCT focused on short-term impacts. Over 1,800 schools participated in the RD study over all four cohort years. The RD design used cutoff-based assignment established by pre-intervention test scores on the Observation Survey of Early Literacy (OS; Clay, 2005). In order to examine the ability of the RD design to replicate the results observed in the RCT, we estimated impacts for both i3 and non-i3 implementing the RD design, and we compare these results to i3 RCT results across four cohorts. Using multilevel statistical models, the performance of students above and below the cutoff score was compared, with students nested within each participating school. The hierarchical linear model (HLM) used to estimate impacts included the centered pretest assignment variable as a covariate at the student level, a parameter for the discontinuity associated with assignment to RR, a random effect for overall school performance (i.e., a random school intercept), a random effect for the pretest slope (i.e., a random school slope), and a random effect for the impact of RR (i.e., a random treatment effect across schools). In accordance with What Works Clearinghouse (WWC) standards for RD studies (WWC, 2017; Schochet et al., 2010), model fit and potential misspecification was assessed graphically via scatterplots and spline curves and also by testing for an interaction between pretest scores and the treatment assignment variable. Assumptions of linearity in the RD analyses were further assessed by testing polynomial parameters and by imposing various restrictions on the bandwidth around the cutscore. More specifically, analyses were restricted to include only students whose pretest scores fell within ±1.0 or ±0.5 standard deviations of the cutscore. Lastly, robustness of the RD was assessed by arbitrarily shifting the cutscore up or down by 0.50 standard deviations to confirm the absence of a discontinuity where none would be expected. Intent-to-Treat (ITT) effect sizes were calculated by dividing raw impact estimates by the mid-year first-grade population standard deviation of OS scores. Standardized effect sizes based on the RD sample were compared to standardized effect sizes from the previously published RCT study involving i3 schools. The complier average causal effect at the cut-point (CACEC) was created utilizing the formula from Bloom (2009, p. 12, equation 12) for those students whose forcing variable score was within 0.25 standard deviations of the cutscore. Results showed that RD estimates for short-term impacts on OS scores in i3 schools ranged from 0.65 to 0.78 SDs, and from 0.81 to 0.84 SDs in non-i3 schools. These RD estimates are remarkably similar to the RCT estimates of short-term impacts in 1st grade and were highly consistent under numerous model robustness checks (WWC, 2017). The similarity of short-term impact estimates between the RCT and RD designs confirms the validity of the RD design used in this study of Reading Recovery.