Impact Evaluation of Progress Learning in the Douglas County School System

The current study was a retrospective mixed-methods quasi-experimental design (QED) study to determine the effects of Progress Learning on Grades 6-8 mathematics and ELA achievement by comparison growth on the Georgia Milestones Mathematics and ELA assessments of students who received Progress Learning services, in relation to students that did not receive Progress Learning. Supplementary analyses examining the associations between Progress Learning usage metrics and achievement gains are also performed in this study. The results of the main impact analyses showed a positive and statistically significant impact of Progress Learning on student mathematics achievement, with treatment students outgaining comparison students by more than 4 points. The results of the main ELA impact analysis showed a directionally positive, though not statistically significant, impact on ELA achievement, with treatment students outgaining comparison students by more than 3 points. Effect sizes of these analyses ranged between 0.06 to 0.09 SDs, indicating small, though practically meaningful, program impacts of Progress Learning on student achievement, especially in mathematics. Usage analyses showed significant positive associations between student-level Progress Learning usage metrics and achievement gains. Correlations between average Progress Learning activity scores and achievement gains were of particular note, with observed correlations of magnitude above .4 in ELA and above .6 in mathematics. This gives preliminary evidence supporting modest to moderate predictive validity of Progress Learning activity scores in relation to Georgia Milestones scores. These associations remained significant and positive when controlling for prior achievement and demographics, using HLMs similar to those used in the main impact analyses.