Can Repeated Aggregate Cross-Sectional Data Be Used to Measure Average Student Learning Rates? A Validation Study of Learning Rate Measures in the Stanford Education Data Archive. CEPA Working Paper No. 19-08

In this paper we compare two approaches to measuring the average rate at which students learn in a given school or district. One type of measure--longitudinal growth measures--relies on student-level longitudinal data. A second type--cohort growth measures--relies only on repeated aggregated, cross-sectional data. Because student-level data is often not readily available, cohort growth measures are sometimes the only type available. The estimated school and district learning rates reported in the Stanford Education Data Archive (SEDA), for example, are cohort growth measures based on aggregated data. Understanding how much researchers and policymakers can rely on these cohort growth estimates requires one to know how well, and under what conditions, the estimates obtained from this approach align with those based on longitudinal data. In this report we address these questions. We do so by using longitudinal student data from three states (Massachusetts, Michigan, and Tennessee) to construct both average gain score measures (longitudinal growth) and change-in-average measures (cohort growth) for each public school district and school serving students in any of grades 3-8 in the three states. We then compare the two sets of estimates in order to assess how well the latter replicates the former. We do this separately for districts and schools. We find that the longitudinal and cohort growth measures are generally highly correlated in these three states. On average, the cohort growth measures largely rank schools and districts similarly to longitudinal growth measures. The correlations at the district-level (r=0.87) are somewhat higher than the school-level correlations (r=0.80), which reflects the fact that there is less student mobility among districts than among schools. Additionally, in cases where student mobility in and out of schools or districts is high, the measures are less well aligned. Mobility rates are higher, on average, in small schools and districts, schools with long grade spans, and in charter schools. As a result the alignment of the two measures is weaker in these cases. We conclude that the cohort growth measures are useful proxies for longitudinal growth measures in most, but not all cases. [This work was completed in collaboration with Michigan's Center for Educational Performance and Information (CEPI) and the Tennessee Education Research Alliance (TERA).]