Evaluating Student Performance amidst the COVID-19 Pandemic Using Propensity Score Matching

The purpose of the current research study was to investigate the extent to which student performance was influenced by the disruption to learning that occurred due to the COVID-19 pandemic. To explain observed differences in performance, we compared outcomes from a state-wide summative testing program between students who were administered the test in 2019 and 2021. Because the background and demographic characteristics of the two testing populations differed on several variables, including race/ethnicity, locale, and subgroup membership, propensity score matching was used to develop comparable groups for analyses. Independent samples t-tests were then conducted to assess whether there were significant differences in mean scaled scores and domain subscores between the two matched groups. In addition, the same procedures were used to develop and analyze differences for specific student subgroups (students with disabilities, economically disadvantaged students, and English language learners). Overall, results indicated that the 2021 test-takers earned lower scores than students who tested in 2019 for all subjects and grade levels, except English Language Arts (ELA) grade 8. Moreover, larger score differences were observed for mathematics than ELA. Lastly, results from subgroup analyses indicated that in mathematics all subgroups earned significantly lower scores in 2021 than 2019, whereas in ELA grades 5 through 7, English Language Learners and economically disadvantaged subgroups earned lower scores in 2021 than 2019. The results of this study suggest that after controlling for differences between the 2021 and 2019 populations, student performance was likely influenced by the disruption to learning due to pandemic-related factors.