Effects of Online Credit Recovery on High School Credit Accumulation and Graduation: Evidence from a Multisite Randomized Study

Background: Over the past decade, schools across the United States increasingly turned to online learning (Gemin, Pape, Vashaw, & Watson, 2015; Queen & Lewis, 2011). While many espouse the promise of online credit recovery (OCR) courses, the abrupt shift to online learning during the COVID-19 pandemic renewed concerns about theses courses' effectiveness as tools for equitable outcomes (Tate & Warschauer, 2022). These concerns are exacerbated by a dearth of rigorous and mixed evidence about the effectiveness of OCR courses (Viano, 2018). A correlational study of OCR in Florida found that students in online courses were more likely to earn a C or better than students in face-to-face courses (Hughes, Zhou & Petscher, 2015). A study of North Carolina high school students who failed a course found that students who enrolled in OCR were more likely to graduate but had lower test scores than students who repeated the face-to-face version of the course (Viano & Henry, 2020). In addition, Heinrich et al. (2019) found that in a large urban midwestern district "online course-taking is not benefiting students or reflecting real learning" (p. 2,174). To date, there have been two randomized controlled trials of OCR. The first study found that students who took an OCR course in Chicago were less likely to earn credit and learned less than students who took a face-to-face course (Heppen et al., 2017), but found no statistically significant differences in longer-term outcomes, including high school graduation (Rickles et al., 2018). The second study found that students who took an OCR course in a large, urban school district were, on average, less likely to earn credit for ninth grade English than students who took a teacher-directed course but found no statistically significant differences in credit recovery rates in Algebra 1 or in content knowledge across both subject areas (Rickles et al., in press). Purpose: This paper builds off the second randomized study to examine how OCR affects longer-term high school outcomes relative to the business-as-usual (BAU) teacher-directed credit recovery classes. We address the following research questions: (1) What was the impact of taking OCR on students' credit accumulation over four years of high school?; (2) What was the impact of taking OCR on students' probability of on-time high school graduation?; (3) How did the longer-term effects of OCR vary across sites?; and (4) How did students' prior academic performance and engagement moderate the longer-term effects of OCR? Intervention: The intervention for this study was an Algebra 1 or English 9 online curriculum for the credit recovery course, where an online provider supplied the main course content and curriculum, and the school provided the credentialed in-class teacher who could supplement the digital instruction. The BAU classes primarily relied on traditional teacher-directed instruction, where teachers had latitude in the curriculum and instructional materials for the class. For both the intervention and BAU conditions, students took the class within a standard classroom during the district's 5-week summer session. Key characteristics of the online learning model tested in this study are presented in Exhibit A.1. The theory of action for the study is presented in Exhibit A.2. Population and Setting: The analyses in this paper are based on 613 students in 28 Algebra 1 classes across 13 high schools and 1,124 students in 70 English 9 classes across 19 high schools, all within a large urban district. All students entered 9th grade in the 2017-18 school year (Cohort 1) or the 2018-19 school year (Cohort 2) and failed their Algebra 1 and/or at least one semester of their English 9 course and enrolled in a credit recovery course the summer between their 9th and 10th grade year. For both cohorts, the credit recovery classes took place before the COVID-19 pandemic. Exhibit A.3 presents descriptive statistics and baseline equivalence for the Algebra 1 and English 9 student samples. Research Design: Students were randomly assigned to take their credit recovery course in an online class (treatment) or a BAU class (control). Random assignment took place within blocks defined by school, course, and cohort. In some schools, blocks were further defined by which semesters of the course the students failed during their 9th grade year. Data Collection and Analysis: We utilized extant district student background and course data for all analyses in this paper. Measures are described in Exhibit A.4. All analyses were conducted separately for Algebra and English. We estimated average treatment effects for the intent-to-treat student sample using regression models that control for student characteristics and randomization blocks. For the analysis of between-site heterogeneity, we estimated site-specific treatment effects by adding site-by-treatment interaction terms (fixed effects) to the main impact model specification. For the student-level moderator analysis we estimated separate moderation models for two indicators of prior academic performance (8th grade assessment scores and 9th grade GPA) and an indicator of school engagement (9th grade attendance). Findings: We are conducting analyses for this paper and therefore do not currently have findings to present. We expect to have full results by August 2023. Conclusions: The findings will provide needed evidence about the relative effectiveness of OCR to keep students on track toward graduation over time and obtain a high school degree. While earlier results from this study indicate that the online classes were more difficult to pass but may have improved short-term content knowledge for certain students in certain subject areas, it is not clear whether those proximal effects had implications for students' progress toward graduation. To help interpret the findings, the paper will use data from a larger sample of the district's students, focus group data from participating teachers, and interview data from students who took credit recovery during the pandemic to help situate the impact findings within the broader context of teaching, learning, and high school progression during the pandemic. Within this context, findings about the longer-term effects of OCR should inform future efforts to address the needs of students who fall behind early in their high school career.