Search

Union County Public Schools (UCPS) received a U.S. Department of Education Teacher and School Leader Incentive Program grant in 2017, supporting the EPIC program aimed at transforming education in 13 low-performing, high-poverty schools. The EPIC initiative had two primary goals: redesigning the Human Capital Management System and Performance-Based Compensation System to attract, retain, and sustain effective educators, and enhancing teacher and school leader effectiveness to positively impact student academic achievement. UCPS implemented a Performance-Based Compensation System (PBCS) that provided monetary incentives to individual teachers and school leaders based on attendance, growth targets, and evaluations. The EPIC impact study evaluated the program's effects on student outcomes using a short-interrupted time series design with comparison (SIT-C), comparing standardized achievement scores in math and reading for Grades 5 and 8, and high school levels, with matched comparison schools. The study spanned five years, tracking proficiency scores before and after EPIC implementation. Results at the elementary level showed that neither the treatment nor comparison groups achieved the projected mean percent proficient in post-intervention years for math. However, the treatment group exceeded the mean in reading, unlike the comparison group. In middle schools, the treatment group surpassed the projected mean percent proficient in both math and reading, while the comparison group did not. At the high school level, neither group achieved the projected mean percent proficient in math post-intervention, but the treatment group exceeded it in English compared to the comparison group. Teacher outcomes indicated improvements in retention and diversity across EPIC schools. Seven of the 13 schools saw a substantial increase in teacher retention, and the percentage of teachers of color increased by 10 points over five years. Teacher attendance showed improvement, with a decrease in average days absent. UCPS's EPIC program demonstrated positive outcomes in teacher retention, diversity, and attendance, with varying impacts on student proficiency across grade levels. The study emphasized the need for cautious interpretation of data due to external factors like the COVID pandemic. [The report was submitted to Union County Public Schools.]

PREPARE, implemented by High Point University, was a Teacher Quality Partnership grant offering a Master of Education for elementary education, secondary mathematics, and special education (in grant years 2-5). The residency model included a living wage stipend to support time spent earning the accelerated master's degree. The PREPARE impact evaluation utilized a multi-year, multiple-cohort quasi-experimental design (QED) to examine the effect of a teacher residency master's program on teacher evaluations during the first one to three beginning teacher years (North Carolina Educator Evaluation System, 2 scales) (NCEES) and impact on student academic growth using the education value-added scale (EVAAS). Baseline equivalence was achieved by aligning treatment and comparison beginning teachers through full-method propensity score matching with replacement based on grade level and/or subject area teaching. Results showed that in the treatment group, NCEES mean ratings for both content and pedagogy increased in each subsequent cohort. In the comparison group, mean ratings declined in each subsequent cohort. The improvement in evaluation ratings by cohort is likely a result of improved instruction and returning to in-person coursework and teaching after the isolation of residency students during the COVID-19 pandemic. Students in the PREPARE program Cohort 1 (2020-2021) experienced COVID-related hybrid instruction. Many of these restrictions were lifted for Cohort 2 (2021-2022) and instruction returned to all in-person for Cohort 3 (2022-2023). Results of the analyses of EVAAS data for secondary teachers showed significant effects for the cohort covariate. When an interaction term between the treatment and cohort variables was added to the model, cohort remained significant, but the treatment and interaction terms were not significant. This indicates that beyond the effect of being in the treatment group, the school year of the intervention had an impact on the EVAAS outcome scores. Our implementation analysis found positive results for the preparation and induction support provided by the IHEs. Additionally, the program was implemented with high fidelity in four of the five years. The COVID year was not scored as many of the planned items were either not completed or were modified. Retention of beginning teachers after one year was good at 90% but fell to 73% by the end of year three. Both results were below our targeted threshold of 95% and 85%. Overall, the program successfully graduated 94 teachers. An additional 12 were enrolled in coursework at the time of this report. The partnerships among the two IHEs and two LEAs benefitted the teacher residents, beginning teachers, and the students in the classrooms compared to teachers not trained in PREPARE.

Implemented by Pfeiffer University, REPERTOIRE: Recruiting Educators, Preparing Educators, and Retaining Teachers to Optimize Interest in Rural Education was a collaborative partnership with Montgomery and Stanly County Schools in rural North Carolina. Pfeiffer University and the Division of Education and Division of Arts and Sciences provided undergraduate education for the teacher education program. In addition, the districts implemented a teacher cadet program as a component of a "Grow Your Own" pathway. The REPERTOIRE impact evaluation used a quasi-experimental design (QED) to examine the effect of an undergraduate teacher program to train highly qualified teachers to teach high-need students in rural school districts, along with retaining these teachers for at least four years. In addition, the study assessed the impact of induction and mentoring support as evidence by state teacher evaluations conducted by a principal or assistant principal during the first one to three beginning teacher years. Our beginning teacher sample was too small to measure the impact on student academic achievement or growth. Our undergraduate program graduated 59% of students that earned licensure; however, only 8% of those graduates pursued a teaching position in one of our two districts. A total of 10 teachers from before and during the grant cycle were hired by the two districts. At the start of three years, 75% were still teaching and of those far enough into their career to be counted for a fourth teaching year, just 50% remained in the two districts. The teachers were still teaching, but they had transferred to another district. Challenges with data collection and recommendations for future evaluation studies are discussed. [This report was co-produced by Pfeiffer University.]

The vision for the Leadership Richland One MSAP project was that all students will have the opportunity to succeed academically through a high-quality education in an environment characterized by diverse social, economic, ethnic, and racial backgrounds. To improve student achievement, major project goals and objectives focus on four areas: desegregation, academic success, magnet themes, and professional development. The purpose of the Evidence of Promise Study is to determine how the magnet school program impacted student achievement and reduced minority group isolation. The Leadership Richland One Evidence of Promise Study was designed to answer the following research questions: (1) What effect does the MSAP project have on student academic achievement performance in ELA and math? (Study 1). (2) What effect does the MSAP project have on minority group isolation? (Study 2). For study 1, treatment and comparison groups were assessed for differences in outcomes in math and ELA academic achievement disaggregated by minority and non-minority status. Results for analyses across three different grade level configurations showed significant differences between minority and non-minority students in achievement across both subject areas. Students in the magnet program at Sanders Middle had higher math achievement scores than comparison students after controlling for the impact of minority group status. The significant impact of magnet program participation on achievement cut across both minority/non-minority student status. There was also a significant interaction between minority status and magnet program participation; non-minority students who were in the magnet program had the highest math achievement test scores. Students with the lowest math achievement test scores were minorities who were not in the magnet program. The difference in achievement test scores between minorities and non-minorities was lower when students were in the magnet program, thus reducing the achievement gap in math. For study 2, we assessed whether Richland One School District's minority group isolation (MGI) goal to reduce MGI of black or African American students in the magnet schools was achieved. Reduction of MGI was achieved as each magnet school experienced a decrease in the percent of total school population that was black/African American from the first year of the grant to the sixth year of the grant, as well as a greater percent reduction of black/African American students compared to other schools in the district.

The Lyceum, implemented by Guilford County Schools (GCS), was an arts education program designed to promote arts integration for all GCS students and exposure to Entertainment Arts & Technology for students at Penn-Griffin School for the Arts. The Lyceum impact evaluation used a quasi-experimental design (QED) to examine the effect of Entertainment Arts & Technology and group or private lessons on academic achievement. The study examined outcomes for students in school years 2020-21, 2021-22, and 2022-23. Outcomes for Lyceum (treatment) students were compared to students from the same schools who did not participate in group or private lessons or in the Entertainment Arts & Technology curriculum offered during the three school years (comparison). Regression analysis was conducted in ELA and math for three academic year cohorts of implementation. There were significant differences in ELA achievement between treatment and comparison students only within Cohort One. Results indicated that comparison students had higher scale scores than treatment students. Additional significant results were found with respect to minority group status and gender with girls outscoring boys and with nonminority students outscoring minorities. There were no significant differences in ELA achievement in Cohorts 2 and 3. There were no significant differences in math for any of the three cohorts. Our second research question included regression analyses comparing district data for four subgroup populations in ELA and math. In two years (2022 & 2023), district implementation of arts integration resulted in a smaller decrease in score for English learners when compared to four other districts for math end-of-grade test scores. In 2023, Guilford reported higher mean scores than the comparison districts. In two analyses from 2023, the Lyceum students scored lower than the comparison districts for black students and English learners on end-of-grade ELA scores. The third research question compared our treatment schools to the district ELA and math averages. In all years, comparing baseline and outcome years, only the participating high school exceeded the mean proficiency levels in both subjects. Given the delay in launching the Entertainment Arts & Technology courses in the studio space and the interruptions due to the pandemic, additional data collection and analyses could confirm or expand the impact of arts integration on all district student academic outcomes and for those participating in Entertainment Arts & Technology. While we did find some statistically significant results in our three studies, results should be interpreted cautiously given our small sample size and limited time and intensity of the intervention. [This report was prepared for the Guildford County Schools.]

BoSTEM, implemented by United Way of Massachusetts Bay and Merrimack Valley and Boston Afterschool and Beyond, is a network of STEM-focused after school programs in the Boston area that provide high-quality science, technology, engineering, and math (STEM) opportunities to Boston middle school students. Through an innovative citywide coalition of nonprofits, schools, researchers, and industry partners, BoSTEM aims to close the opportunity and achievement gap for youth traditionally underrepresented in STEM through exciting, hands-on learning and career mentorship. The BoSTEM impact evaluation used a quasi-experimental design (QED) to examine the effect of BoSTEM on social-emotional learning skills. The study examined outcomes for students in school years 2020-21 and 2021-22. Outcomes for BoSTEM students were compared to a sample of students who participated in business-as-usual afterschool STEM programs that did not receive the BoSTEM structure of support that included performance feedback, coaching, and professional development opportunities. Findings show there was no significant difference between BoSTEM and comparison students in development of social-emotional learning skills. Challenges with data collection and recommendations for future evaluation studies are discussed.

EMPOWER was an Investing in Innovation (i3) development grant awarded to Cabarrus County Schools by the Office of Innovation and Improvement, U.S. Department of Education. EMPOWER provided social-emotional, academic, and non-cognitive supports in magnet school settings to students from low-income families. Family engagement and teacher professional learning on cultural responsiveness were key pillars of the EMPOWER program. A dual-language immersion plus rigor model was implemented at treatment elementary schools while a STEM or IB model was implemented at treatment secondary schools. Literacy skills were assessed for the elementary study using the DIBELS standardized assessment. Math achievement was assessed for the secondary study using standardized statewide math test scores. The impact evaluation used a longitudinal quasi-experimental design (QED) to examine the effect of EMPOWER on literacy skills and math achievement. Hierarchical linear modeling was run to analyze outcomes for EMPOWER students compared to a matched sample of comparison students with similar baseline scores who participated in business-as-usual, traditional academic instruction. Results showed no statistically significant impact on literacy skills or math achievement. [The report was submitted to the Cabarrus County (NC) Schools.]

"G[superscript 2]ROW STEM" is an Investing in Innovation (i3) development grant funded by the Office of Innovation and Improvement, U.S. Department of Education. "G[superscript 2]ROW STEM" targets high-need middle school students, particularly females and minorities, that tend to be underrepresented in STEM careers. "G[superscript 2]ROW STEM" focuses on providing students with engaging, hands-on, project-based, extended-learning experiences to inspire interest in STEM and improve achievement. The impact evaluation used a quasi-experimental design (QED) to examine the effect of "G[superscript 2]ROW STEM" on academic achievement in science and math. "G[superscript 2]ROW STEM" student enrollment began during the 2016-17 school year and students were followed for three years, through the 2018-19 school year. Academic achievement outcomes for "G[superscript 2]ROW STEM" students were compared to a matched sample of students within the same grade in the same schools who participated in business-as-usual, traditional academic instruction with no extended learning experiences. Academic achievement for comparison students was also tracked over a three-year period. Results showed statistically significant differences between treatment and comparison groups in achievement test scores across both math and science. Overall, treatment students gained seven NCE score points more than the comparison group students did after up to three years in the "G[superscript 2]ROW STEM" program. This is roughly equivalent to one-third of a standard deviation. Significant differences were also noted for student subgroups with non-minority students realizing slightly higher gains in NCE scores than minority students by the end of the program (over all analyses a difference of 2.25 NCE points).

NExT LEVEL is a National Professional Development grant funded by the Office of English Language Acquisition, U.S. Department of Education. The grant partnered with three school districts and provided introductory training in working with English Learners (ELs) for select mainstream classroom teachers and teacher candidates; continued coursework for teachers to obtain add-on ESL certification; support for parents and families of ELs; and professional development for university teacher educators. The impact evaluation used a quasi-experimental design to determine the impact of the program on EL students' academic achievement in ELA and math. EL students who had a NExT LEVEL teacher for at least one class during the study period were compared to well-matched students in the same districts who did not have a NExT LEVEL teacher at any time during the study period. The two groups were assessed for differences in achievement outcomes on measures of standardized math and ELA after one year and after three years. In terms of ELA outcomes, there was a significant effect of NExT LEVEL after one year; however, the effect was in the opposite direction from what was expected. Comparison students had significantly higher one-year ELA achievement than the NExT LEVEL students. At three years, there was no significant effect of the treatment on three-year ELA outcomes. The one-year effect on mathematics was statistically significant, but with comparison students outperforming their NExT LEVEL counterparts. There was no significant impact on three-year mathematics performance. Several limitations of this study are discussed, including the mediating impact of the COVID-19 pandemic and the lack of data on teacher implementation of NExT LEVEL instructional strategies.

"ACCESS", A Culture Creating Effective Systems for Success, was a three-year Investing in Innovation (i3) development grant funded by the Office of Innovation and Improvement, U.S. Department of Education. The "ACCESS" grant led the way for a district-wide redesign of educational practices based on the novel approach of embedding a culture of technology-based education (tech culture) within rural schools and the broader community, with an intentional effort to record, evaluate, refine, and disseminate an effective model for implementation across the state and nation. The impact evaluation included two studies that examined the effect of "ACCESS" on the academic achievement of students in grades 5-10 as measured by NC standardized End-of-Grade assessments and the American College Testing (ACT) standardized test. The middle grades study (Study 1) used a comparative short interrupted time series (CSITS) design to assess the impact of the intervention on mathematics and reading achievement in grades 5 through 8 after two years of program exposure. The high school study (Study 2) used a quasi-experimental design (QED) to assess the impact of ACCESS on ACT composite scores taken in grade 10 after two years of program exposure. For both studies, we compared the outcomes of ACCESS participants with matched samples of non-participants in other NC school districts. Propensity score matching (PSM) was used to match ACCESS samples with comparison students at baseline; baseline equivalence was established all pre-test assessment measures. Both studies met What Works Clearinghouse (WWC) Group Design Standards with Reservations. The results indicated no statistically significant differences in achievement between the "ACCESS" treatment group and the business-as-usual comparison groups as measured by the ACT composite scores taken in grade 10 or NC End-of-Grades 5-8 standardized reading and mathematics exams. To contextualize the findings, researchers considered the duration of students' exposure to "ACCESS", alignment between technology integration and NC standardized assessments, and contextual factors that may have narrowed the divide between MCS and other school districts from which comparison schools/samples were drawn. The report concludes with suggestions for future research and implications for education policy. [Funding for this report came from the U.S. Department of Education under its Investing in Innovation (i3) initiative within the Office of Innovation and Improvement through Grant U411C140128 to Montgomery County Schools.]

"UNISON" was an Investing in Innovation (i3) development grant funded by the Office of Innovation and Improvement, U.S. Department of Education to Take Stock in Children (TSIC), a state-wide, 501(c)(3) non-profit, private-public partnership in Florida. UNISON provided a whole school mentoring approach and focused on developing students' non-cognitive skills with a focus on improving academic success and attainment thereby impacting graduation rates in low-performing schools. Intervention strategies included a whole-school culture program with clubs and activities focused on sustaining the culture of kindness and compassion, BARR teacher training, block meetings, and implementation of the I-Time curricula, TSIC 1:1 mentoring, group mentoring, summer Leadership Institute, and parent engagement. The grant funded a collective impact coordinator for each county and a college success coach in each school. "UNISON" targeted three treatment schools in northeast Florida, and the impact study included 15 comparison schools using a comparative short interrupted time series (CSITS) to measure the effect on graduation rates after three years of intervention strategies. Using the publicly available graduation rate data from the Florida Department of Education website, baseline equivalence was established based on rural/urban county designations and baseline graduation rates from 2014-15 prior to the intervention. The study found a statistically significant positive difference in the one urban and two rural treatment schools compared to the six urban and nine rural "business-as-usual" comparison schools. The treatment schools gained almost 20 percentage points in their graduation rate during the three-year intervention from 72.43% to 91.73%, while comparison schools moved from 72.98% to 83.40%. [The report was written for Take Stock in Children.]

"Partners Advancing Childhood Education" ("PACE") is an Investing in Innovation (i3) development grant funded by the Office of Innovation and Improvement, U.S. Department of Education. "PACE" is a family engagement program that enhances family engagement skills among educators and leadership skills among parents to support the academic and social-emotional development of children in pre-K through third grade (ages three to eight). The impact evaluation used a quasi-experimental design (QED) to examine the effect of "PACE" participants' mathematics, language, literacy, and social emotional development. "PACE" enrolled three cohorts of students over 3 years (cohort 1: 2014; cohort 2: 2015; cohort 3: 2016) and followed each cohort over a full year (September through September). Outcomes for "PACE" students were compared to a sample of students with similar characteristics who participated in business-as-usual, traditional academic instruction in matched pre-K and elementary programs. Comparison students were recruited using similar methods and time frames as those used for "PACE" students. Using hierarchical linear modeling, we found no statistically significant difference in mathematics, language, literacy, or social emotional development gains among "PACE" students compared to those exposed to business-as-usual. [This report was prepared for the United Way of Greater Atlanta.]

"INSPIRE" is an Investing in Innovation (i3) development grant funded by the Office of Innovation and Improvement, U.S. Department of Education. "INSPIRE" provides an innovative integrated K-12 STEM pipeline approach focused on STEM course content and instructional redesign. The INSPIRE model was implemented in Cabarrus County Schools (CCS), which is among the largest school systems in North Carolina, serving nearly 30,000 students in 39 schools. The impact evaluation included two studies that examined the effect of INSPIRE on mathematics and science achievement as measured by North Carolina standardized End-of-Grades assessments. The elementary study (Study 1) used a three-year, longitudinal, single-cohort quasi-experimental design (QED) to assess the impact of INSPIRE on math achievement at the end of 5th grade after two years of program exposure. The secondary study (Study 2) used an individual-level, longitudinal, randomized controlled trial (RCT) with blocking by school level and cohort to assess the effects of INSPIRE on math and science achievement at the end of 7th and 10th grades after two years of program exposure. For both studies, we compared the outcomes of INSPIRE students with similar students from schools that did not offer a STEM program. For the elementary study, propensity score matching (PSM) was used to match INSPIRE elementary students and comparison student samples at baseline (on pre-test math achievement scores, gender, minority status, and economically disadvantaged status) and baseline equivalence was established all pre-test assessment measures; this study met What Works Clearinghouse (WWC) Group Design Standards with Reservations. For the secondary study, the overall and differential attrition rates were low based on the WWC attrition standards (WWC, 2017); this study met WWC Group Design Standards without Reservations. The results of the elementary study indicated a statistically significant difference between the "INSPIRE" treatment group and the business-as-usual comparison group on the math achievement outcome. Comparison students reported a statistically significant higher increase in math achievement than "INSPIRE" students. Results of the secondary study indicated no statistically significant difference between the "INSPIRE" treatment group and the business-as-usual comparison group on the math and science achievement outcome. The duration of students' exposure to INSPIRE, fidelity of implementation, alignment between PBL instruction and NC standardized assessments, and contextual factors that might have weakened the intervention strength relative to business-as-usual conditions are discussed as possible factors that account for these findings. The report concludes with suggestions for future research and implications for education policy.

"12 for Life" is an Investing in Innovation (i3) development grant funded by the Office of Innovation and Improvement, U.S. Department of Education. "12 for Life" provides a rigorous STEM curriculum, combined with on-the-job-training, work/life skills development, mentoring, and employment opportunities to high school students who are at high risk of dropping out of school. The impact evaluation used a quasi-experimental design (QED) to examine the effect of "12 for Life" on grade point average (GPA), number of suspensions, and incidence of dropping out of school. "12 for Life" students who enrolled in the program during the 2014-15 school year were followed for three years. Outcomes for "12 for Life" students were compared to a matched sample of students with similar risk factors for dropping out of school who started 10th grade in fall 2014 and who participate in business-as-usual, traditional academic instruction in the high school environment. Comparison students were followed for three years. Results showed no statistically significant impact on grade point average (GPA) at the end of 12th grade, number of suspensions, or incidence of dropping out of school.