Search

The Arkansas Department of Education partnered with the Regional Educational Laboratory Southwest to study the feasibility and effectiveness of using brief email and text message communications to increase the number of parent and guardian visits to the Reading Initiative for Student Excellence (R.I.S.E.) state literacy website. In November 2021 the department sent test messages to families to determine the percentage of households with children in kindergarten-grade 6 in Arkansas public schools that had a working email address or cell phone number and whether the percentage differed by school locale (rural or nonrural) or demographic composition (percentage of economically disadvantaged students, Black students and Hispanic students, or English learner students). Subsequently, the study team randomly assigned 700 Arkansas public elementary schools to one of eight conditions, which varied the mode of communication (email only or email and text message), the presentation of information (no graphic or with a graphic), and the type of sender (generic sender or known sender). In January 2022 households with children in these schools were sent three rounds of communications with information about literacy and a link to the R.I.S.E. website. The study examined the impact of these communications on whether parents and guardians clicked the link to visit the website (click rate) and conducted an exploratory analysis of differences in how long they spent on the website (time on page). The study found that nearly all households had a working email address or cell phone number. Households with children in rural schools or schools with higher percentages of economically disadvantaged students were less likely than other households to have a working email address. The percentage of households that had a working cell phone number did not vary by school locale or demographic composition. Nearly one in five households opened at least one email, and nearly 80 percent of households were delivered at least one text message. The click rate was highest for households assigned to receive both emails and text messages, with a graphic, and from a known sender. The higher click rate can be attributed to adding text messages and to sending communications from a known sender. The impact on the click rate was similar for schools in different locales and with different demographic compositions. The Arkansas Department of Education can use the findings from this study to inform decisions about how best to communicate with families about literacy and other evidence-based resources. The findings can also be used by other state and local education agencies to inform decisions about using email or text message communications to share resources with families. [For the Study Snapshot, see ED624335. For the appendixes, see ED624336.]

Louisiana's Believe and Prepare pilot program, supported by grants from the Louisiana Department of Education, aimed to prepare teacher candidates or in-service teachers through a residency with a mentor and a competency-based curriculum. To improve teacher preparation and teacher residencies, state and teacher education leaders in Louisiana sought to better understand the early career outcomes for participants in the pilot program. This study analyzed data for the three cohorts that participated in the program between 2014/15 and 2016/17. A majority (76 percent) of pilot participants were enrolled in a university-based teacher preparation program. The study examined certification, employment, and retention outcomes for a subset of pilot participants who were teacher candidates or early career teachers (together referred to as early career Believe and Prepare pilot participants). About 30 percent of early career Believe and Prepare pilot participants who attained a Level 1 professional certificate in 2015/16-2017/18 were certified in a high-need subject, as defined by the Louisiana Department of Education (middle grades math and science, secondary math and science, or special education), and 28 percent of participants who entered teaching in 2015/16-2018/19 taught in a high-need subject in their first year of teaching. Early career pilot program participants who completed a residency in a primary school were more likely than those who completed a residency in a nonprimary school to attain a Level 1 professional certificate. Participants who completed a residency in a charter school were less likely than those who completed a residency in a noncharter school to attain a Level 1 professional certificate. (Louisiana does not require Level 1 certifications for charter schools.) Of early career Believe and Prepare teachers who entered teaching in 2015/16-2017/18, 89 percent were retained in the state for a second year, 76 percent were retained in the same district, and 71 percent were retained in the same school. Among these teachers the within-state retention rate was lowest for teachers in high-need subjects, and the within-school retention rate was lowest for secondary and middle grades math and science teachers. [For the Study Snapshot, see ED612483. For the appendixes, see ED612484.]

These appendices accompany the report "Final Report of the i3 Impact Study of Making Sense of SCIENCE, 2016-17 through 2017-18." Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment. It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd's Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students' science achievement through improving science instruction. Under this grant, WestEd and Empirical Education also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale-up study of Making Sense of SCIENCE. The impact evaluation, which is the focus of this report, was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016-17 and 2017-18 school years. The study randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual ("control") group, which received the professional learning (delayed-treatment) after the study ended. The study found that Making Sense of SCIENCE had a positive impact on teacher content knowledge (effect size = 0.56, p = .006) and a positive impact on a holistic scale of teacher pedagogical content knowledge (effect size = 0.41, p = 0.026). The study also yielded positive and significant impacts on the amount of time teachers spent on science instruction (effect size = 0.40, p = 0.015) and on the emphasis that teachers placed on NGSS-aligned instructional practices, with statistically significant effect sizes ranging from 0.40 to 0.49 standard deviations. This suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice. In regard to student science achievement, the study did not find statistically significant results for the full sample of students (effect size = 0.06, p = 0.494), or for the sample of students in the bottom third of incoming math and English Language Arts achievement, with effect sizes of 0.22 standard deviations (p = 0.099) and 0.073 standard deviations (p = 0.567), respectively. Notably, with the exception of one negative effect size, additional analyses on student achievement using different measures and samples yielded positive, but not statistically significant, effect sizes ranging from 0.02 to 0.12 standard deviations. Evaluators offer three potential contributors to the findings of limited impact on student science achievement. First, given the timing of the study in relation to the release of the NGSS in 2013, finding a suitable NGSS-aligned student science assessment was a challenge. Second, most study schools and districts had not yet adopted NGSS-aligned curricula and did not have access to NGSS-aligned curriculum resources. Third, the sample of teachers was unstable across the two years, with the percentage of teachers leaving the school congruous to the percentage observed at the national level. [For the full report, see ED609253; for the research summary, see ED609256.]

Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment. It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd's Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students' science achievement through improving science instruction. Under this grant, WestEd and Empirical Education also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale-up study of Making Sense of SCIENCE. The impact evaluation, which is the focus of this report, was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016-17 and 2017-18 school years. The study randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual ("control") group, which received the professional learning (delayed-treatment) after the study ended. The study found that Making Sense of SCIENCE had a positive impact on teacher content knowledge (effect size = 0.56, p = 0.006) and a positive impact on a holistic scale of teacher pedagogical content knowledge (effect size = 0.41, p = 0.026). The study also yielded positive and significant impacts on the amount of time teachers spent on science instruction (effect size = 0.40, p = 0.015) and on the emphasis that teachers placed on NGSS-aligned instructional practices, with statistically significant effect sizes ranging from 0.40 to 0.49. This suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice. In regard to student science achievement, the study did not find statistically significant results for the full sample of students (effect size = 0.06, p = 0.494), or for the sample of students in the bottom third of incoming math and English Language Arts achievement, with effect sizes of 0.22 (p = 0.099) and 0.073 (p = 0.567), respectively. Notably, with the exception of one negative effect size, additional analyses on student achievement using different measures and samples yielded positive, but not statistically significant, effect sizes ranging from 0.02 to 0.12. Evaluators offer three potential contributors to the findings of limited impact on student science achievement. First, given the timing of the study in relation to the release of the NGSS in 2013, finding a suitable NGSS-aligned student science assessment was a challenge. Second, most study schools and districts had not yet adopted NGSS-aligned curricula and did not have access to NGSS-aligned curriculum resources. Third, the sample of teachers was unstable across the two years, with the percentage of teachers leaving the school congruous to the percentage observed at the national level. [For the appendices, see ED609254; for the research summary, see ED609256.]

Empirical Education conducted rapid-cycle evaluations (RCEs) in three districts in the southcentral and southeastern United States. Each study evaluated the impact of the Dynamic Learning Project (DLP) coaching on teachers' and students' impactful use of technology and student achievement. The research questions, samples, and analytic methods varied between districts, due to the availability of data. They found strong evidence of positive impacts of DLP coaching on teacher and student technology usage, as measured by Hapara Analytics measurement of Google Suite usage, in two districts where this data was available. These findings were particularly evident, and consistent in their increased frequency and duration, for metrics that captured digital interactions between students and teachers. In all three districts studied, they found evidence of positive impacts of DLP coaching on student achievement in math and ELA. These findings, across three districts in three states with different outcome measures and varying grade levels, demonstrate the promise of the DLP coaching model. They also see little evidence of differential impacts of DLP coaching for varying student subgroups. This finding demonstrates promising evidence that DLP coaching equitably impacts students across, gender, race, ELL status, and economically disadvantaged status. The impact of DLP coaching on both the short-term and long-term outcomes hypothesized in the program logic model, as well as evidence that the associations between mediating teacher and student edtech outcomes and longer-term student learning outcomes are positive and significant. These findings are consistent with the goals of the DLP project. However, given the small sample sizes and other limitations of the data collection, the authors caution that many results were inconclusive and warrant further investigation.

This paper examines the "ecological validity" of a randomized controlled trial (RCT) conducted in 42 high schools to measure the impact of a content literacy program. This study of comparable schools implementing the same program in different contexts highlights characteristics that are often not attended to in rigorous effectiveness research but are pertinent to understanding the effectiveness and scalability of the program. In the current study, a direct comparison of school achievement was not possible since the outcome measure used in the RCT was researcher-administered. However, the authors are able to look at teacher characteristics associated with higher student achievement in the RCT and they can look at teacher (and principal) characteristics associated with school-level gains in program participation in Scale-up. The authors found that RCT schools had lower levels of program-related characteristics associated with scale-up. Scale-up schools had similar levels of program-related teacher characteristics associated with greater achievement. Schools that were successful in growing participating internally were not more likely to have the characteristics associated with achievement gains. This suggests that program effectiveness in a controlled study may not be indicative of the program's prospects of wider adoption and, in fact, such studies may not create the best conditions for future program sustainability. Tables and figures are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Empirical Education Inc. conducted a quasi-experiment investigating the impact of the Alabama Math, Science, Technology Initiative (AMSTI) training. The study compared classes of fully-trained AMSTI teachers to classes taught by teachers with no AMSTI training within the same schools. The outcome measure was the ACT Aspire in math, science, and reading from spring 2017. In the current study, we compared student achievement for classes of fully-trained AMSTI teachers to classes of teachers with no AMSTI training within the same school. The study used data from the state student information system, aggregated at the class level, including 2016-17 demographic data and spring 2017 ACT Aspire outcomes, as well as training records from the AMSTI database. This report describes the research questions addressed, the research methodology used, the data that was collected from the state, the results of the study, and potential areas for improvement and follow-on research.

Empirical Education Inc. is the independent evaluator of SRI International's 2014 Investing in Innovation (i3) Development grant called Redesigning Secondary Courses to Improve Academic Outcomes for Adolescents with Disabilities and Other Underperforming Adolescents. The goal of the grant is to develop "Enhanced Units" that combine research-based content enhancement routines, collaboration strategy, and technology components for secondary U.S. History and biology classes. This report presents findings of a randomized control trial (RCT) during the 2017-18 school year. The RCT measured the impact of "Enhanced Units" on higher order content skills (as measured through unit tests) in high school biology and U.S. History classes in three districts in Virginia and California. SRI, the Center for Applied Special Education Technology (CAST), and their research and practitioner partners developed "Enhanced Units" (EU) with the goal of integrating research-based content enhancement routines with technological enhancements to improve student content learning and higher order reasoning, especially for students with disabilities or other learning challenges. This study also documents the extent to which the core components of EU were implemented with fidelity. The authors provide descriptive results on classroom practices (as measured by teacher surveys) and contextual factors that support or hinder implementation (as described during teacher interviews). Future improvements to EU should focus on answering the question: "What is/are the best way(s) for teachers to present SIM routines to their students, particularly for students with learning challenges through SIM intervention?"

Empirical Education Inc. is the independent evaluator of WestEd's 2012 i3 Development grant for Internet-based Reading Apprenticeship Improving Science Education (iRAISE). This interim report provides an update on the progress of the randomized control trial (RCT) during the 2014-2015 school year. The RCT measures the impact of iRAISE on student reading literacy, as measured by the Educational Testing Service (ETS) literacy assessment, in high school science classes in 27 schools in Michigan and Pennsylvania. This report provides a summary of the 2013-2014 pilot study and its relationship to the RCT, the design summary and research questions for the RCT, an updated study timeline, preliminary results for fidelity of implementation (FOI), and initial teacher impressions from the monthly teacher surveys.

In October 2010, WestEd's Strategic Literacy Initiative (SLI) won an i3 "Validation" grant to scale up and validate the Reading Apprenticeship (RA) model in three core secondary content area classes: U.S. history, biology, and English language arts. SLI's proposal stated two goals. Goal 1: To transform academic literacy teaching and learning in high school subject areas so that students are able to achieve high standards. Goal 2: To build LEA capacity to disseminate, support, and sustain academic literacy improvement in high school subject areas within and beyond their regions. Goal 1 is being addressed through a longitudinal randomized control trial (RCT) conducted in approximately 40 schools in Pennsylvania and California. Goal 2, the focus of this paper, is being addressed through the Scale-up Study, a formative evaluation of the scale-up process. This five year study spans four states: Utah, Michigan, Indiana, and Pennsylvania (schools other than those participating in the RCT). During the grant period, four consecutive cohorts of teachers and schools were invited to participate in the RAISE initiative. This report focuses on the first cohort of RAISE teachers and schools, who were introduced and trained in Reading Apprenticeship during the 2011-12 school year. Using longitudinal teacher survey data (from AY 2011-12 and 2012-13), this report first examines trends over time of key indicators that participants are taking up RAISE activities and indicators of scale-up outcome variables. Then, this report examines if changes in the indicators of implementation and support from the first year (AY 2011-12) to the second year (AY 2012-13) predict changes in scale-up outcomes over the same period of time. [For "Year 1 Interim Report of Reading Apprenticeship/RAISE Scale-Up," see ED583146.]

In October 2010, WestEd's Strategic Literacy Initiative (SLI) won an i3 "Validation" grant to scale up and validate the Reading Apprenticeship (RA) model in three core secondary content area classes: U.S. history, biology, and English language arts. SLI's proposal stated two goals. Goal 1: To transform academic literacy teaching and learning in high school subject areas so that students are able to achieve high standards. Goal 2: To build LEA capacity to disseminate, support, and sustain academic literacy improvement in high school subject areas within and beyond their regions. Goal 1 is being addressed through a longitudinal randomized control trial (RCT) conducted in approximately 40 schools in Pennsylvania and California. Goal 2, the focus of this paper, is being addressed through the Scale-up Study, a formative evaluation of the scale-up process. This five year study spans four states: Utah, Michigan, Indiana, and Pennsylvania (schools other than those participating in the RCT). During the grant period, four consecutive cohorts of teachers and schools have been invited to participate in the RAISE initiative. The purpose of this interim report is to provide formative feedback to the SLI team, site coordinators, and implementing schools, districts, and states regarding the scale-up process in the first year of implementation. Specifically, this report provides a brief review of the scale-up literature/research and the logic model guiding this evaluation; an overview of key RAISE project events in Year 1; RAISE participation at the teacher, school, district, and state levels from Year 1; and a subset of descriptive statistics from the Year 1 teacher and administrator surveys to assess the extent to which the project's goals are being recognized.