Search

The Arkansas Department of Education partnered with the Regional Educational Laboratory Southwest to study the feasibility and effectiveness of using brief email and text message communications to increase the number of parent and guardian visits to the Reading Initiative for Student Excellence (R.I.S.E.) state literacy website. In November 2021 the department sent test messages to families to determine the percentage of households with children in kindergarten-grade 6 in Arkansas public schools that had a working email address or cell phone number and whether the percentage differed by school locale (rural or nonrural) or demographic composition (percentage of economically disadvantaged students, Black students and Hispanic students, or English learner students). Subsequently, the study team randomly assigned 700 Arkansas public elementary schools to one of eight conditions, which varied the mode of communication (email only or email and text message), the presentation of information (no graphic or with a graphic), and the type of sender (generic sender or known sender). In January 2022 households with children in these schools were sent three rounds of communications with information about literacy and a link to the R.I.S.E. website. The study examined the impact of these communications on whether parents and guardians clicked the link to visit the website (click rate) and conducted an exploratory analysis of differences in how long they spent on the website (time on page). The study found that nearly all households had a working email address or cell phone number. Households with children in rural schools or schools with higher percentages of economically disadvantaged students were less likely than other households to have a working email address. The percentage of households that had a working cell phone number did not vary by school locale or demographic composition. Nearly one in five households opened at least one email, and nearly 80 percent of households were delivered at least one text message. The click rate was highest for households assigned to receive both emails and text messages, with a graphic, and from a known sender. The higher click rate can be attributed to adding text messages and to sending communications from a known sender. The impact on the click rate was similar for schools in different locales and with different demographic compositions. The Arkansas Department of Education can use the findings from this study to inform decisions about how best to communicate with families about literacy and other evidence-based resources. The findings can also be used by other state and local education agencies to inform decisions about using email or text message communications to share resources with families. [For the Study Snapshot, see ED624335. For the appendixes, see ED624336.]

Regional Educational Laboratory Southwest partnered with the Arkansas Department of Education (ADE) to examine Arkansas's middle and high school indicators of postsecondary readiness and success, building on an earlier study of these indicators (Hester et al., 2021). Academic indicators include attaining proficiency on state achievement tests, grade point average, enrollment in advanced courses, and community service learning. Behavioral indicators include attendance, suspension, and expulsion. Using data on statewide grade 6 cohorts from 2008/09 and 2009/10, the study examined the percentages of students who attained the readiness and success indicators and the percentages of students who attained postsecondary readiness and success outcomes by gender, race/ethnicity, eligibility for the National School Lunch Program, English learner student status, disability status, age, and district locale. The study also examined whether the predictive accuracy, specificity, and strength of the indicators varied by these student groups. Three key findings emerged. First, the attainment of indicators of postsecondary readiness and success differed substantially for nearly all student groups, with the number of substantial differences on academic indicators exceeding those on behavioral indicators. The largest number of substantial differences in the attainment of academic indicators were between Black and White students, between students eligible and ineligible for the National School Lunch Program (an indicator of economic disadvantage), and between students who entered grade 6 before and after age 13. Second, attainment of postsecondary readiness and success outcomes varied substantially across student groups, with the largest differences between students with and without a disability. Third, predictive accuracy (the percentage of students with the same predicted and actual outcomes) and strength (the relative importance of a single indicator) were similar across student groups in most cases. Leaders at ADE and in Arkansas districts can use these findings to identify appropriate indicators of postsecondary readiness and success and to target supports toward student groups who most need them. These findings can help leaders identify and address disparities such as inequitable access to resources and supportive learning environments. [For the appendixes, see ED624015.]

The Texas Education Agency launched the Grow Your Own (GYO) grant program in 2018 to encourage districts to develop or expand existing high-quality education and training courses for high school students and to support district-employed paraprofessionals (including instructional aides and long-term substitute teachers) to pursue certifications that would allow them to enter full-time teaching roles. This study aimed to help state education leaders in Texas understand the progress of districts in implementing the GYO program and the early outcomes of participants. This study analyzed data from 2015/16 through 2020/21 for districts that received GYO funding in the first two grant cycles and districts in the same geographic locales within the same regions that did not receive GYO funding (comparison districts). The study found that the majority of districts awarded a GYO grant were in rural areas and small towns. GYO districts were more likely to have a smaller enrollment and had a higher average percentage of Hispanic students than comparison districts. The findings suggest that the program appeared to meet the Texas Education Agency's goal of providing opportunities to students and paraprofessionals in rural and small school settings and students of color to participate in GYO activities. The study also found that the percentage of students completing education and training courses in GYO districts was low (about 10 percent) during the grant years, and the percentage was similar in comparison districts before and after the grant awards. A disproportionate share of students who completed education and training courses in GYO districts were female. Although it is too soon to tell whether the GYO program will, over time, increase the size and diversity of the state's teacher pool, leaders at the Texas Education Agency can use these early findings to both understand the progress of districts in achieving the GYO grant program aims and help identify aspects of the program that might need further investigation. [For the appendices to this report, see ED623385.]

Digital learning platforms (DLPs) are beginning to become open to research. Specifically, in our work in SEERNet, developers are extending five platforms, each used in either K-12 or higher education by more than 100,000 users, to enable third-party researchers to explore, develop, and test improvements. SEERNet seeks to enable equity-relevant research aligned with the Institute of Education Sciences (IES) Standards for Excellence in Education Research (SEER) principles. It also seeks to support research that is more agile (or rapid), is more open, and scales from research to impacts on practice. We review the emerging tensions among the goal of equity-relevant research and desires for agile, open, and scalable research. We argue that designing and developing technical capabilities for agile, open, and scalable research will not be enough. Based on a series of interviews we conducted with experts in social sciences and equity-focused research, we argue that researchers will have to rethink how they plan and undertake their research. Five shifts could help. First, researchers could deliberately reframe their designs away from a comprehensive, monolithic study to smaller, agile cycles that test a smaller conjecture each time. Second, researchers could shift from designing new educational resources to determining how well-used resources could be elaborated and refined to address equity issues. Third, researchers could utilize variables that capture student experiences to investigate equity when they cannot obtain student demographic variables. Fourth, researchers could work in partnership with educators on equity problems that educators prioritize and want help in solving. Fifth, researchers could acknowledge that achieving equity is not only a technological or resource-design problem, but requires working at the classroom and systems levels too. In SEERNet, we look forward to working with the research community to find ways to address equity through research using well-used digital learning platforms, and to simultaneously conduct research that is more agile, more open, and more directly applicable at scale.

A study was conducted to evaluate the impact of CAPIT Reading on student early literacy achievement. The study utilized a quasi-experimental comparison group design using data from 12 elementary schools in a suburban school district in Oklahoma during the 2019-20 school year. CAPIT Reading is a comprehensive PK-2 literacy solution that includes a digital phonics curriculum and teacher professional development. Four schools used CAPIT to supplement their literacy instruction for kindergarten students while eight schools did not. The study linked CAPIT usage data and district demographic and achievement data to estimate the impact of CAPIT on the Letter Word Sounds Fluency (LWSF) and Early Literacy Composite scores of the aimsweb reading assessment administered by the district in August and January. We found a positive impact of CAPIT Reading on student early reading achievement on the aimsweb assessment for kindergarten students estimated at 4.4 test score points for the aimsweb Early Literacy Composite score (effect size 0.17, p = 0.01) and 7.8 points for the LWSF score (effect size 0.29, p < 0.001). This impact on the LWSF score is equivalent to a 29% increase in growth for the average CAPIT student from the fall to winter tests. We found limited evidence of differential impact favoring student subgroups, meaning that this positive impact for CAPIT users did not vary according to student characteristics such as eligibility for free and reduced-price lunch, race, or gender. We found that the impact on aimsweb overall was marginally greater for special education students by 4.9 points (p = 0.09) and that the impact on LWSF scores was greater for English Language Learners by 7.4 points (p = 0.09). Impact of CAPIT reading does not vary significantly across other student groups. [This paper was produced by Evidentally, Inc.]

Louisiana's Believe and Prepare pilot program, supported by grants from the Louisiana Department of Education, aimed to prepare teacher candidates or in-service teachers through a residency with a mentor and a competency-based curriculum. To improve teacher preparation and teacher residencies, state and teacher education leaders in Louisiana sought to better understand the early career outcomes for participants in the pilot program. This study analyzed data for the three cohorts that participated in the program between 2014/15 and 2016/17. A majority (76 percent) of pilot participants were enrolled in a university-based teacher preparation program. The study examined certification, employment, and retention outcomes for a subset of pilot participants who were teacher candidates or early career teachers (together referred to as early career Believe and Prepare pilot participants). About 30 percent of early career Believe and Prepare pilot participants who attained a Level 1 professional certificate in 2015/16-2017/18 were certified in a high-need subject, as defined by the Louisiana Department of Education (middle grades math and science, secondary math and science, or special education), and 28 percent of participants who entered teaching in 2015/16-2018/19 taught in a high-need subject in their first year of teaching. Early career pilot program participants who completed a residency in a primary school were more likely than those who completed a residency in a nonprimary school to attain a Level 1 professional certificate. Participants who completed a residency in a charter school were less likely than those who completed a residency in a noncharter school to attain a Level 1 professional certificate. (Louisiana does not require Level 1 certifications for charter schools.) Of early career Believe and Prepare teachers who entered teaching in 2015/16-2017/18, 89 percent were retained in the state for a second year, 76 percent were retained in the same district, and 71 percent were retained in the same school. Among these teachers the within-state retention rate was lowest for teachers in high-need subjects, and the within-school retention rate was lowest for secondary and middle grades math and science teachers. [For the Study Snapshot, see ED612483. For the appendixes, see ED612484.]

The Collaboration and Reflection to Enhance Atlanta Teacher Effectiveness (CREATE) program aims to increase teacher effectiveness and retention through fostering spaces that support the development of critically conscious, compassionate, and highly skilled educators. Specifically, CREATE's three-year teacher residency program seeks to recruit, support, and retain a diverse, highly effective, and culturally competent educator workforce that is committed to working in high-needs schools and to improving student outcomes. With funding from the Supporting Effective Educator Development (SEED) grant, this evaluation focused on the effectiveness of CREATE's teacher residency program for three staggered cohorts of pre-service and early career teachers from 2017--18 through 2021--2022. Using a quasi-experimental design, the study compares the experiences and outcomes of (1) CREATE residents and (2) comparison subjects participating in the traditional credentialing pathway, with both groups joining the study when they were enrolled in their final year at Georgia State University's College of Education and Human Development (GSU CEHD). The study followed participants for three years: their field experience/student-teaching year, and their first two years as official teachers of record. Researchers assessed primary impacts after participants completed two years as credentialed teachers. Program implementation and the experiences of early career teachers were measured across all study years. By the end of their second year of teaching, CREATE teachers reported more favorable levels of self-compassion, teacher burnout, and stress management and empathy related to teaching than teachers in the comparison group. The research team did not find a statistically significant impact of CREATE on student achievement in English Language Arts, as measured by the Georgia Milestones assessment in grades 3 through 8. The researchers hypothesize that this may have been impacted by a limited sample size of teachers teaching in the tested grade levels and subject areas. A prior study of CREATE [see ED611803] demonstrated a positive impact of CREATE on uninterrupted graduation and retention of teachers, especially for Black educators, with most of the benefit accrued early through a difference in graduation rates. In the current study, graduation rates were high for CREATE and comparison participants (>90%) and the impact on teacher retention was not replicated. These findings pose questions about the differences in programming, recruitment, and hiring between the years during which each study was conducted, and about how COVID-19 may have affected the dynamics of teacher hiring and retention.

Teacher residencies are surfacing as a promising model for teacher preparation. One such residency program--Collaboration and Reflection to Enhance Atlanta Teacher Effectiveness (CREATE)--seeks to raise student achievement in local high-needs schools by increasing the effectiveness and retention of both new and veteran educators. CREATE aims to achieve this by developing critically-conscious, compassionate, and skilled educators who are committed to teaching practices that prioritize racial justice and interrupt inequities. This quasi-experiment, funded by an Investing in Innovations (i3) grant, follows two staggered cohorts of study participants (CREATE and comparison teachers) for three years per cohort, starting with the first cohort in 2015-16. Confirmatory analyses found no statistically significant effects of CREATE on two Teacher Assessment on Performance Standards (TAPS) ratings for Instructional Strategies (p = 0.221) and Positive Learning Environment (p = 0.192), or on student achievement on ELA (p = 0.454), math (p = 0.569), and general achievement (p = 0.234). However, exploratory analyses (pre-registered) found very promising results showing undisrupted retention over a three-year time period (spanning graduation from Georgia State University's College of Education and Human Development, entering teaching, and retention into the second year of teaching) for the CREATE group, relative to the comparison group (p = 0.038). We also observed that the favorable impact is driven largely by higher continuous retention among Black educators in CREATE relative to those in the comparison group (p = 0.021). The percentages of teachers in CREATE, as averaged across the two study cohorts, who maintain an uninterrupted trajectory of graduating from GSU-CEHD, and taught in their first and second year are 94.8%, 87.4% and 84.6%, respectively. In the matched comparison group the corresponding values are 87.9%, 72.9%, and 68.0%. Among Black teachers in CREATE, the values are 98.6%, 96.3% and 95.5%, respectively. In the matched comparison group of Black teachers, the corresponding values are 85.7%, 68.5%, and 62.8%. These results are further corroborated by statistically significant or marginally statistically significant differential impacts favoring Black teachers on several potential mediators of impact: resilience, self-efficacy, and stress management and empathy related to teaching. The positive findings on retention, particularly for Black educators, are important given that 44% of teachers in Georgia leave the profession within the first five years, with evidence in the literature indicating that in the South, Black teachers experience higher turnover rates than non-Black teachers (Carver-Thomas & Darling-Hammond, 2017). The authors discuss a number of factors that could have led to null impacts on confirmatory outcomes. The lack of variation in teacher performance ratings, which is a known issue in Georgia and in the literature at large (Weisberg et al, 2009; Kraft & Gilmour, 2017) could have contributed to the results for the TAPS ratings. Studies of teacher residency programs also suggest that impact on student achievement might not be present or detected in earlier years of teaching. Additionally, analyses of student achievement in this study were limited by small sample sizes of teachers in tested grades and subjects. The authors hope to address these limitations and to investigate the promising outcomes on retention, especially for Black educators, in current and future studies of CREATE, for which we have secured funding for programming and research through the eighth cohort. [For the appendices, see ED611803.]

These appendices accompany the report "Final Report of the i3 Impact Study of Making Sense of SCIENCE, 2016-17 through 2017-18." Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment. It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd's Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students' science achievement through improving science instruction. Under this grant, WestEd and Empirical Education also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale-up study of Making Sense of SCIENCE. The impact evaluation, which is the focus of this report, was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016-17 and 2017-18 school years. The study randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual ("control") group, which received the professional learning (delayed-treatment) after the study ended. The study found that Making Sense of SCIENCE had a positive impact on teacher content knowledge (effect size = 0.56, p = .006) and a positive impact on a holistic scale of teacher pedagogical content knowledge (effect size = 0.41, p = 0.026). The study also yielded positive and significant impacts on the amount of time teachers spent on science instruction (effect size = 0.40, p = 0.015) and on the emphasis that teachers placed on NGSS-aligned instructional practices, with statistically significant effect sizes ranging from 0.40 to 0.49 standard deviations. This suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice. In regard to student science achievement, the study did not find statistically significant results for the full sample of students (effect size = 0.06, p = 0.494), or for the sample of students in the bottom third of incoming math and English Language Arts achievement, with effect sizes of 0.22 standard deviations (p = 0.099) and 0.073 standard deviations (p = 0.567), respectively. Notably, with the exception of one negative effect size, additional analyses on student achievement using different measures and samples yielded positive, but not statistically significant, effect sizes ranging from 0.02 to 0.12 standard deviations. Evaluators offer three potential contributors to the findings of limited impact on student science achievement. First, given the timing of the study in relation to the release of the NGSS in 2013, finding a suitable NGSS-aligned student science assessment was a challenge. Second, most study schools and districts had not yet adopted NGSS-aligned curricula and did not have access to NGSS-aligned curriculum resources. Third, the sample of teachers was unstable across the two years, with the percentage of teachers leaving the school congruous to the percentage observed at the national level. [For the full report, see ED609253; for the research summary, see ED609256.]

Federal initiatives and other research have led states across the nation to sharpen their focus on teacher evaluation in recent years. In 2009 a seminal report, "The Widget Effect," from The New Teacher Project revealed that in districts using a binary rating system to evaluate teachers, less than 1 percent of teachers received an unsatisfactory rating. The remaining 99 percent were, in effect, like widgets, undifferentiated as individual professionals. Since then, a growing body of research on teacher evaluation systems has indicated that classroom observation ratings often cluster around the middle or high end of evaluation scales. The research has also found that observation ratings are susceptible to several biases, such as incoming student achievement and school, classroom, and rater characteristics. Thus many education decision-makers lack sufficient information to make personnel decisions and to effectively support teacher growth and development. Texas is among the states that have identified teacher evaluation and support as a high priority for education policy. In 2014/15 the Texas Education Agency piloted the Texas Teacher Evaluation and Support System (T-TESS) in 57 school districts. The pilot was followed by a refinement phase in 2015/16 and statewide rollout in 2016/17. During the pilot year teacher overall ratings were based solely on rubric ratings on 16 dimensions across four domains (planning, instruction, learning environment, and professional practices and responsibilities), although T-TESS also includes a student growth measure that was piloted simultaneously. The study examined the statistical properties of the T-TESS rubric from the 2014/15 pilot year to explore the extent to which it differentiates teachers on teaching quality and to investigate its internal consistency and efficiency. The study also explored the relationships between rubric ratings and school characteristics to investigate whether certain types of schools have teachers with higher or lower ratings. The study's findings have several implications for practice and research. The findings suggest that the T-TESS rubric demonstrates potential to be an effective, consistent, and efficient evaluation rubric. Thus, the Texas Education Association and local education agencies have a promising tool for providing evidence-based feedback and targeted professional development. Future research could try to validate ratings based on the T-TESS rubric against other measures of teacher effectiveness (for example, student growth). Such validation could shed light on whether a relationship exists between rubric ratings and a teacher's contribution to student achievement. Moreover, future studies could explore whether relationships exist between the T-TESS rubric and classroom and district characteristics. Such analysis may unearth the extent to which effective teachers are equally distributed within schools and within and across districts. Finally, further research could explore whether implementing teacher evaluation systems translates into improvements in teacher effectiveness or in long-term outcomes, such as teacher retention and student achievement.

Recruiting and retaining effective teachers are serious concerns throughout Oklahoma. The Oklahoma State School Boards Association (2016) reported 500 teacher vacancies at the beginning of the 2015/16 school year, according to a survey of school districts, and 53 percent of respondents said the teacher shortage was worse than in the previous year. For years, Oklahoma rural school district administrators have reported difficulty retaining teachers who could cross state lines for higher pay and lower class sizes or seek employment in other industries. In 2013 the Oklahoma State Superintendent of Public Instruction established the Oklahoma Educator Workforce Shortage Task Force to recommend measures to alleviate the "significant and widespread shortages" of classroom teachers. The task force was succeeded in September 2015 by the Teacher Shortage Task Force, which was established to identify and recommend successful strategies for curbing the statewide teacher shortage crisis and which recommended several strategies for placing highly qualified teachers in all Oklahoma classrooms. The state's teacher shortage, as well as the unique context of rural schools in Oklahoma, led members of the Regional Educational Laboratory Southwest Oklahoma Rural Schools Research Alliance to seek information about factors associated with successful teacher recruitment and retention in Oklahoma. The goal was to develop effective strategies for recruiting and retaining teachers in rural schools. In response, this study identified factors that can support teacher recruitment and retention, particularly malleable factors that can be controlled through policies and interventions. This report refers to these factors as indicators of the characteristics of teachers or districts that predict successful teacher recruitment and retention. While associations between indicators and outcomes cannot be interpreted as causal--a specific indicator is not necessarily the cause of a related outcome--the results from this study can be used to pinpoint potential problems and inform future policies. The results can also provide a rationale for experimental evaluations of programs aiming to improve teacher recruitment and retention. The study first explores patterns of teacher job mobility in Oklahoma, including teachers' probability of remaining employed in the same district for a given number of years, the proportion of teachers who leave rural school districts and move to another rural school district, the proportion of teachers who receive tenure, and the one year retention probability for each successive year of employment. Patterns of teacher job mobility are examined for any differences between rural and nonrural school districts. The study was designed to identify teacher, district, and community characteristics in rural Oklahoma that predict which teachers are most likely to be successfully recruited (defined as having completed a probationary period of three years and obtained tenure in their fourth year of teaching) and retained longer term (defined as the duration of employment of tenured teachers in a given school district). This study covers the 10 school years between 2005/06 and 2014/15 and uses teacher and district data from the Oklahoma State Department of Education, Oklahoma Office of Educational Quality and Accountability, and community characteristics from data in federal noneducation sources and publicly available geographic information systems from Google Maps.

Empirical Education conducted rapid-cycle evaluations (RCEs) in three districts in the southcentral and southeastern United States. Each study evaluated the impact of the Dynamic Learning Project (DLP) coaching on teachers' and students' impactful use of technology and student achievement. The research questions, samples, and analytic methods varied between districts, due to the availability of data. They found strong evidence of positive impacts of DLP coaching on teacher and student technology usage, as measured by Hapara Analytics measurement of Google Suite usage, in two districts where this data was available. These findings were particularly evident, and consistent in their increased frequency and duration, for metrics that captured digital interactions between students and teachers. In all three districts studied, they found evidence of positive impacts of DLP coaching on student achievement in math and ELA. These findings, across three districts in three states with different outcome measures and varying grade levels, demonstrate the promise of the DLP coaching model. They also see little evidence of differential impacts of DLP coaching for varying student subgroups. This finding demonstrates promising evidence that DLP coaching equitably impacts students across, gender, race, ELL status, and economically disadvantaged status. The impact of DLP coaching on both the short-term and long-term outcomes hypothesized in the program logic model, as well as evidence that the associations between mediating teacher and student edtech outcomes and longer-term student learning outcomes are positive and significant. These findings are consistent with the goals of the DLP project. However, given the small sample sizes and other limitations of the data collection, the authors caution that many results were inconclusive and warrant further investigation.

Regional Educational Laboratory Southwest conducted this literature review to identify malleable factors that can be measured in K-12 settings and that predict students' postsecondary science, technology, engineering, and math (STEM) success (defined as enrolling in, persisting in, and completing a postsecondary STEM major or degree), particularly for Hispanic students. The review found that courses taken in high school and interest or confidence in STEM were strong predictors of postsecondary STEM success for students of all racial/ethnic subgroups. Yet racial/ethnic minority students were less likely than White students to take the highest level math and science courses, and despite similar levels of STEM interest, racial/ethnic minority students were less likely to achieve postsecondary STEM success. Other indicators of postsecondary STEM success included high school grade point average, class rank, math and science achievement, and SAT or ACT scores. Grades in math and science courses were found to be less predictive of postsecondary STEM success for racial/ethnic minority students than for White students. [For the full report, see ED565641.]

The number of jobs in science, technology, engineering, and math (STEM) is growing rapidly and is expected to increase by approximately 1 million in the United States between 2012 and 2022 (Vilorio, 2014). People of many racial/ethnic minorities, however, including Hispanic people, are underrepresented among recipients of STEM degrees and among employees in STEM fields (Beede et al., 2011). Regional Educational Laboratory Southwest conducted this review of the research literature to identify malleable factors that can be measured in K-12 settings and that predict students' postsecondary STEM success (defined as enrolling in, persisting in, and completing a postsecondary STEM major or degree), particularly for Hispanic students. Identifying these predictive malleable factors can help policymakers and district and school administrators develop and implement interventions that increase the percentage of Hispanic students succeeding in postsecondary STEM majors and pursuing STEM careers. This report addressed two research questions: (1) What K-12 indicators predict postsecondary STEM success?; and (2) To what extent do K-12 indicators of postsecondary STEM success differ for Hispanic and non-Hispanic students? Key findings from this literature review include the following: (1) The number of high school math and science courses taken and the level of those courses predict postsecondary STEM success for all student subgroups; (2) Interest or confidence in STEM showed statistically significant predictive relationships with students' postsecondary STEM success; (3) Indicators related to achievement in middle school and high school predicted students' postsecondary STEM success; (4) Statistically significant high school predictors of postsecondary STEM success included schools' academic rigor, percentage of students enrolled in college preparatory programs, students' satisfaction with their teachers, and levels of parent participation; (5) Few studies examined K-12 predictors of postsecondary STEM success specifically for Hispanic students. Appended are: (1) About the literature review; (2) Abstracts of reviewed studies; and (3) Detailed findings by study.

This paper examines the "ecological validity" of a randomized controlled trial (RCT) conducted in 42 high schools to measure the impact of a content literacy program. This study of comparable schools implementing the same program in different contexts highlights characteristics that are often not attended to in rigorous effectiveness research but are pertinent to understanding the effectiveness and scalability of the program. In the current study, a direct comparison of school achievement was not possible since the outcome measure used in the RCT was researcher-administered. However, the authors are able to look at teacher characteristics associated with higher student achievement in the RCT and they can look at teacher (and principal) characteristics associated with school-level gains in program participation in Scale-up. The authors found that RCT schools had lower levels of program-related characteristics associated with scale-up. Scale-up schools had similar levels of program-related teacher characteristics associated with greater achievement. Schools that were successful in growing participating internally were not more likely to have the characteristics associated with achievement gains. This suggests that program effectiveness in a controlled study may not be indicative of the program's prospects of wider adoption and, in fact, such studies may not create the best conditions for future program sustainability. Tables and figures are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Empirical Education Inc. conducted a quasi-experiment investigating the impact of the Alabama Math, Science, Technology Initiative (AMSTI) training. The study compared classes of fully-trained AMSTI teachers to classes taught by teachers with no AMSTI training within the same schools. The outcome measure was the ACT Aspire in math, science, and reading from spring 2017. In the current study, we compared student achievement for classes of fully-trained AMSTI teachers to classes of teachers with no AMSTI training within the same school. The study used data from the state student information system, aggregated at the class level, including 2016-17 demographic data and spring 2017 ACT Aspire outcomes, as well as training records from the AMSTI database. This report describes the research questions addressed, the research methodology used, the data that was collected from the state, the results of the study, and potential areas for improvement and follow-on research.

Empirical Education Inc. is the independent evaluator of SRI International's 2014 Investing in Innovation (i3) Development grant called Redesigning Secondary Courses to Improve Academic Outcomes for Adolescents with Disabilities and Other Underperforming Adolescents. The goal of the grant is to develop "Enhanced Units" that combine research-based content enhancement routines, collaboration strategy, and technology components for secondary U.S. History and biology classes. This report presents findings of a randomized control trial (RCT) during the 2017-18 school year. The RCT measured the impact of "Enhanced Units" on higher order content skills (as measured through unit tests) in high school biology and U.S. History classes in three districts in Virginia and California. SRI, the Center for Applied Special Education Technology (CAST), and their research and practitioner partners developed "Enhanced Units" (EU) with the goal of integrating research-based content enhancement routines with technological enhancements to improve student content learning and higher order reasoning, especially for students with disabilities or other learning challenges. This study also documents the extent to which the core components of EU were implemented with fidelity. The authors provide descriptive results on classroom practices (as measured by teacher surveys) and contextual factors that support or hinder implementation (as described during teacher interviews). Future improvements to EU should focus on answering the question: "What is/are the best way(s) for teachers to present SIM routines to their students, particularly for students with learning challenges through SIM intervention?"

In 2012, WestEd received a "Development" grant from the U.S. Department of Education's Investing in Innovation (i3) competition to develop and implement Internet-based Reading Apprenticeship Improving Science Education ("iRAISE"). "iRAISE" was implemented in Michigan and Pennsylvania and was provided to over 100 teachers who served approximately 20,000 students during the grant period. This report presents findings from the randomized control trial of "iRAISE," which took place during the 2014-15 school year and investigated the impact of the program on teacher and student outcomes. Data sources for this report include teacher surveys; PD observations and attendance records; school district student records; and an assessment of students' literacy skills. Despite levels of implementation that did not meet the expectations of the program developers, teachers self-reported that they did change their classroom practice as a result of the "iRAISE" program, and impacts of "iRAISE" were greater for students who were performing at lower levels of incoming achievement. Given that "iRAISE" had an impact on teacher practices in literacy instruction and increased benefits for low-achieving students-consistent with positive findings from prior studies, evaluators express confidence in the promise of low-cost, accessible, and high-quality online-only professional Development (PD), addressing the needs of schools struggling to meet the demands of literacy for college and career readiness. Appended are the following: (1) Considerations for Statistical Power; (2) Details of the Approach to Estimating Impacts; (3) Reporting the Results; (4) A Post-Experimental Method to Assessing Impact under Strong Implementation; (5) Fidelity of Implementation; and (6) Teacher Survey Constructs.

We report on the scaling up of a high school content literacy framework, Reading Apprenticeship, over a period of four years as part of the independent evaluation of an Investing in Innovation grant from the U.S. Department of Education to WestEd's Strategic Literacy Institute (SLI). Our goal was to understand the school processes that support successful implementation and promote scaling of an innovation. We focused on teachers, principals and other school-level program leadership who were also the focus for SLI's innovation designed to support scale. Our work was guided by research literature that provided insights into scale-up as increasing local ownership and depth of commitment, as well as growth in numbers. We provided formative feedback to SLI while contributing to the empirical methods and evidence for studying scale-up processes. Over the five years of scaling up, the RAISE project (for Reading Apprenticeship Improving Secondary Education) reached approximately 533,000 students, and 1,719 teachers in 239 schools in Pennsylvania, Michigan, Indiana, and Utah. Surveys of teachers and principals, as well as records of participation in project activities provided a rich longitudinal dataset. In addition to survey analysis, four schools served as case studies where school leaders and program support staff were interviewed over two years to offer a concrete picture of decision processes and resources. We used descriptive statistics to track changes over time in features of program implementation, as well as resulting changes in attitudes and commitment among teachers. In the first year, teachers were successfully integrating Reading Apprenticeship practices and cited collaboration and support from other teachers as most effective for capacity building. We found that uptake of RAISE activities leveled off by the third year. We also found a greater spread in responses across schools by the third year, calling for further investigation into school-level processes and the characteristics of schools that may be associated with the gain or loss of RAISE participants over time. Regression methods were used to identify predictors of increased or decreased participation. We found that teacher participation in team meetings during the first year and school-wide commitment predicted the increase in number of teachers participating in a school. Increases were unrelated to demographics and resources available to the school. We hypothesized that a process, consistent with SLI's innovation for scaling Reading Apprenticeship, which involved a cross-disciplinary teacher team within the school, led to increased and sustained program participation. The scale-up study was conducted in parallel to a [randomized controlled trial] RCT in which 22 treatment and 20 control schools from Pennsylvania and California participated for three years. We used the predictors of increased participation in comparing the scale-up and RCT treatment group schools as contexts for the implementation and impact of Reading Apprenticeship. Scale-up schools had more of the characteristics associated with growth suggesting greater long term sustainability. We raise the issue of whether the implementation under the constraints of the RCT is a good model for measuring impact in the context of scaling up an innovation. The following are appended: (1) Detailed RAISE Scale-up Logic Model; (2) State Maps of RAISE Scale-up, By Year; (3) Results from GL Analysis; and (4) Survey Questions Used in Chapter 8.

Empirical Education Inc. is the independent evaluator of WestEd's 2012 i3 Development grant for Internet-based Reading Apprenticeship Improving Science Education (iRAISE). This interim report provides an update on the progress of the randomized control trial (RCT) during the 2014-2015 school year. The RCT measures the impact of iRAISE on student reading literacy, as measured by the Educational Testing Service (ETS) literacy assessment, in high school science classes in 27 schools in Michigan and Pennsylvania. This report provides a summary of the 2013-2014 pilot study and its relationship to the RCT, the design summary and research questions for the RCT, an updated study timeline, preliminary results for fidelity of implementation (FOI), and initial teacher impressions from the monthly teacher surveys.

In October 2010, WestEd's Strategic Literacy Initiative (SLI) won an i3 "Validation" grant to scale up and validate the Reading Apprenticeship (RA) model in three core secondary content area classes: U.S. history, biology, and English language arts. SLI's proposal stated two goals. Goal 1: To transform academic literacy teaching and learning in high school subject areas so that students are able to achieve high standards. Goal 2: To build LEA capacity to disseminate, support, and sustain academic literacy improvement in high school subject areas within and beyond their regions. Goal 1 is being addressed through a longitudinal randomized control trial (RCT) conducted in approximately 40 schools in Pennsylvania and California. Goal 2, the focus of this paper, is being addressed through the Scale-up Study, a formative evaluation of the scale-up process. This five year study spans four states: Utah, Michigan, Indiana, and Pennsylvania (schools other than those participating in the RCT). During the grant period, four consecutive cohorts of teachers and schools were invited to participate in the RAISE initiative. This report focuses on the first cohort of RAISE teachers and schools, who were introduced and trained in Reading Apprenticeship during the 2011-12 school year. Using longitudinal teacher survey data (from AY 2011-12 and 2012-13), this report first examines trends over time of key indicators that participants are taking up RAISE activities and indicators of scale-up outcome variables. Then, this report examines if changes in the indicators of implementation and support from the first year (AY 2011-12) to the second year (AY 2012-13) predict changes in scale-up outcomes over the same period of time. [For "Year 1 Interim Report of Reading Apprenticeship/RAISE Scale-Up," see ED583146.]

In October 2010, WestEd's Strategic Literacy Initiative (SLI) won an i3 "Validation" grant to scale up and validate the Reading Apprenticeship (RA) model in three core secondary content area classes: U.S. history, biology, and English language arts. SLI's proposal stated two goals. Goal 1: To transform academic literacy teaching and learning in high school subject areas so that students are able to achieve high standards. Goal 2: To build LEA capacity to disseminate, support, and sustain academic literacy improvement in high school subject areas within and beyond their regions. Goal 1 is being addressed through a longitudinal randomized control trial (RCT) conducted in approximately 40 schools in Pennsylvania and California. Goal 2, the focus of this paper, is being addressed through the Scale-up Study, a formative evaluation of the scale-up process. This five year study spans four states: Utah, Michigan, Indiana, and Pennsylvania (schools other than those participating in the RCT). During the grant period, four consecutive cohorts of teachers and schools have been invited to participate in the RAISE initiative. The purpose of this interim report is to provide formative feedback to the SLI team, site coordinators, and implementing schools, districts, and states regarding the scale-up process in the first year of implementation. Specifically, this report provides a brief review of the scale-up literature/research and the logic model guiding this evaluation; an overview of key RAISE project events in Year 1; RAISE participation at the teacher, school, district, and state levels from Year 1; and a subset of descriptive statistics from the Year 1 teacher and administrator surveys to assess the extent to which the project's goals are being recognized.

In the fall of 2005 the Maui Educational Consortium began a project that was funded through the Math Science Partnership program to implement the "Cognitive Tutor[R]" (CT) program, published by Carnegie Learning, in some of the Algebra 1 classes in the Maui School District and at the Maui Community College. At that time, Empirical Education was brought on to conduct an experimental evaluation of the "CT" algebra program during the 2005-2006 school year and of the "CT" pre-algebra program during 2006-2007. While both studies found mixed results in terms of student achievement in math, they also found that the implementation of the program, in both school years, was not optimal. During both years, teachers reported a lack of access to computers and delayed start of the program. The authors conducted this follow-on study of the successes and challenges of implementing "CT" among pre-algebra and algebra teachers in five schools within the Maui School District and at the Maui Community College. Results showed that: (1) Many teachers (40%) reported that, although computers are centrally located in computer labs rather than in their classroom, their school provides extremely well in terms of installing equipment and networks; (2) Teachers with greater exposure to "CT" all agreed that computer technology can help students learn math, enhance their teaching capabilities, and improve instruction in math; (3) The data reveal that while teachers are not able to divide the instructional time as recommended, teachers using "CT" as their main curriculum were closer to that ideal than the others, with the long term/main use teachers nearly ideal; (4) Teachers generally expressed positive opinions about collaboration; (5) Teachers expressed overall positive attitudes about implementing "CT" and in the level of student engagement, particularly with the software component; and (6) Some teachers are concerned that the reading level of the "CT" textbook is not aligned with their students' reading level and that the content of the program is not aligned with the Hawaii state math content standards pacing guides. (Contains 17 figures, 9 tables, and 2 footnotes.)