Search

The Arkansas Department of Education partnered with the Regional Educational Laboratory Southwest to study the feasibility and effectiveness of using brief email and text message communications to increase the number of parent and guardian visits to the Reading Initiative for Student Excellence (R.I.S.E.) state literacy website. In November 2021 the department sent test messages to families to determine the percentage of households with children in kindergarten-grade 6 in Arkansas public schools that had a working email address or cell phone number and whether the percentage differed by school locale (rural or nonrural) or demographic composition (percentage of economically disadvantaged students, Black students and Hispanic students, or English learner students). Subsequently, the study team randomly assigned 700 Arkansas public elementary schools to one of eight conditions, which varied the mode of communication (email only or email and text message), the presentation of information (no graphic or with a graphic), and the type of sender (generic sender or known sender). In January 2022 households with children in these schools were sent three rounds of communications with information about literacy and a link to the R.I.S.E. website. The study examined the impact of these communications on whether parents and guardians clicked the link to visit the website (click rate) and conducted an exploratory analysis of differences in how long they spent on the website (time on page). The study found that nearly all households had a working email address or cell phone number. Households with children in rural schools or schools with higher percentages of economically disadvantaged students were less likely than other households to have a working email address. The percentage of households that had a working cell phone number did not vary by school locale or demographic composition. Nearly one in five households opened at least one email, and nearly 80 percent of households were delivered at least one text message. The click rate was highest for households assigned to receive both emails and text messages, with a graphic, and from a known sender. The higher click rate can be attributed to adding text messages and to sending communications from a known sender. The impact on the click rate was similar for schools in different locales and with different demographic compositions. The Arkansas Department of Education can use the findings from this study to inform decisions about how best to communicate with families about literacy and other evidence-based resources. The findings can also be used by other state and local education agencies to inform decisions about using email or text message communications to share resources with families. [For the Study Snapshot, see ED624335. For the appendixes, see ED624336.]

Regional Educational Laboratory Southwest partnered with the Arkansas Department of Education (ADE) to examine Arkansas's middle and high school indicators of postsecondary readiness and success, building on an earlier study of these indicators (Hester et al., 2021). Academic indicators include attaining proficiency on state achievement tests, grade point average, enrollment in advanced courses, and community service learning. Behavioral indicators include attendance, suspension, and expulsion. Using data on statewide grade 6 cohorts from 2008/09 and 2009/10, the study examined the percentages of students who attained the readiness and success indicators and the percentages of students who attained postsecondary readiness and success outcomes by gender, race/ethnicity, eligibility for the National School Lunch Program, English learner student status, disability status, age, and district locale. The study also examined whether the predictive accuracy, specificity, and strength of the indicators varied by these student groups. Three key findings emerged. First, the attainment of indicators of postsecondary readiness and success differed substantially for nearly all student groups, with the number of substantial differences on academic indicators exceeding those on behavioral indicators. The largest number of substantial differences in the attainment of academic indicators were between Black and White students, between students eligible and ineligible for the National School Lunch Program (an indicator of economic disadvantage), and between students who entered grade 6 before and after age 13. Second, attainment of postsecondary readiness and success outcomes varied substantially across student groups, with the largest differences between students with and without a disability. Third, predictive accuracy (the percentage of students with the same predicted and actual outcomes) and strength (the relative importance of a single indicator) were similar across student groups in most cases. Leaders at ADE and in Arkansas districts can use these findings to identify appropriate indicators of postsecondary readiness and success and to target supports toward student groups who most need them. These findings can help leaders identify and address disparities such as inequitable access to resources and supportive learning environments. [For the appendixes, see ED624015.]

The Texas Education Agency launched the Grow Your Own (GYO) grant program in 2018 to encourage districts to develop or expand existing high-quality education and training courses for high school students and to support district-employed paraprofessionals (including instructional aides and long-term substitute teachers) to pursue certifications that would allow them to enter full-time teaching roles. This study aimed to help state education leaders in Texas understand the progress of districts in implementing the GYO program and the early outcomes of participants. This study analyzed data from 2015/16 through 2020/21 for districts that received GYO funding in the first two grant cycles and districts in the same geographic locales within the same regions that did not receive GYO funding (comparison districts). The study found that the majority of districts awarded a GYO grant were in rural areas and small towns. GYO districts were more likely to have a smaller enrollment and had a higher average percentage of Hispanic students than comparison districts. The findings suggest that the program appeared to meet the Texas Education Agency's goal of providing opportunities to students and paraprofessionals in rural and small school settings and students of color to participate in GYO activities. The study also found that the percentage of students completing education and training courses in GYO districts was low (about 10 percent) during the grant years, and the percentage was similar in comparison districts before and after the grant awards. A disproportionate share of students who completed education and training courses in GYO districts were female. Although it is too soon to tell whether the GYO program will, over time, increase the size and diversity of the state's teacher pool, leaders at the Texas Education Agency can use these early findings to both understand the progress of districts in achieving the GYO grant program aims and help identify aspects of the program that might need further investigation. [For the appendices to this report, see ED623385.]

Louisiana's Believe and Prepare pilot program, supported by grants from the Louisiana Department of Education, aimed to prepare teacher candidates or in-service teachers through a residency with a mentor and a competency-based curriculum. To improve teacher preparation and teacher residencies, state and teacher education leaders in Louisiana sought to better understand the early career outcomes for participants in the pilot program. This study analyzed data for the three cohorts that participated in the program between 2014/15 and 2016/17. A majority (76 percent) of pilot participants were enrolled in a university-based teacher preparation program. The study examined certification, employment, and retention outcomes for a subset of pilot participants who were teacher candidates or early career teachers (together referred to as early career Believe and Prepare pilot participants). About 30 percent of early career Believe and Prepare pilot participants who attained a Level 1 professional certificate in 2015/16-2017/18 were certified in a high-need subject, as defined by the Louisiana Department of Education (middle grades math and science, secondary math and science, or special education), and 28 percent of participants who entered teaching in 2015/16-2018/19 taught in a high-need subject in their first year of teaching. Early career pilot program participants who completed a residency in a primary school were more likely than those who completed a residency in a nonprimary school to attain a Level 1 professional certificate. Participants who completed a residency in a charter school were less likely than those who completed a residency in a noncharter school to attain a Level 1 professional certificate. (Louisiana does not require Level 1 certifications for charter schools.) Of early career Believe and Prepare teachers who entered teaching in 2015/16-2017/18, 89 percent were retained in the state for a second year, 76 percent were retained in the same district, and 71 percent were retained in the same school. Among these teachers the within-state retention rate was lowest for teachers in high-need subjects, and the within-school retention rate was lowest for secondary and middle grades math and science teachers. [For the Study Snapshot, see ED612483. For the appendixes, see ED612484.]

These appendices accompany the report "Final Report of the i3 Impact Study of Making Sense of SCIENCE, 2016-17 through 2017-18." Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment. It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd's Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students' science achievement through improving science instruction. Under this grant, WestEd and Empirical Education also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale-up study of Making Sense of SCIENCE. The impact evaluation, which is the focus of this report, was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016-17 and 2017-18 school years. The study randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual ("control") group, which received the professional learning (delayed-treatment) after the study ended. The study found that Making Sense of SCIENCE had a positive impact on teacher content knowledge (effect size = 0.56, p = .006) and a positive impact on a holistic scale of teacher pedagogical content knowledge (effect size = 0.41, p = 0.026). The study also yielded positive and significant impacts on the amount of time teachers spent on science instruction (effect size = 0.40, p = 0.015) and on the emphasis that teachers placed on NGSS-aligned instructional practices, with statistically significant effect sizes ranging from 0.40 to 0.49 standard deviations. This suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice. In regard to student science achievement, the study did not find statistically significant results for the full sample of students (effect size = 0.06, p = 0.494), or for the sample of students in the bottom third of incoming math and English Language Arts achievement, with effect sizes of 0.22 standard deviations (p = 0.099) and 0.073 standard deviations (p = 0.567), respectively. Notably, with the exception of one negative effect size, additional analyses on student achievement using different measures and samples yielded positive, but not statistically significant, effect sizes ranging from 0.02 to 0.12 standard deviations. Evaluators offer three potential contributors to the findings of limited impact on student science achievement. First, given the timing of the study in relation to the release of the NGSS in 2013, finding a suitable NGSS-aligned student science assessment was a challenge. Second, most study schools and districts had not yet adopted NGSS-aligned curricula and did not have access to NGSS-aligned curriculum resources. Third, the sample of teachers was unstable across the two years, with the percentage of teachers leaving the school congruous to the percentage observed at the national level. [For the full report, see ED609253; for the research summary, see ED609256.]

Federal initiatives and other research have led states across the nation to sharpen their focus on teacher evaluation in recent years. In 2009 a seminal report, "The Widget Effect," from The New Teacher Project revealed that in districts using a binary rating system to evaluate teachers, less than 1 percent of teachers received an unsatisfactory rating. The remaining 99 percent were, in effect, like widgets, undifferentiated as individual professionals. Since then, a growing body of research on teacher evaluation systems has indicated that classroom observation ratings often cluster around the middle or high end of evaluation scales. The research has also found that observation ratings are susceptible to several biases, such as incoming student achievement and school, classroom, and rater characteristics. Thus many education decision-makers lack sufficient information to make personnel decisions and to effectively support teacher growth and development. Texas is among the states that have identified teacher evaluation and support as a high priority for education policy. In 2014/15 the Texas Education Agency piloted the Texas Teacher Evaluation and Support System (T-TESS) in 57 school districts. The pilot was followed by a refinement phase in 2015/16 and statewide rollout in 2016/17. During the pilot year teacher overall ratings were based solely on rubric ratings on 16 dimensions across four domains (planning, instruction, learning environment, and professional practices and responsibilities), although T-TESS also includes a student growth measure that was piloted simultaneously. The study examined the statistical properties of the T-TESS rubric from the 2014/15 pilot year to explore the extent to which it differentiates teachers on teaching quality and to investigate its internal consistency and efficiency. The study also explored the relationships between rubric ratings and school characteristics to investigate whether certain types of schools have teachers with higher or lower ratings. The study's findings have several implications for practice and research. The findings suggest that the T-TESS rubric demonstrates potential to be an effective, consistent, and efficient evaluation rubric. Thus, the Texas Education Association and local education agencies have a promising tool for providing evidence-based feedback and targeted professional development. Future research could try to validate ratings based on the T-TESS rubric against other measures of teacher effectiveness (for example, student growth). Such validation could shed light on whether a relationship exists between rubric ratings and a teacher's contribution to student achievement. Moreover, future studies could explore whether relationships exist between the T-TESS rubric and classroom and district characteristics. Such analysis may unearth the extent to which effective teachers are equally distributed within schools and within and across districts. Finally, further research could explore whether implementing teacher evaluation systems translates into improvements in teacher effectiveness or in long-term outcomes, such as teacher retention and student achievement.

Recruiting and retaining effective teachers are serious concerns throughout Oklahoma. The Oklahoma State School Boards Association (2016) reported 500 teacher vacancies at the beginning of the 2015/16 school year, according to a survey of school districts, and 53 percent of respondents said the teacher shortage was worse than in the previous year. For years, Oklahoma rural school district administrators have reported difficulty retaining teachers who could cross state lines for higher pay and lower class sizes or seek employment in other industries. In 2013 the Oklahoma State Superintendent of Public Instruction established the Oklahoma Educator Workforce Shortage Task Force to recommend measures to alleviate the "significant and widespread shortages" of classroom teachers. The task force was succeeded in September 2015 by the Teacher Shortage Task Force, which was established to identify and recommend successful strategies for curbing the statewide teacher shortage crisis and which recommended several strategies for placing highly qualified teachers in all Oklahoma classrooms. The state's teacher shortage, as well as the unique context of rural schools in Oklahoma, led members of the Regional Educational Laboratory Southwest Oklahoma Rural Schools Research Alliance to seek information about factors associated with successful teacher recruitment and retention in Oklahoma. The goal was to develop effective strategies for recruiting and retaining teachers in rural schools. In response, this study identified factors that can support teacher recruitment and retention, particularly malleable factors that can be controlled through policies and interventions. This report refers to these factors as indicators of the characteristics of teachers or districts that predict successful teacher recruitment and retention. While associations between indicators and outcomes cannot be interpreted as causal--a specific indicator is not necessarily the cause of a related outcome--the results from this study can be used to pinpoint potential problems and inform future policies. The results can also provide a rationale for experimental evaluations of programs aiming to improve teacher recruitment and retention. The study first explores patterns of teacher job mobility in Oklahoma, including teachers' probability of remaining employed in the same district for a given number of years, the proportion of teachers who leave rural school districts and move to another rural school district, the proportion of teachers who receive tenure, and the one year retention probability for each successive year of employment. Patterns of teacher job mobility are examined for any differences between rural and nonrural school districts. The study was designed to identify teacher, district, and community characteristics in rural Oklahoma that predict which teachers are most likely to be successfully recruited (defined as having completed a probationary period of three years and obtained tenure in their fourth year of teaching) and retained longer term (defined as the duration of employment of tenured teachers in a given school district). This study covers the 10 school years between 2005/06 and 2014/15 and uses teacher and district data from the Oklahoma State Department of Education, Oklahoma Office of Educational Quality and Accountability, and community characteristics from data in federal noneducation sources and publicly available geographic information systems from Google Maps.

Regional Educational Laboratory Southwest conducted this literature review to identify malleable factors that can be measured in K-12 settings and that predict students' postsecondary science, technology, engineering, and math (STEM) success (defined as enrolling in, persisting in, and completing a postsecondary STEM major or degree), particularly for Hispanic students. The review found that courses taken in high school and interest or confidence in STEM were strong predictors of postsecondary STEM success for students of all racial/ethnic subgroups. Yet racial/ethnic minority students were less likely than White students to take the highest level math and science courses, and despite similar levels of STEM interest, racial/ethnic minority students were less likely to achieve postsecondary STEM success. Other indicators of postsecondary STEM success included high school grade point average, class rank, math and science achievement, and SAT or ACT scores. Grades in math and science courses were found to be less predictive of postsecondary STEM success for racial/ethnic minority students than for White students. [For the full report, see ED565641.]

The number of jobs in science, technology, engineering, and math (STEM) is growing rapidly and is expected to increase by approximately 1 million in the United States between 2012 and 2022 (Vilorio, 2014). People of many racial/ethnic minorities, however, including Hispanic people, are underrepresented among recipients of STEM degrees and among employees in STEM fields (Beede et al., 2011). Regional Educational Laboratory Southwest conducted this review of the research literature to identify malleable factors that can be measured in K-12 settings and that predict students' postsecondary STEM success (defined as enrolling in, persisting in, and completing a postsecondary STEM major or degree), particularly for Hispanic students. Identifying these predictive malleable factors can help policymakers and district and school administrators develop and implement interventions that increase the percentage of Hispanic students succeeding in postsecondary STEM majors and pursuing STEM careers. This report addressed two research questions: (1) What K-12 indicators predict postsecondary STEM success?; and (2) To what extent do K-12 indicators of postsecondary STEM success differ for Hispanic and non-Hispanic students? Key findings from this literature review include the following: (1) The number of high school math and science courses taken and the level of those courses predict postsecondary STEM success for all student subgroups; (2) Interest or confidence in STEM showed statistically significant predictive relationships with students' postsecondary STEM success; (3) Indicators related to achievement in middle school and high school predicted students' postsecondary STEM success; (4) Statistically significant high school predictors of postsecondary STEM success included schools' academic rigor, percentage of students enrolled in college preparatory programs, students' satisfaction with their teachers, and levels of parent participation; (5) Few studies examined K-12 predictors of postsecondary STEM success specifically for Hispanic students. Appended are: (1) About the literature review; (2) Abstracts of reviewed studies; and (3) Detailed findings by study.

This paper examines the "ecological validity" of a randomized controlled trial (RCT) conducted in 42 high schools to measure the impact of a content literacy program. This study of comparable schools implementing the same program in different contexts highlights characteristics that are often not attended to in rigorous effectiveness research but are pertinent to understanding the effectiveness and scalability of the program. In the current study, a direct comparison of school achievement was not possible since the outcome measure used in the RCT was researcher-administered. However, the authors are able to look at teacher characteristics associated with higher student achievement in the RCT and they can look at teacher (and principal) characteristics associated with school-level gains in program participation in Scale-up. The authors found that RCT schools had lower levels of program-related characteristics associated with scale-up. Scale-up schools had similar levels of program-related teacher characteristics associated with greater achievement. Schools that were successful in growing participating internally were not more likely to have the characteristics associated with achievement gains. This suggests that program effectiveness in a controlled study may not be indicative of the program's prospects of wider adoption and, in fact, such studies may not create the best conditions for future program sustainability. Tables and figures are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]