Search

These appendices accompany the report "Final Report of the i3 Impact Study of Making Sense of SCIENCE, 2016-17 through 2017-18." Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment. It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd's Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students' science achievement through improving science instruction. Under this grant, WestEd and Empirical Education also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale-up study of Making Sense of SCIENCE. The impact evaluation, which is the focus of this report, was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016-17 and 2017-18 school years. The study randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual ("control") group, which received the professional learning (delayed-treatment) after the study ended. The study found that Making Sense of SCIENCE had a positive impact on teacher content knowledge (effect size = 0.56, p = .006) and a positive impact on a holistic scale of teacher pedagogical content knowledge (effect size = 0.41, p = 0.026). The study also yielded positive and significant impacts on the amount of time teachers spent on science instruction (effect size = 0.40, p = 0.015) and on the emphasis that teachers placed on NGSS-aligned instructional practices, with statistically significant effect sizes ranging from 0.40 to 0.49 standard deviations. This suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice. In regard to student science achievement, the study did not find statistically significant results for the full sample of students (effect size = 0.06, p = 0.494), or for the sample of students in the bottom third of incoming math and English Language Arts achievement, with effect sizes of 0.22 standard deviations (p = 0.099) and 0.073 standard deviations (p = 0.567), respectively. Notably, with the exception of one negative effect size, additional analyses on student achievement using different measures and samples yielded positive, but not statistically significant, effect sizes ranging from 0.02 to 0.12 standard deviations. Evaluators offer three potential contributors to the findings of limited impact on student science achievement. First, given the timing of the study in relation to the release of the NGSS in 2013, finding a suitable NGSS-aligned student science assessment was a challenge. Second, most study schools and districts had not yet adopted NGSS-aligned curricula and did not have access to NGSS-aligned curriculum resources. Third, the sample of teachers was unstable across the two years, with the percentage of teachers leaving the school congruous to the percentage observed at the national level. [For the full report, see ED609253; for the research summary, see ED609256.]

Empirical Education Inc. is the independent evaluator of SRI International's 2014 Investing in Innovation (i3) Development grant called Redesigning Secondary Courses to Improve Academic Outcomes for Adolescents with Disabilities and Other Underperforming Adolescents. The goal of the grant is to develop "Enhanced Units" that combine research-based content enhancement routines, collaboration strategy, and technology components for secondary U.S. History and biology classes. This report presents findings of a randomized control trial (RCT) during the 2017-18 school year. The RCT measured the impact of "Enhanced Units" on higher order content skills (as measured through unit tests) in high school biology and U.S. History classes in three districts in Virginia and California. SRI, the Center for Applied Special Education Technology (CAST), and their research and practitioner partners developed "Enhanced Units" (EU) with the goal of integrating research-based content enhancement routines with technological enhancements to improve student content learning and higher order reasoning, especially for students with disabilities or other learning challenges. This study also documents the extent to which the core components of EU were implemented with fidelity. The authors provide descriptive results on classroom practices (as measured by teacher surveys) and contextual factors that support or hinder implementation (as described during teacher interviews). Future improvements to EU should focus on answering the question: "What is/are the best way(s) for teachers to present SIM routines to their students, particularly for students with learning challenges through SIM intervention?"

Professional graduate program curricula are consistently evaluated for their efficacy as they relate students' needs and achievement. Particularly, teachers in the TESOL field are challenged to become effective in their profession by adapting and developing learning materials and designing successful curriculum that imparts all relevant concepts and knowledge that contributes to this field. Based on the aforementioned reasons, it is imperative to continually evaluate and reexamine the effectiveness of TESOL master's programs, as well as to investigate what information and skills this program promotes to prepare graduate students for their current and future careers. Two appendixes are included: (1) Demographics; and (2) Interview Questions.

We report on the scaling up of a high school content literacy framework, Reading Apprenticeship, over a period of four years as part of the independent evaluation of an Investing in Innovation grant from the U.S. Department of Education to WestEd's Strategic Literacy Institute (SLI). Our goal was to understand the school processes that support successful implementation and promote scaling of an innovation. We focused on teachers, principals and other school-level program leadership who were also the focus for SLI's innovation designed to support scale. Our work was guided by research literature that provided insights into scale-up as increasing local ownership and depth of commitment, as well as growth in numbers. We provided formative feedback to SLI while contributing to the empirical methods and evidence for studying scale-up processes. Over the five years of scaling up, the RAISE project (for Reading Apprenticeship Improving Secondary Education) reached approximately 533,000 students, and 1,719 teachers in 239 schools in Pennsylvania, Michigan, Indiana, and Utah. Surveys of teachers and principals, as well as records of participation in project activities provided a rich longitudinal dataset. In addition to survey analysis, four schools served as case studies where school leaders and program support staff were interviewed over two years to offer a concrete picture of decision processes and resources. We used descriptive statistics to track changes over time in features of program implementation, as well as resulting changes in attitudes and commitment among teachers. In the first year, teachers were successfully integrating Reading Apprenticeship practices and cited collaboration and support from other teachers as most effective for capacity building. We found that uptake of RAISE activities leveled off by the third year. We also found a greater spread in responses across schools by the third year, calling for further investigation into school-level processes and the characteristics of schools that may be associated with the gain or loss of RAISE participants over time. Regression methods were used to identify predictors of increased or decreased participation. We found that teacher participation in team meetings during the first year and school-wide commitment predicted the increase in number of teachers participating in a school. Increases were unrelated to demographics and resources available to the school. We hypothesized that a process, consistent with SLI's innovation for scaling Reading Apprenticeship, which involved a cross-disciplinary teacher team within the school, led to increased and sustained program participation. The scale-up study was conducted in parallel to a [randomized controlled trial] RCT in which 22 treatment and 20 control schools from Pennsylvania and California participated for three years. We used the predictors of increased participation in comparing the scale-up and RCT treatment group schools as contexts for the implementation and impact of Reading Apprenticeship. Scale-up schools had more of the characteristics associated with growth suggesting greater long term sustainability. We raise the issue of whether the implementation under the constraints of the RCT is a good model for measuring impact in the context of scaling up an innovation. The following are appended: (1) Detailed RAISE Scale-up Logic Model; (2) State Maps of RAISE Scale-up, By Year; (3) Results from GL Analysis; and (4) Survey Questions Used in Chapter 8.