Search

This paper examines the "ecological validity" of a randomized controlled trial (RCT) conducted in 42 high schools to measure the impact of a content literacy program. This study of comparable schools implementing the same program in different contexts highlights characteristics that are often not attended to in rigorous effectiveness research but are pertinent to understanding the effectiveness and scalability of the program. In the current study, a direct comparison of school achievement was not possible since the outcome measure used in the RCT was researcher-administered. However, the authors are able to look at teacher characteristics associated with higher student achievement in the RCT and they can look at teacher (and principal) characteristics associated with school-level gains in program participation in Scale-up. The authors found that RCT schools had lower levels of program-related characteristics associated with scale-up. Scale-up schools had similar levels of program-related teacher characteristics associated with greater achievement. Schools that were successful in growing participating internally were not more likely to have the characteristics associated with achievement gains. This suggests that program effectiveness in a controlled study may not be indicative of the program's prospects of wider adoption and, in fact, such studies may not create the best conditions for future program sustainability. Tables and figures are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

We report on the scaling up of a high school content literacy framework, Reading Apprenticeship, over a period of four years as part of the independent evaluation of an Investing in Innovation grant from the U.S. Department of Education to WestEd's Strategic Literacy Institute (SLI). Our goal was to understand the school processes that support successful implementation and promote scaling of an innovation. We focused on teachers, principals and other school-level program leadership who were also the focus for SLI's innovation designed to support scale. Our work was guided by research literature that provided insights into scale-up as increasing local ownership and depth of commitment, as well as growth in numbers. We provided formative feedback to SLI while contributing to the empirical methods and evidence for studying scale-up processes. Over the five years of scaling up, the RAISE project (for Reading Apprenticeship Improving Secondary Education) reached approximately 533,000 students, and 1,719 teachers in 239 schools in Pennsylvania, Michigan, Indiana, and Utah. Surveys of teachers and principals, as well as records of participation in project activities provided a rich longitudinal dataset. In addition to survey analysis, four schools served as case studies where school leaders and program support staff were interviewed over two years to offer a concrete picture of decision processes and resources. We used descriptive statistics to track changes over time in features of program implementation, as well as resulting changes in attitudes and commitment among teachers. In the first year, teachers were successfully integrating Reading Apprenticeship practices and cited collaboration and support from other teachers as most effective for capacity building. We found that uptake of RAISE activities leveled off by the third year. We also found a greater spread in responses across schools by the third year, calling for further investigation into school-level processes and the characteristics of schools that may be associated with the gain or loss of RAISE participants over time. Regression methods were used to identify predictors of increased or decreased participation. We found that teacher participation in team meetings during the first year and school-wide commitment predicted the increase in number of teachers participating in a school. Increases were unrelated to demographics and resources available to the school. We hypothesized that a process, consistent with SLI's innovation for scaling Reading Apprenticeship, which involved a cross-disciplinary teacher team within the school, led to increased and sustained program participation. The scale-up study was conducted in parallel to a [randomized controlled trial] RCT in which 22 treatment and 20 control schools from Pennsylvania and California participated for three years. We used the predictors of increased participation in comparing the scale-up and RCT treatment group schools as contexts for the implementation and impact of Reading Apprenticeship. Scale-up schools had more of the characteristics associated with growth suggesting greater long term sustainability. We raise the issue of whether the implementation under the constraints of the RCT is a good model for measuring impact in the context of scaling up an innovation. The following are appended: (1) Detailed RAISE Scale-up Logic Model; (2) State Maps of RAISE Scale-up, By Year; (3) Results from GL Analysis; and (4) Survey Questions Used in Chapter 8.