Search

This report presents estimates from a large-scale, multi-district RCT (randomized controlled trial) on the effectiveness of a summer reading program on improving student reading comprehension for economically disadvantaged grade 3 students reading below the 50th percentile nationally. This study focused on the summer between grades 3 and 4 for three reasons: (1) independent reading demands increase dramatically in grades 3 and 4 (Chall 1983; National Research Council 1998); (2) the grade 3 to grade 4 transition was not a focus of previous studies; and (3) Texas state assessment data are available for the first time for students beginning in grade 3, and those data were used to control for baseline differences in this study. Each student in the treatment group was sent a single shipment of eight books matched to his or her reading level and interest area during the first part of the summer (June/July 2009), followed by a reminder postcard each week for six weeks. Eight books were chosen because this was the number used in two of the summer reading programs shown to have statistically significant positive effects on reading comprehension for specific subgroups (Kim 2006) or for the entire sample (Kim and White 2008). Seven previous studies examined summer reading programs, and five found a statistically significant improvement in reading achievement following implementation of a reading program (Allington et al. 2010; Butler 2010; Crowell and Klein 1981; Kim 2006; Kim and White 2008). Of the five studies that used an RCT design, three found a statistically significant effect on reading achievement (Allington et al. 2010; Kim 2006; Kim and White 2008). The current study's confirmatory finding did not replicate the findings from these studies. Two of the five RCT studies found that students sent books over the summer reported reading more books than did students who were not sent books (Kim 2007; Kim and Guryan 2010); an exploratory analysis in the current study found similar results. The summer reading program examined in this study did not include teacher support, instructional components, or parent involvement, which several previous studies had included to varying degrees--four RCTs (Kim 2006, 2007; Kim and Guryan 2010; Kim and White 2008) and one quasi-experiment (Butler 2010). These other components could potentially account for differences in observed effects across studies. Also, the program examined in the current study spanned a single summer, whereas the program examined in Allington et al. (2010) spanned three summers. Further, the current study sample consisted of economically disadvantaged students reading below the 50th percentile nationally, while the samples in the studies with statistically significant results consisted of students with economically diverse backgrounds (Kim 2006, 2007; Kim and Guryan 2010; Kim and White 2008) and were not composed exclusively of students reading below the 50th percentile nationally (Allington et al. 2010; Butler 2010; Crowell and Klein 1981; Kim 2006, 2007; Kim and Guryan 2010; Kim and White 2008). One possible inference to draw from this study, and the more recent work of Kim and colleagues (Kim and Guryan 2010; Kim and White 2008), is that some of the components that Kim and his colleagues added--in particular, personalized teacher encouragement of each student to read the books during the summer and brief, small group lessons on strategies for reading--may be essential components to success. Although such additions may be costly and time intensive for the teaching staff, many teachers find this type of activity a rewarding part of their jobs. Future scale-up research could continue to examine the issue of varied types of teacher and parent support components that Kim included (Kim 2006, 2007; Kim and Guryan 2010; Kim and White 2008). Allington (2010) found that when students were provided books over a period of three summers, even without any additional support components, student reading significantly improved. Therefore, it may be that teacher and parent support components are necessary for a summer reading program to be effective during a single summer, but may be less important if students participate in summer reading programs over a longer time period. Appended are: (1) Description of the Lexile Framework[R] for Reading; (2) Findings from previous studies of summer reading programs; (3) Student interest survey, explanatory letter, postcard, and summer reading survey; (4) Power analysis; (5) Texas Assessment of Knowledge and Skills-Lexile linking study; (6) Recruitment and study sample details; (7) Participating district profiles; (8) Description of the grade 3 Texas Assessment of Knowledge and Skills-Reading; (9) Random assignment; (10) Missing Data; (11) Summer reading survey results; (12) Models used for primary, sensitivity, and exploratory analyses; and (13) Tables of analytic output. (Contains 44 tables, 3 figures and 48 footnotes.)

The 2004 reauthorization of the Individuals with Disabilities Education Act (IDEA) approved schools' use of alternative methods for determining student eligibility for special education services. IDEA encourages schools to intervene as soon as there is a valid indication that a student might experience academic difficulties, rather than after performance falls well below grade-level. The Response to Intervention (RtI) framework is an approach for providing instructional support to students at risk for these difficulties. RtI models typically have three tiers of increasing intensity of instruction (Gersten et al. 2008; Gersten et al. 2009). Tier 1 involves research-based core instruction delivered with high fidelity in the classroom by the classroom teacher and universal screening of all students to determine who should receive additional instructional support. Tier 2 involves focused/intensive instruction, often in small groups, for children at risk for failing in the Tier 1 setting. Tier 3 involves even more intensive instruction for students not responding to the Tier 2 interventions and often comprises individual tutoring, referral to a school psychologist, or both. Despite increasing interest, there is little research on the effectiveness of recommended best practices in RtI (Gersten et al. 2008; Gersten et al. 2009). A recent literature review of grades K-3 mathematics interventions suitable for use in Tier 2 revealed just nine relevant studies (Newman-Gonchar, Clarke, and Gersten 2009), with just one that was a rigorous evaluation of an intervention, and that used a randomized controlled trial (RCT) design (Fuchs et al. 2005). The Fuchs et al. (2005) study examined the impact of "Number Rockets", a small-group tutoring intervention for grade 1 students at risk for mathematics difficulties, and found statistically significant positive effects on several measures of mathematics proficiency. But that study was an efficacy trial (one implemented under ideal conditions), involved considerable monitoring and support for experienced tutors, and was conducted in a single district. This study builds on the Fuchs et al. (2005) study and is the first large-scale effectiveness trial (one intended to approximate real-world implementation) of "Number Rockets". The current study addresses the following confirmatory research question: Do grade 1 students at risk in mathematics who participate in "Number Rockets" perform better than at-risk control students on the Test of Early Mathematics Ability-Third Edition (TEMA-3; Ginsburg and Baroody 2003)? The study also investigated three exploratory research questions: (1) Does "Number Rockets" have a differential impact on grade 1 students at risk in mathematics, based on baseline mathematics proficiency?; (2) Do grade 1 students who participate in "Number Rockets" score differently than control students on the Woodcock-Johnson-Third Edition Letter/Word (WJ-III Letter/Word; Woodcock, McGrew, and Mather 2001) subtest?; and (3) Do the impacts of "Number Rockets" vary significantly depending on the average number of lessons delivered within a school? This RCT was implemented in 76 schools in four urban districts across four of the five Regional Educational Laboratory Southwest states. The target student population was grade 1 students at risk for mathematics difficulties who received mathematics instruction in English in a regular education classroom. The main finding of this effectiveness study is that grade 1 students at-risk for difficulties in grade 1 mathematics benefited from participation in the "Number Rockets" intervention. At-risk students in the intervention group showed statistically significant higher performance on the TEMA-3, a broad measure of student proficiency in mathematics, than at-risk students in the control group. This finding was observed in a sample of 994 students from 76 schools in four urban districts across four states. The results of all three exploratory analyses (related to differences in baseline mathematics proficiency, performance on a reading test, and number of tutoring sessions) were not statistically significant. Appended are: (1) Study timeline; (2) Power analysis assumptions; (3) Parent consent form; (4) Screener subtest details and descriptive statistics; (5) Student mobility; (6) Fidelity measures; (7) Models used for confirmatory, exploratory, and sensitivity analyses; (8) Lessons; (9) Complete sample lesson Topic 6, Day 1; (10) Details of tutor training; (11) Tutor background survey; (12) Details of fidelity coder training; and (13) Complete multilevel model results for chapter 4 (confirmatory and sensitivity) and chapter 5 (exploratory and sensitivity) analyses. (Contains 39 tables, 13 figures and 83 footnotes.) [This report was prepared for the National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences by Regional Educational Laboratory Southwest administered by the Edvance Research.]

Collaborative Strategic Reading (CSR) is a set of instructional strategies designed to improve the reading comprehension of students with diverse abilities (Klingner and Vaughn 1996). Teachers implement CSR at the classroom level using scaffolded instruction to guide students in the independent use of four comprehension strategies; students apply the strategies to informational text while working in small cooperative learning groups. The current study is a randomized controlled trial (RCT) examining the effect of CSR on student reading comprehension. Within each participating linguistically diverse school, grade 5 social studies classrooms were randomly assigned to either the CSR condition (using CSR when delivering social studies curricula) or to the control condition (a business-as-usual condition). The implementation period was one school year. This study focused on the following confirmatory research question: In linguistically diverse schools, do grade 5 students in CSR classrooms have higher average reading comprehension posttest scores on the Group Reading Assessment and Diagnostic Evaluation (GRADE) than students in control classrooms? In addition, the study examined three exploratory research questions about CSR's effect on two subgroups of students: (1) Do grade 5 former and current English language learner (FC-ELL) students in CSR classrooms have higher average reading comprehension posttest scores on the GRADE than FC-ELL students in control classrooms?; (2) Do grade 5 non-ELL students in CSR classrooms have higher average reading comprehension posttest scores on the GRADE than non-ELL students in control classrooms?; and (3) Does CSR have a differential impact on GRADE reading comprehension posttest scores for grade 5 FC-ELL and non-ELL students? The intent of these exploratory analyses was to examine whether there is an effect for each subgroup separately, as well as whether there is a differential effect between the subgroups. The primary finding of this study is that CSR did not have a statistically significant impact on student reading comprehension. Nine sensitivity analyses--including alternative statistical approaches, an alternative approach for handling missing data, and different sample specifications--showed that the findings were robust to different analytic approaches. Three exploratory analyses were also conducted to examine the effects of CSR on FC-ELL and non-ELL students. Statistically significant effects on student reading comprehension were not identified for either subgroup, and no statistically significant differential impacts were identified. It is often the case that RCTs, because of their greater rigor, do not support the findings of prior quasi-experiments (Glazerman, Levy, and Myers 2002, 2003). With all other design features held constant, randomization yields stronger evidence about program impacts than do quasi-experiments (Boruch 1997; Shadish, Cook, and Campbell 2002). The current investigation evaluated the impact of CSR in an effectiveness trial designed to approximate a district's implementation of CSR. Data on the fidelity of implementation suggest that professional development was generally delivered according to plan. Data on teacher fidelity of CSR implementation showed that 78.8 percent of teachers reported using CSR two or more times a week, as instructed. However, the single observation conducted for each classroom found that 21.6 percent of CSR teachers were using all five core teacher strategies, which the study defined as full procedural fidelity; 56.8 percent of teachers were observed using three or fewer strategies. Appendices include: (1) Identification and exit criteria for English language learner students in Oklahoma and Texas; (2) Assumptions used to determine statistical power and observed power; (3) Random assignment; (4) Analysis of consent rate at baseline; (5) Estimation methods; (6) Frequently asked questions about contamination; (7) Attrition analyses; (8) Response rates for demographic data; (9) Fall and spring teacher surveys; (10) Fall coaching observation form; (11) Multiple imputation; (12) Assigning students to cooperative learning groups; (13) Critical procedural behaviors for Collaborative Strategic Reading strategies; (14) Observer training for the subscale Expository Reading Comprehension observation instrument and interrater reliability; (15) Descriptive statistics on Group Reading Assessment and Diagnostic Evaluation scores; (16) Baseline equivalence results for multiply imputed analytic dataset; and (17) Full analytic output tables. A glossary is included. (Contains 52 tables, 3 figures and 46 footnotes.