False Impression: How a Widely Cited Study Vastly Overstates the Benefits of Charter Schools. Issue Brief

This issue brief focuses on one particular report, released in September 2009, that has been widely cited by charter school advocates because it appears to show remarkable results. In the report, "How New York City's Charter Schools Affect Achievement," Stanford University professor Caroline Hoxby and her colleagues Sonali Murarka and Jenny Kang probe the academic achievement of 30,000 New York City students who had applied to charter schools and had been randomly separated by lotteries into charter schools (the "lotteried-in" students) and traditional schools (the "lotteried-out" students). Hoxby and her colleagues made the headline-grabbing assertion that, on average, for students that attended from kindergarten through grade eight, New York City charter schools could close the "Scarsdale-Harlem gap"--that is, the achievement gap between students in Harlem and students in the much more affluent suburb of Scarsdale--by 66 percent in English and 86 percent in math. This is a shocking finding that, if true, would suggest that charters could be a magic bullet after all. But Hoxby's colleague at Stanford, Sean Reardon, the education researcher and expert in social sciences methodology, scrutinized Hoxby's report and uncovered serious design flaws in the study. Reardon's analysis largely undercuts the claims of dramatic gains that have attracted so much media attention. Reardon found that the design of Hoxby's study destroyed the way in which the lottery randomly separated students into charter schools and traditional schools. A second problem Reardon finds with the design of the Hoxby study for grades four through eight is its implicit assumption that a charter school student will continue to make the same amount of academic gain each year throughout the student's educational career. These two concerns would not be problematic if the study was a truly longitudinal one, following the performance of a single cohort of students from an initial lottery before kindergarten through eighth grade. However, the data for the majority of subjects in the study were for students who had been in charter schools for only three or four years. Reardon's review of Hoxby and colleagues' study highlights further points of concern. These further concerns are also presented in this brief. (Contains 12 endnotes.