The Proficiency Illusion

At the heart of the No Child Left Behind Act (NCLB) is the call for all students to be "proficient" in reading and mathematics by 2014. Yet the law expects each state to define proficiency as it sees fit and design its own tests. This study investigated three research questions related to this policy: (1) How consistent are various states' expectations for proficiency in reading and mathematics? In other words, is it harder to pass some states' tests than others? (2) Is there evidence that states' expectations for proficiency have changed since NCLB's enactment? If so, have they become more or less difficult to meet? In other words, is it getting easier or harder to pass state tests? (3) How closely are proficiency standards calibrated across grades? Are the standards for earlier grades equivalent in difficulty to those for later grades (taking into account obvious grade-linked differences in subject content and children's development)? In other words, is a state's bar for achievement set straight, sloping, or uneven? This study used data from schools whose pupils participated both in state testing and in assessment by the Northwest Evaluation Association (NWEA) to estimate proficiency cut scores (the level students need to reach in order to pass the test for NCLB purposes) for assessments in twenty-six states. Among the findings: (1) State tests vary greatly in their difficulty; (2) Most state tests have not changed in difficulty in recent years; (3) Improvements in passing rates on state tests can largely be explained by declines in the difficulty of those tests; (4) Mathematics tests are consistently more difficult to pass than reading tests; and (5) Eighth-grade tests are consistently and dramatically more difficult to pass than those in earlier grades (even after taking into account obvious differences in subject-matter complexity and children's academic development). As a result, students may be performing worse in reading, and worse in elementary school, than is readily apparent by looking at passing rates on state tests. The report presents both national and state findings. The following are appended: (1) Methodology; (2) Summary of Concurrent Validity Studies; (3) Tables A3.1-mathematics and A3.2-reading summarize key information about each of the state alignment studies, showing the year and school term in which the study was conducted, the grades evaluated, and the average number of students in each grade included; (4) Estimated State-Test Proficiency Cut Scores in Reading using MAP (in Percentile Ranks); (5) Estimated State-Test Proficiency Cut Scores in Mathematics using MAP (in Percentile Ranks); (6) Changes in Proficiency Cut Score Estimates and Reported Proficiency Rates on State Assessments--Reading; (7) Changes in Proficiency Cut Score Estimates and Reported Proficiency Rates on State Assessments--Mathematics; and (8) How Consistent Are the Results from This Study and the NCES Mapping 2005 State Proficiency Standards Study? [A foreword by Chester E. Finn, Jr. and Michael J. Petrilli is also included. This report represents a collaboration of the Thomas B. Fordham Institute and the Northwest Evaluation Association.]