Search

Content standards, not performance standards, have been almost the sole focus of state policies and recent conversations about academic standards. Without rigorous content and performance standards, we cannot adequately prepare students for the global marketplace. A recent AIR study shows that state performance standards are consistently low and extremely variable across states. Low state standards allow many students to be described as "proficient," when they are not adequately prepared for success beyond high school. To address this issue, this report recommends that states should adopt evidence-based methods for setting standards, such as the benchmark method, to ensure students are prepared to compete in the global marketplace.

This paper describes a statistical linking between the 2011 National Assessment of Educational Progress (NAEP) in Grade 4 reading and the 2011 Progress in International Reading Literacy Study (PIRLS) in Grade 4 reading. The primary purpose of the linking study is to obtain a statistical comparison between NAEP (a national assessment) and PIRLS (an international assessment). By expressing both assessments in the same metric, the linking study can provide international benchmarks for the NAEP Grade 4 reading achievement levels. At each level, the linking shows that the NAEP Grade 4 reading achievement levels are higher than the PIRLS international benchmarks. This finding provides one piece of validity evidence that NAEP results are internationally competitive. Two appendices present: (A) Linking 2011 PIRLS to 2011 NAEP Grade 4 Reading; and (B) Jackknife Standard Errors of Linking Parameters.

Since its first assessment in 1969, the National Assessment of Educational Progress (NAEP) has made a unique contribution to our understanding of American education. It is the only national source of information on the educational achievement of U.S. students, and it is the only vehicle by which states can compare the progress of their students against a common standard. Assessment results reported by NAEP complement the states' own reports of progress under No Child Left Behind (NCLB) and track the status of achievement gaps for traditionally disadvantaged student groups. NAEP is carried out under the guidance of the National Assessment Governing Board (Governing Board) and the National Center for Education Statistics (NCES). Throughout the course of its history, NAEP has frequently sought to improve by studying its own processes, instruments, and procedures. In keeping with this tradition, in fall 2011, NCES asked the NAEP Validity Studies (NVS) Panel, which operates under contract to NCES, to undertake two inter-related studies, one in reading/writing and one in mathematics, to examine the content of the current NAEP frameworks and item pools at Grades 4, 8, and 12 in relation to the Common Core State Standards (CCSS). The primary question under investigation is whether NAEP can continue to serve as an independent monitor of student achievement and state assessments following the implementation of the CCSS. This report addresses the relations between the NAEP reading and writing frameworks and the CCSS in English language arts (CCSS-ELA), and the relations between the NAEP reading and writing items and the CCSS-ELA. It does not address the relations between NAEP reading and writing items and items developed by the Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (Smarter Balanced) to assess the CCSS-ELA because those items were not available at the time of this study. The report concludes with recommendations to NCES regarding broader issues on the alignment between NAEP reading and writing and CCSS-ELA, including the extent of alignment that is appropriate to support NAEP's role as an independent monitor of student achievement. An appendix presents a list of the Reading and Writing Panelists. [For the main report, "Examining the Content and Context of the Common Core State Standards: A First Look at Implications for the National Assessment of Educational Progress," see ED545237.]

School district accountability efforts have been challenged to develop procedures that determine the effects of their programs on student achievement within the typical implementation setting (i.e., nonrandom student assignment to nested school within program placements). Disentangling program effects from contextual and participant factors involves strategic considerations for data, design, and analysis. This paper presents an application of a two-level student-program hierarchical linear modeling (HLM) procedure to examine value-added effects of the Prince George's County, Maryland, Magnet School Programs at improving student academic performance. Examples of data handling and preparation for multilevel analysis are provided. Results from this study were primary considerations in the district superintendent's decisions regarding the expansion, continuation, and elimination of particular magnet school programs. The main findings in the study were: (1) overall, elementary students in magnet programs perform better than nonmagnet students; (2) this outcome is largely due to the fact that more able students are self-selected for the magnet program; and (3) when student ability is accounted for in the evaluation design, students in magnet school programs do not perform as well as students in nonmagnet programs. This last finding was further confirmed using unadjusted data in which talented and gifted (TAG) students in magnet programs did not perform as well as TAG students in nonmagnet schools. Appendixes contain tables of target data and program variables and a discussion of the statistical underpinnings of the HLM models used. (Contains 4 figures and 10 references.) (Author/SLD)

Recently, there has been a significant expansion in the use of performance assessment in large scale testing programs. Although there has been significant support from curriculum and policy stakeholders, the technical feasibility of large scale performance assessments has remained a question. This report is intended to contribute to the debate by reviewing some of the technical issues that must be addressed by any developer of large-scale performance assessments. The report is also intended to surface issues, articulate problems, and where possible, give advice on how to proceed. The report is divided into five chapters, each focusing on a major technical topic. "Validity of Performance Assessments" (Samuel Messick) defines validity as a property of inferences and interpretations made from test scores. In performance assessment, the primary adverse consequence that must be investigated is the potential negative impact on individuals or groups based on sources of invalidity. "Generalizability of Performance Assessments" (Robert L. Brennan) provides an overview of generalizability theory and integrates literature on the reliability of performance assessment with the conceptual framework of generalizability theory. "Comparability" (Edward H. Haertel and Robert L. Linn) stresses that in order to provide indicators of trends in academic achievement, large scale performance assessments must be comparable across administrations. "Setting Performance Standards for Performance Assessments: Some Fundamental Issues, Current Practice, and Technical Dilemmas" (Richard M. Jaeger, Ina V. S. Mullis, Mary Lyn Bourque, and Sharif Shakrani) describes the myriad ways performance standards are used and addresses the need for new methods of establishing such standards for performance assessments of students and teachers. Two new approaches to setting performance standards, iterative judgmental policy capturing and a multistage dominant profile procedure are outlined. References follow each of the chapters. (Contains five tables and eight figures.) (SLD)

This study investigates a linking of the 1991 International Assessment of Educational Progress (IAEP) and the 1992 National Assessment of Educational Progress (NAEP) mathematics assessments. Data to allow such a linking were collected in 1992 from students in the United States who were administered both instruments. Modeling was done using a regression analysis and that model was used as the basis for projecting IAEP scores from non-United States countries onto the NAEP scale. This study focuses on the percentage of students from the IAEP countries who were predicted to fall above the three National Assessment Governing Board (NAGB) achievement levels and investigates four sources of error. These sources of error pertain to not having or not knowing: (1) the true relationship between the IAEP and NAEP assessments; (2) results for the entire IAEP population; (3) simple random samples of students; and (4) the true proficiency level of every student. Results of the study were very encouraging. The relationship between the IAEP and NAEP assessments was quite strong and could be modeled well. (Contains 16 references.) (Author/DDR)

This report deals with a variety of ways that have been, or could be, used to interpret the scales used in the National Assessment of Educational Progress (NAEP). Policymakers, researchers, and other users of assessment results need to understand the methods used for reporting NAEP. Having a reference is particularly important as the methods of reporting are changing. Chapter 1 covers the following methods that have been used, or could be used, to interpret scales: (1) percentage correct for each item; (2) average percentage correct; (3) item mapping; (4) scale anchoring; (5) achievement levels; (6) using scoring rubrics; and (7) benchmarking. The contrast between anchor levels and achievement levels is discussed. Chapter 2 discusses the distinction between norm-referenced and criterion-referenced interpretations, and the validity of the inferences drawn from NAEP interpretations. Issues of validity are especially important with regard to achievement levels, because they represent an effort to go beyond describing, to prescribing recommended levels of achievement for the nation. Eleven figures and four tables present analysis data. An appendix provides exemplar exercises for scale anchoring and for achievement levels. (SLD)