Search

This brief summarizes "Comparability in Balanced Assessment Systems for State Accountability," published in "Educational Measurement: Issues and Practice" (Evans & Lyons 2017). The study evaluated comparability claims in local scoring of performance assessments across districts participating in New Hampshire's Performance Assessment of Competency Education (PACE) pilot project. The study applied two judgmental methods for estimating comparability that are used in international contexts: consensus scoring and rank ordering. Across all districts, the consensus scoring yielded scores that were positive, meaning they were a bit lower than the teacher-given scores. Analysis revealed uneven scoring across districts, suggesting that there remains a need for additional training on scoring and within-district calibration, as well as for increased cross-district calibration. The rank-order method results revealed scoring differences across districts, most notably in one district in which teachers were scoring their student work a full standard deviation below (more rigorous) where the judges placed the same student work within the sample. Researchers found that applying the two methods in the context of the PACE system highlighted some strengths and limitations of each method. [For "Comparability in Balanced Assessment Systems for State Accountability," see EJ1154674.]

This field report from 2017's National Conference on Student Assessment shares possibilities for flexibility and innovation in assessment and accountability made possible by the Every Student Succeeds Act.

Evidence of comparability is generally needed whenever there are variations in the conditions of an assessment administration, including variations introduced by the administration of an assessment on multiple digital devices (e.g., tablet, laptop, desktop). This article is meant to provide a comprehensive examination of issues relevant to the comparability of scores across devices, and as such provide a starting point in designing and implementing a research agenda to support the comparability of any assessment program. This work starts with a conceptual framework rooted in the idea of a comparability claim--a conceptual statement about how each student is expected to perform on each of the devices in question. Then a review of the available literature is provided, focusing on how aspects of the devices (touch screens, keyboards, screen size, and displayed content) and aspects of the assessments (content area and item type) relate to student performance and preference. Building on this literature, recommendations to minimize threats to comparability are then provided. The article concludes with ways to gather evidence to support claims of comparability.

The purpose of this study was to test methods that strengthen the comparability claims about annual determinations of student proficiency in English language arts, math, and science (Grades 3-12) in the New Hampshire Performance Assessment of Competency Education (NH PACE) pilot project. First, we examined the literature in order to define comparability outside the bounds of strict score interchangeability and explored methods for estimating comparability that support a balanced assessment system for state accountability such as the NH PACE pilot. Second, we applied two strategies--consensus scoring and a rank-ordering method--to estimate comparability in Year 1 of the NH PACE pilot based upon the expert judgment of 85 teachers using 396 student work samples. We found the methods were effective for providing evidence of comparability and also detecting when threats to comparability were present. The evidence did not indicate meaningful differences in district average scoring and therefore did not support adjustments to district-level cut scores used to create annual determinations. The article concludes with a discussion of the technical challenges and opportunities associated with innovative, balanced assessment systems in an accountability context.