Search

Large-scale performance assessment programs are a longstanding reform tool. However, standard setting can be a challenge for assessment programs that use primarily non-standardized assessments. The purpose of this paper is to extend this field of research by explaining the standard setting methodology applied to one more recent instantiation of a state performance assessment program. The second purpose of this paper is to discuss the data quality control and quality assurance challenges experienced after five years of applying the standard setting method. Recognizing the burgeoning interest again in large-scale performance assessment programs, the goal and intended contribution of this paper is to inform future decisions about selecting appropriate standard setting methods and dealing with unanticipated challenges that may arise during implementation based upon the lessons learned from one program. It is likely that other large-scale performance assessment programs may face similar operational challenges, especially those that do not rely on standardized tests or standardized administration procedures to produce annual determinations of student proficiency or other scores used for accountability purposes. Assessment system designers can use the insights in this paper to consider standard setting methods and how those methods may need to be adapted to promote technical quality.

There is renewed interest around including performance assessments in state and local assessment systems to spur positive changes in classroom instruction and student learning. Previous research has identified the external conditions that mediate the role of assessment in changing instructional practices. We extend that work by focusing on the internal classroom conditions that support improvements in student learning. We identified six key instructional practices from three teacher quality frameworks that may result from policy changes that include complex, performance-based assessments. For each practice, we explored the bidirectional relationships among the instructional core of students, teachers, and content. We argue that altering these relationships requires teachers and students to have both the disposition and the capacity to change, and we identify the assumptions that need to hold in order for those changes to occur in response to the inclusion of performance assessments in state and/or local assessment systems.

Previous writings focus on why centering assessment design around students' cultural, social, and/or linguistic diversity is important and how performance-based assessment can support such aims. This article extends previous work by describing how a culturally responsive classroom assessment framework was created from a culturally responsive education (CRE) pedagogical framework. The goal of the framework was to guide the design and evaluation of curriculum-embedded, classroom performance assessments. Components discussed include: modification of evidence-centered design processes, teacher and/or student adaptation of construct irrelevant aspects of task prompts, addition of cultural meaningfulness questions to think alouds, and revision of task quality review protocols to promote CRE design features. Future research is needed to explore the limitations of the framework applied, and the extent to which students perceive the classroom summative assessments designed do indeed allow them to better show all they know and can do in ways related to their cultural, social, and/or linguistic identities.

New Hampshire's Performance Assessment of Competency Education (PACE) innovative assessment system uses student scores from classroom performance assessments as well as other classroom tests for school accountability purposes. One concern is that not having annual state testing may incentivize schools and teachers away from teaching the breadth of the state content standards. This study examined the effects of PACE on Grade 8 test scores after 5 years of implementation using propensity score matching followed by hierarchical linear modeling. The results suggest that PACE students perform about the same, on average, in mathematics and ELA as non-PACE students on the state assessment. There was no evidence of differential effects for students who had an individualized education program or were granted FRL. Findings for this limited sample suggest schools and teachers did not sacrifice the breadth of students' opportunity to learn the state content standards while piloting a state performance assessment reform.

New Hampshire's Performance Assessment of Competency Education (PACE) pilot received a waiver from federal statutory requirements related to state annual achievement testing starting in the 2014-15 school year. PACE is considered an "innovative" assessment and accountability system because performance assessments are used to help determine student proficiency in most federally required grades and subjects instead of the state achievement test. One key criterion for success in the early years of the PACE innovative assessment system is "no harm" on the statewide accountability test. This descriptive study examines the effect of PACE on Grades 8 and 11 mathematics and English language arts student achievement during the first three years of implementation (2014-15, 2015-16, and 2016-17 school years) and the extent to which those effects vary for certain student subgroups using results from the state's accountability tests (Smarter Balanced and SATs). Findings suggest that students in PACE schools tend to exhibit small positive effects on the Grades 8 and 11 state achievement tests in both subjects in comparison to students attending non-PACE comparison schools. Lower achieving students tended to exhibit small positive differential effects, whereas male students tended to exhibit small negative differential effects. Implications for research, policy, and practice are discussed.

This brief summarizes "Comparability in Balanced Assessment Systems for State Accountability," published in "Educational Measurement: Issues and Practice" (Evans & Lyons 2017). The study evaluated comparability claims in local scoring of performance assessments across districts participating in New Hampshire's Performance Assessment of Competency Education (PACE) pilot project. The study applied two judgmental methods for estimating comparability that are used in international contexts: consensus scoring and rank ordering. Across all districts, the consensus scoring yielded scores that were positive, meaning they were a bit lower than the teacher-given scores. Analysis revealed uneven scoring across districts, suggesting that there remains a need for additional training on scoring and within-district calibration, as well as for increased cross-district calibration. The rank-order method results revealed scoring differences across districts, most notably in one district in which teachers were scoring their student work a full standard deviation below (more rigorous) where the judges placed the same student work within the sample. Researchers found that applying the two methods in the context of the PACE system highlighted some strengths and limitations of each method. [For "Comparability in Balanced Assessment Systems for State Accountability," see EJ1154674.]

Summative performance assessments in teacher education, such as the Performance Assessment for California Teachers (PACT) and the edTPA, have been heralded through polices intended to enhance the quality of the teaching profession and raise its stature among other professions. However, the development and implementation of the PACT, and subsequently the edTPA, have not been without controversy and debate. The purpose of this article is to assess the implementation, impact, and evolution of the PACT and edTPA. To do so, we review the growing body of literature on the impact and implementation of the PACT and critically analyze the state policies surrounding the edTPA. We raise questions about policy and practical implications of the evolution of the PACT and edTPA.

This exploratory study uses data from 413 principals to examine whether and how competency-based education has been implemented in the Northeast states and the extent to which there is variation in implementation between states with different policies. Results suggest that competency-based practices that are most similar to current practices are reported more and practices that diverge from current practices are reported less. There were statistically significant differences between states with "advanced" and nonexistent competency-based education policies on three measures. Secondary principals could use this study to understand key features of the reform and the likely barriers and challenges to implementation regardless of their state policy context.

Background: Large electricity-generating wind turbines emit both audible sound and inaudible infrasound at very low frequencies that are outside of the normal human range of hearing. Sufferers of wind turbine syndrome (WTS) have attributed their ill-health and particularly their sleep disturbance to the signature pattern of infrasound. Critics have argued that these symptoms are psychological in origin and are attributable to nocebo effects. Objectives: We aimed to test the effects of 72 h of infrasound (1.6-20 Hz at a sound level of ~ 90 dB pk re 20 pPa, simulating a wind turbine infrasound signature) exposure on human physiology, particularly sleep. Methods: We conducted a randomized double-blind triple-arm crossover laboratory-based study of 72 h exposure with a >10-d washout conducted in a noise-insulated sleep laboratory in the style of a studio apartment. The exposures were infrasound (~90 dB pk), sham infrasound (same speakers not generating infrasound), and traffic noise exposure [active control; at a sound pressure level of 40-50 dB L[A.sub.eq,night] and 70 dB LA[F.sub.max] transient maxima, night (2200 to 0700 hours)]. The following physiological and psychological measures and systems were tested for their sensitivity to infrasound: wake after sleep onset (WASO; primary outcome) and other measures of sleep physiology, wake electroencephalography, WTS symptoms, cardiovascular physiology, and neurobehavioral performance. Results: We randomized 37 noise-sensitive but otherwise healthy adults (18-72 years of age; 51% female) into the study before a COVID19-related public health order forced the study to close. WASO was not affected by infrasound compared with sham infrasound (-1.36 min; 95% CI: -6.60, 3.88, p = 0.60) but was worsened by the active control traffic exposure compared with sham by 6.07 min (95% CI: 0.75, 11.39, p = 0.02). Infrasound did not worsen any subjective or objective measures used. Discussion: Our findings did not support the idea that infrasound causes WTS. High level, but inaudible, infrasound did not appear to perturb any physiological or psychological measure tested in these study participants. https://doi.org/10.1289/EHP10757, Introduction Large electricity-generating wind turbines emit both audible sound and inaudible infrasound at very low frequencies that are outside of the normal human range of hearing. There have been several [...]

The purpose of this study is to critically evaluate value-added accountability measures currently enacted in the United States at the federal and state levels to assess teacher preparation programme (TPP) effectiveness. We draw on Newton and Shaw's framework for the evaluation of testing policy to evaluate the technical quality and social acceptability of using K-12 student test scores to assess TPP effectiveness. Through six guiding questions, we examine the assumptions and arguments that support value-added assessment, culminating in overall judgments about the acceptability of implementing (or continuing to implement) the testing policy. Findings suggest policy-makers may have more pragmatic concerns about the efficiency of value-added assessment, while TPPs may have more theoretical concerns about the validity of value-added assessment. The relevance of this evaluation approach to improving policy-related decision-making will also be discussed.

The purpose of this study was to test methods that strengthen the comparability claims about annual determinations of student proficiency in English language arts, math, and science (Grades 3-12) in the New Hampshire Performance Assessment of Competency Education (NH PACE) pilot project. First, we examined the literature in order to define comparability outside the bounds of strict score interchangeability and explored methods for estimating comparability that support a balanced assessment system for state accountability such as the NH PACE pilot. Second, we applied two strategies--consensus scoring and a rank-ordering method--to estimate comparability in Year 1 of the NH PACE pilot based upon the expert judgment of 85 teachers using 396 student work samples. We found the methods were effective for providing evidence of comparability and also detecting when threats to comparability were present. The evidence did not indicate meaningful differences in district average scoring and therefore did not support adjustments to district-level cut scores used to create annual determinations. The article concludes with a discussion of the technical challenges and opportunities associated with innovative, balanced assessment systems in an accountability context.

This study investigates the predictive validity and policy impact of Council for Accreditation of Educator Preparation minimum admission requirements in Standard 3.2 on teacher preparation programs (TPPs), their applicants, and the broader field of educator preparation. Undergraduate grade point average (GPA) and Graduate Record Examination (GRE) scores from 533 program graduates in one master's-level TPP were examined for their ability to predict graduate GPA and the effect minimum admissions criteria had on enrollment. Findings indicate that only undergraduate GPA is moderately related to a program graduate's success, controlling for student background characteristics. The study also finds that implementing GRE scores as a criterion in admissions decisions significantly reduces the number of admitted candidates so that the program may no longer be financially sustainable. These findings suggest many negative consequences may result from minimum admission requirements and more research is needed to evaluate the potential impact on other TPPs, teacher labor markets, and student learning outcomes.

Background: Competency-based education (CBE) is a systems-change approach intended to re-shape traditional understandings of what, when, where, and how students learn and demonstrate academic knowledge and skills. Research on the factors that affect K-12 CBE implementation and the efficacy of different approaches has not yet been meticulously reviewed. Aims: The purpose of this literature review was to examine the research on K-12 CBE for factors that affect implementation, student outcomes, and the relationship between implementation and student outcomes. Methods: A systematic literature review was conducted that included 25 peer-reviewed studies and unpublished reports from 2000 to 2019 related to K-12 students. Results: Facilitators and barriers that affect K-12 CBE implementation were fairly consistent across studies. Factors perceived as barriers in some contexts were viewed as facilitators in others--it all depended on stage of implementation. Findings about the outcomes of CBE for K-12 students reflected mixed results with respect to claims that CBE implementation supports (a) academic achievement and progress; (b) intrinsic motivation and engagement; and (c) other important academic outcomes. Discussion: Undergirding all findings in this review is the difficulty of isolating the research on implementation and outcomes of K-12 CBE approaches in some "pure form." It may make more sense for the field to coalesce around a common continuum of practices in relation to the key elements of CBE from more traditional models to more competency-based models. Also, assessment as a key feature of CB implementation was absent from most of the studies reviewed which is notable given that determining competence is fundamentally an assessment decision. Directions for future research are discussed. Conclusion: For many, the promise of CBE and related practices is that student achievement will improve and minimize equity gaps. This systematic review serves to amplify what is known about CBE approaches and what still needs investigation.