Setting Performance Standards on Polytomously Scored Assessments: An Adjustment to the Extended Angoff Method.

Setting performance standards on constructed-response assessments involving polytomously scored exercises presents a challenge for measurement practitioners. Some standard setting methods designed for use with multiple-choice, dichotomously scored assessments entail aggregating item performance estimates across a panel of experts. For these items, the experts are asked to predict the probability that a minimally competent candidate will correctly answer each of the items in the test. When working with constructed-response, polytomously scored assessments, panelists are often asked to predict the score that would be obtained by a minimally competent candidate and these expected score values are aggregated to determine the passing score. The resultant cutscore often has been found in practice to be unrealistically high. This study investigates the effectiveness of an adjustment technique to reduce the possible inflation of cutscores. Candidates are asked to estimate the proportion of minimally competent candidates who will answer the item (or pass the examination) correctly, and proportions are used as weights in computing the adjusted minimum passing score. The study applied the adjusted extended Angoff approach to a high school writing assessment involving 23 teachers. Application of the adjustment procedure was less than successful for a variety of reasons. The adjustment was minimal and panelists felt that it was unnecessary. In addition, the ramifications of revealing Round 2 results in order to gather these adjustments had negative consequences. Research is needed to study other possible adjustment strategies. (Contains one table and six references.) (SLD)