Interjudge Consensus and Intrajudge Consistency: Is It Possible To Have Both in Standard Setting?

Eleven judges representing 11 different geographic regions in the United States participated in a standard-setting session designed to determine the possibility of obtaining interjudge consensus and intrajudge consistency simultaneously. Each judge had experience in the field for which standards were being set. The judges rated 65 multiple-choice test items that are part of a national health certification examination. All items had been field tested. The judges participated in a rater-training session that included a discussion of the Angoff method and the conceptualization of skills needed for minimum competence. After all items were rated, handouts containing a list of all ratings for all judges were reviewed and discussed by the judges. All judges changed their ratings on at least 4 of the items, and 2 judges changed their ratings on 20 or more items. Results indicate that the procedures used to improve interjudge consensus (that is, provision of feedback via the handouts) did not have an adverse affect on intrajudge consistency. In fact, results show that techniques designed to improve consensus also improved consistency. Two data tables and two bar graphs are included. (TJH)