Utility of a Concept-Focusing Strategy on Judgmental Standard Setting Results.

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to estimate how well a randomly selected hypothetical examinee who is representative of a well-defined target group, usually a minimally competent candidate (MCC), will perform on each item on a test. Item performance estimates are difficult for panelists to make accurately; however, the validity of these performance standards rests on the ability of the panelist to conceptualize the skills and abilities of the MCC accurately and make accurate performance estimates. This study investigated the utility of a strategy to aid in the conceptualization of the MCC. Panelists were asked to envision the MCC and the typical student, and then make performance estimates, first for the typical student and then for the MCC. Results showed that the strategy resulted in significantly lower performance standards than did the traditional approach. Validity data were used to evaluate the accuracy of the judgments resulting from the experimental and traditional approaches. More research is needed to clarify the utility of concept-focusing strategy on the judgmental performance standards. (Contains 2 tables and 10 references.) (Author/SLD)