Search

This paper reports on a standard-setting method designed for complex performance assessments with multiple performance categories. The method studied, the Analytical Judgment Method, involves panelists' making analytical classification decisions for each of the test's components individually. It also allows for discussion and reconsideration of panelists' initial classification decisions before moving on to the next test component. The method was used in two sites, Georgia and Michigan, on two different occasions using the 1996 eighth grade National Assessment of Educational Progress Science Assessment. The Georgia administration focused on the comparability of results from the Analytical Judgment Method when two different classification strategies, a sorting strategy and a direct classification strategy, were used. In Michigan, 2 versions of the rating scale were used, 1 with 12 classification categories and 1 with 7. Different performance descriptors were used in the two states. Sixteen teacher panelists in Georgia divided into 2 panels evaluated 50 papers per panel. In Michigan, 2 groups of 4 panelists each evaluated 50 papers. Results did not vary much as a function of the analytic strategy or whether a long or short classification scale was used. Discussion by panelists affected the results only when a seven-point scale was used, and the result was higher performance standards after discussion. The sorting method resulted in lower performance standards for Basic and Proficient levels, but not for Advanced. Results suggest that the decision regarding the method of analyzing the data and the choice of classification scale may be made as a matter of convenience or personal preference. (Contains nine tables, one figure, and nine references.) (SLD)