Search

This paper presents a comparison of two commonly used methods, Angoff (W. Angoff, 1971) and Bookmark (D. Lewis, H. Mitzel, and D. Green, 1996), for setting cut scores on selected response tests. This comparison is presented through an application to a grade 7 mathematics assessment in a suburban Midwestern school district. Training and operational methods and procedures for each method are described in detail along with comparative results for the application. Although the Angoff method is more widely used, the Bookmark method has several strengths. It provides judges with an opportunity to focus on performance of the "Barely Proficient" students without worrying about estimating item difficulty. It also may be a more efficient method in terms of the length of time it takes for judges to make their bookmark placements. When results were compared, the recommended cut scores provided by the two methods were very similar, with the Bookmark method producing a lower standard deviation. (Author/SLD)

A key component of the Angoff (W. Angoff, 1971) method for setting cut scores is the target examinee. Expert judges are asked, following training and discussion, to consider the ability and the likely performance of some subset of examinees and to then estimate, item by item, the likely performance of such examinees on the test for which a cut score is desired. Questions of interest for this study were: (1) how teachers who serve as judges on cut score panels describe the target examinee; (2) how teachers' descriptions of the target examinee compare to the definitions provided by workshop facilitators as a frame for the discussion of target performance; (3) how teachers' descriptions of the target examinee compare to the behavioral characteristics arrived at during the workshop training; and (4) how descriptions of the target examinee compare across workshops. The results of two studies involving panel groups of 22 fourth-grade teachers and 20 ninth-grade teachers suggest that teachers who serve as judges in cut score setting processes are influenced by the definitions and training used in the standard setting study. Further, it appears that some common notion of the just competent examinee might be at work. Future studies should examine teachers' conception of the just competent students (given the purpose of the intended use of the cut score) before, as well as after, Angoff workshops. An appendix contains salient phrases extracted from teacher responses to the study prompts. (SLD)

A study was conducted to explore empirically the effect of different definitions of the target examinee on the judgment of panelists setting a passing score. Two cut score studies were done in a school district for the same test within a 6-month period, and different definitions of the target candidate were provided for each study. In October 1998, 15 teachers, all of whom were middle school or high school mathematics teachers, participated in the first standard setting exercise. The target student was called the "Barely Master Student," a student whose skills are sufficient to justify graduation, "but just barely." The methods and procedures for the April study were essentially the same, but the target examinee was designed to be more skilled. The 10 panelists who participated in both studies typically set higher standards when the target student definition represented a more skilled student. The findings suggest that under the same conditions, cut scores can be very reliably set across time and panels, but when elements of the cut score study change, the cut score may change accordingly. (SLD)

This paper reports the results of using several alternative methods of setting cut scores. The methods used were: (1) a variation of the Angoff method (1971); (2) a variation of the borderline group method; and (3) an advanced impact method (G. Dillon, 1996). The results discussed are from studies undertaken to set the cut scores for fourth grade reading and mathematics in a public school system. Twenty-two elementary school teachers served as judges. The three methods tended to result in similar cut scores with certain predictable variations. When the cut score was set in the lower tail of the distribution, the Angoff method tended to result in a cut score that was lower than the cut score set using the borderline group method. Recommendations are made for supplementing the Angoff method with additional data from alternative methods to improve the appropriateness of this method when setting performance standards in school settings. (Contains 15 references.) (Author/SLD)

Judgmental standard setting methods, such as the W. H. Angoff (1971) method, use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore, the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers, (L. A. Shephard 1994; J. Impara, 1997) have called into question the ability of the judges to make accurate item performance estimates for the target subgroups of candidates, such as minimally competent candidates (MCCs). The purpose of this study was to examine the validity of the judges' estimates of MCC performance that are used to determine a minimum passing score in a standard setting study context. Results from the operational standard setting workshops for a certification program in financial management that used 29 judges in 1996 and 30 in 1997 provide evidence that item performance estimates were valid. Factors that might have influenced this high degree of validity in the item performance estimates in the standard setting study are discussed. (Contains 10 references.) (SLD)

Many school districts are developing assessments that incorporate both selected response and constructed response formats. Scores on these assessments can be used for a variety of purposes ranging from subject remediation to promotion decisions. These policy decisions are informed by recommendations for Minimum Passing Scores (MPSs) from standard setting studies. This paper presents a model for setting MPSs on mixed assessments using a combination of two standard setting methodologies. For multiple-choice items, an Angoff standard setting approach is used. For constructed response items, a paper selection strategy is used. The model is described in detail and an illustrative case is presented for a large metropolitan school district in the Midwest. (Contains 4 tables and 11 references.) (Author/SLD)

Judgmental standard setting methods, such as the Angoff method (W. Angoff, 1971), use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers (L. Shepard, 1994; J. Impara, 1997) have called into question the ability of the judges to make accurate item performance estimates for target subgroups of candidates, such as minimally competent candidates. The purpose of this study was to examine both the reliability and accuracy of item performance estimates from an Angoff standard setting application using as a model the framework suggested by M. Kane (1994). Results from the standard setting example involving 29 judges for an international certification program in financial management provide evidence that item performance estimates were both valid and reliable. Factors that might have influenced this high degree of reliability and validity in the item performance estimates in a standard setting study are discussed. (Contains five tables and nine references.) (Author/SLD)

This paper reports on a standard-setting method designed for complex performance assessments with multiple performance categories. The method studied, the Analytical Judgment Method, involves panelists' making analytical classification decisions for each of the test's components individually. It also allows for discussion and reconsideration of panelists' initial classification decisions before moving on to the next test component. The method was used in two sites, Georgia and Michigan, on two different occasions using the 1996 eighth grade National Assessment of Educational Progress Science Assessment. The Georgia administration focused on the comparability of results from the Analytical Judgment Method when two different classification strategies, a sorting strategy and a direct classification strategy, were used. In Michigan, 2 versions of the rating scale were used, 1 with 12 classification categories and 1 with 7. Different performance descriptors were used in the two states. Sixteen teacher panelists in Georgia divided into 2 panels evaluated 50 papers per panel. In Michigan, 2 groups of 4 panelists each evaluated 50 papers. Results did not vary much as a function of the analytic strategy or whether a long or short classification scale was used. Discussion by panelists affected the results only when a seven-point scale was used, and the result was higher performance standards after discussion. The sorting method resulted in lower performance standards for Basic and Proficient levels, but not for Advanced. Results suggest that the decision regarding the method of analyzing the data and the choice of classification scale may be made as a matter of convenience or personal preference. (Contains nine tables, one figure, and nine references.) (SLD)

Setting performance standards on constructed-response assessments involving polytomously scored exercises presents a challenge for measurement practitioners. Some standard setting methods designed for use with multiple-choice, dichotomously scored assessments entail aggregating item performance estimates across a panel of experts. For these items, the experts are asked to predict the probability that a minimally competent candidate will correctly answer each of the items in the test. When working with constructed-response, polytomously scored assessments, panelists are often asked to predict the score that would be obtained by a minimally competent candidate and these expected score values are aggregated to determine the passing score. The resultant cutscore often has been found in practice to be unrealistically high. This study investigates the effectiveness of an adjustment technique to reduce the possible inflation of cutscores. Candidates are asked to estimate the proportion of minimally competent candidates who will answer the item (or pass the examination) correctly, and proportions are used as weights in computing the adjusted minimum passing score. The study applied the adjusted extended Angoff approach to a high school writing assessment involving 23 teachers. Application of the adjustment procedure was less than successful for a variety of reasons. The adjustment was minimal and panelists felt that it was unnecessary. In addition, the ramifications of revealing Round 2 results in order to gather these adjustments had negative consequences. Research is needed to study other possible adjustment strategies. (Contains one table and six references.) (SLD)

In the traditional Angoff Standard Setting Method, experts are instructed to predict the possibility that a randomly selected, hypothetical minimally competent candidate will be able to answer each multiple choice question in the test correctly. These item performance estimates are averaged across panelists and aggregated to determine the minimum passing score for the test. Some applications have used a modification of this method where panelists are instructed to provide their item performance estimates in deciles, with each decile representing a 10-point probability range. The purpose of the study was to investigate the validity of this approach, in terms of comparability of results to that which would occur from the traditional, open-ended administration procedures. Differences were found between the minimum passing scores across the two methods. A variation that gathered restricted item performance estimates for the initial round and reverted to the full probability scale for round 2 was shown to reduce these differences. Discussion focuses on situations where this variation to the modified Angoff method may be particularly attractive. (Contains one table and four references.) (Author/SLD)

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to estimate how well a randomly selected hypothetical examinee who is representative of a well-defined target group, usually a minimally competent candidate (MCC), will perform on each item on a test. Item performance estimates are difficult for panelists to make accurately; however, the validity of these performance standards rests on the ability of the panelist to conceptualize the skills and abilities of the MCC accurately and make accurate performance estimates. This study investigated the utility of a strategy to aid in the conceptualization of the MCC. Panelists were asked to envision the MCC and the typical student, and then make performance estimates, first for the typical student and then for the MCC. Results showed that the strategy resulted in significantly lower performance standards than did the traditional approach. Validity data were used to evaluate the accuracy of the judgments resulting from the experimental and traditional approaches. More research is needed to clarify the utility of concept-focusing strategy on the judgmental performance standards. (Contains 2 tables and 10 references.) (Author/SLD)

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to predict the probability that a randomly selected, hypothetical minimally competent candidate (MCC) will correctly answer test questions. Item performance predictions are difficult for panelists to do accurately; however, the validity of these performance standards rests on the ability of panelists to conceptualize the skills and abilities of the MCC accurately and to make accurate performance predictions. This study investigated the utility of a strategy to aid in the conceptualization of the MCC. Panelists were asked to envision the typical candidate and the MCC, and then to make item performance predictions for the typical student first and then the MCC. Results with 24 panelists predicting performance of students on a high school mathematics examination showed that the strategy resulted in significantly lower performance standards than did the traditional approach. Validity data were used to evaluate the accuracy of judgments resulting from the experimental and traditional approaches. More research is needed to clarify the utility of a concept focusing strategy on the judgmental performance standards. (Contains six references.) (Author/SLD)

This study investigated the intrajudge consistency of Angoff-based item performance estimates. The examination used was a certification examination in an emergency medicine specialty. Ten expert panelists rated the same 24 items twice during an operational standard setting study. Results indicate that the panelists were highly consistent, in terms of average absolute difference in item performance estimates (mean=0.073, standard deviation=0.066) and in resultant cutscores (16.37 versus 16.25). Features of the standard setting study and psychometric properties of the test were identified as possible contributors to this high level of intrajudge consistency. (Contains one table and eight references.) (Author/SLD)

The number of performance-based assessments is increasing rapidly, but to date there is no established procedure for setting standards on these assessments. This paper describes several extensions to the Angoff procedure to accommodate the characteristics of a performance-based assessment and presents the results of research in applying this procedure. The extensions included a revised task for panelists that involved the specification of expected scores for just barely certifiable candidates on polytomously scored exercises and weights to reflect the relative importance of scoring dimensions at the exercise level and of the exercises themselves. Results obtained from using the new procedure with 12 panelists (teachers of adolescents with experience in pilot tests of the assessment) were mixed. On one hand, panelists seemed to find the new procedure straightforward to apply and were able to reach a high level of agreement among themselves about the standards. Confidence levels in the resulting standards were high. However, when given the choice, panelists definitely preferred a standard-setting procedure that was at least partly conjunctive as opposed to the extended Angoff procedure that was completely compensatory in nature. Ten tables and one figure present study findings. (Contains 7 references.) (SLD)

Several extensions to the Angoff method of standard setting are described that can accommodate characteristics of performance-based assessment. A study involving 12 panelists supported the effectiveness of the new approach but suggested that panelists preferred an approach that was at least partially conjunctive. (SLD)

The three standard-setting approaches described in this special issue are summarized and contrasted: (1) judgmental policy capturing; (2) the extended Angoff method; and (3) the dominant profile method. An integrative summary of findings is followed by recommendations for modifying the methods. (SLD)

This article provides a framework for the rest of the articles in this special issue comparing the utility of three standard-setting methods with complex performance assessments. The context of the standard setting study is described, and the methods are outlined. (SLD)

The comparability of Angoff-based item ratings on a general education test battery made by judges from within-content and across-content domains was studied. Results with 26 college faculty judges indicate that, at least for some tests, item ratings might be essentially equivalent regardless of judge's content specialty. (SLD)