Search

This study examined the time different ability level examinees spend taking a CAT on demanding items to these examinees. It was also found that high able examinees spend more time on the pretest items, which are not tailored to the examinees' ability level, than do lower ability examinees. Higher able examinees showed persistence with test questions, regardless of the item's difficulty level on operational and pretest questions they answer correctly or incorrectly. Results showed that more able students spend more time on all items, regardless of whether the items are answered correctly or incorrectly. These results were consistent for male and female examinees and for US and non-US administration sites.

One Midwestern state has chosen a model of state assessment in which local school districts are responsible for developing the strategies to measure and report their students' performance on state-adopted content standards. When students are not measured on common instruments, district accountability becomes an added challenge. This paper presents a strategy for evaluating locally developed assessments as part of the state assessment system that can be used to inform the need for state accountability. An application of the strategy is described in which a 16-member District Assessment Evaluation Team was recruited to evaluate district assessment portfolios for the state. An analysis of inter-rater agreement on the common districts evaluated by all 19 raters (16 team members and 3 anchor raters) was performed. A goal of 80% agreement or higher was established, but there was some variance in the levels of agreement for each criterion and the overall ratings. Results indicate there was reasonable consistency among raters. Benefits of the proposed strategy include an emphasis on formative rather than summative feedback and on improving assessment strategies at the local level. Locally developed assessments that are aligned to the state's content standards and integrated into the district's curriculum are likely to produce data that is meaningful to the state, yet can inform instruction in the classroom. appended are tables of data used in the study. (Author/SLD)

This paper presents a comparison of two commonly used methods, Angoff (W. Angoff, 1971) and Bookmark (D. Lewis, H. Mitzel, and D. Green, 1996), for setting cut scores on selected response tests. This comparison is presented through an application to a grade 7 mathematics assessment in a suburban Midwestern school district. Training and operational methods and procedures for each method are described in detail along with comparative results for the application. Although the Angoff method is more widely used, the Bookmark method has several strengths. It provides judges with an opportunity to focus on performance of the "Barely Proficient" students without worrying about estimating item difficulty. It also may be a more efficient method in terms of the length of time it takes for judges to make their bookmark placements. When results were compared, the recommended cut scores provided by the two methods were very similar, with the Bookmark method producing a lower standard deviation. (Author/SLD)

Because districts in Nebraska are not measured on common instruments, comparisons are difficult. This study examined a district evaluation strategy that classifies districts into school performance ratings (SPR) based on a combination of three factors. Student performance and non-cognitive indicator data for three grade levels and two content areas from school districts in a southern state were used. Analyses comparing classification decision consistency for three mathematical models were conducted using Spearman rank order correlations for composite scale scores (CSS) and kappa statistics for SPR classifications. Results show that there is a high level of agreement among the three mathematical models considered, suggesting that the preferred model be the one that is easiest to understand and communicate. (Contains 2 tables and 10 references.) (Author/SLD)

Three separate studies were designed to provide information on the alignment of norm-referenced achievement test items in language arts to Nebraska's content standards at grades 4, 8, and 11. The information was used to evaluate the alignment of the tests to the standards as well as compare that alignment across the five tests. There were 20 teachers on the 4th-grade panel, 10 teachers on the 8th grade panel, and 10 on the 11th grade panel. Teachers from school districts across the state were selected to participate in these studies. The teachers evaluated how well items from these tests aligned to content standards in reading, writing, speaking, and listening. Results for language arts showed moderate alignment between the norm-referenced test items and standards. These studies were used as validity evidence for Nebraska's state assessment model that permits school districts to select the norm-referenced test that best meets their needs and then to supplement that test with district developed assessments to address what the norm-referenced tests do not measure. (Contains 3 tables and 10 references.) (SLD)

Test publishers have promoted their commercially available norm-referenced achievement tests as viable solutions to assessment challenges faced by states. They argue that their tests are developed professionally and therefore possess sound psychometric properties not often found in state-specific efforts. This study compared judgments from two sources, test publishers and teachers, on the alignment of test items from two commercially available norm-referenced achievement tests to Nebraska's content standards at three grade levels (4, 8, and 11) in language arts. Study analyses focused on the level of agreement between the state's teachers' perceptions of how well the tests aligned to the standards and the alignment reported by the two test publishers. Panels of 20 4th grade teachers, 10 8th grade teachers, and 10 11th grade teachers studied the alignment. Results indicate that there may be some inconsistency between teachers' and publishers' perceptions. (Author/SLD)

Many states use a statewide assessment strategy to evaluate districts on common measures. Because districts in Nebraska are not measured on common instruments, comparisons of district performance are difficult. The accountability model described in this paper contains three components related to school district performance on academic content standards: (1) student performance on district assessments; (2) technical quality of district assessments; and (3) challenge adjustments based on district demographic characteristics. The model is proposed based on a previous study that examined the appropriate component contributions of the model, the utility of creating composite scores for accountability, and the stability of district classification decisions. The current accountability model does not include a component for improvement over time and is suggested as a strategy for low stakes accountability systems. (Contains 23 references.) (Author/SLD)

Test publishers have promoted their commercially available, norm-referenced achievement tests as viable solutions to assessment challenges faced by states. They argue that their tests are developed professionally, and, therefore, possess sound psychometric properties not often found in state-specific efforts. This study compared judgments from two sources, test publishers from two companies and teachers, on the alignment of test items from two commercially available norm-referenced achievement tests in a Midwestern state's content standards at three grade levels in reading/writing and mathematics. Teacher panels consisted of 10 teachers for each of language and mathematics panels at grade 8 and high school levels and 20 teachers for each panel for grade 4. Analyses in this study focus on the level of agreement between the state's teachers' perceptions of how well the tests aligned to the standards and the alignment reported by the two test publishers. Results indicate that there may be some inconsistency between teachers' and publishers' perceptions. (Author/SLD)

A key component of the Angoff (W. Angoff, 1971) method for setting cut scores is the target examinee. Expert judges are asked, following training and discussion, to consider the ability and the likely performance of some subset of examinees and to then estimate, item by item, the likely performance of such examinees on the test for which a cut score is desired. Questions of interest for this study were: (1) how teachers who serve as judges on cut score panels describe the target examinee; (2) how teachers' descriptions of the target examinee compare to the definitions provided by workshop facilitators as a frame for the discussion of target performance; (3) how teachers' descriptions of the target examinee compare to the behavioral characteristics arrived at during the workshop training; and (4) how descriptions of the target examinee compare across workshops. The results of two studies involving panel groups of 22 fourth-grade teachers and 20 ninth-grade teachers suggest that teachers who serve as judges in cut score setting processes are influenced by the definitions and training used in the standard setting study. Further, it appears that some common notion of the just competent examinee might be at work. Future studies should examine teachers' conception of the just competent students (given the purpose of the intended use of the cut score) before, as well as after, Angoff workshops. An appendix contains salient phrases extracted from teacher responses to the study prompts. (SLD)

A study was conducted to explore empirically the effect of different definitions of the target examinee on the judgment of panelists setting a passing score. Two cut score studies were done in a school district for the same test within a 6-month period, and different definitions of the target candidate were provided for each study. In October 1998, 15 teachers, all of whom were middle school or high school mathematics teachers, participated in the first standard setting exercise. The target student was called the "Barely Master Student," a student whose skills are sufficient to justify graduation, "but just barely." The methods and procedures for the April study were essentially the same, but the target examinee was designed to be more skilled. The 10 panelists who participated in both studies typically set higher standards when the target student definition represented a more skilled student. The findings suggest that under the same conditions, cut scores can be very reliably set across time and panels, but when elements of the cut score study change, the cut score may change accordingly. (SLD)

This paper reports the results of using several alternative methods of setting cut scores. The methods used were: (1) a variation of the Angoff method (1971); (2) a variation of the borderline group method; and (3) an advanced impact method (G. Dillon, 1996). The results discussed are from studies undertaken to set the cut scores for fourth grade reading and mathematics in a public school system. Twenty-two elementary school teachers served as judges. The three methods tended to result in similar cut scores with certain predictable variations. When the cut score was set in the lower tail of the distribution, the Angoff method tended to result in a cut score that was lower than the cut score set using the borderline group method. Recommendations are made for supplementing the Angoff method with additional data from alternative methods to improve the appropriateness of this method when setting performance standards in school settings. (Contains 15 references.) (Author/SLD)

Judgmental standard setting methods, such as the W. H. Angoff (1971) method, use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore, the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers, (L. A. Shephard 1994; J. Impara, 1997) have called into question the ability of the judges to make accurate item performance estimates for the target subgroups of candidates, such as minimally competent candidates (MCCs). The purpose of this study was to examine the validity of the judges' estimates of MCC performance that are used to determine a minimum passing score in a standard setting study context. Results from the operational standard setting workshops for a certification program in financial management that used 29 judges in 1996 and 30 in 1997 provide evidence that item performance estimates were valid. Factors that might have influenced this high degree of validity in the item performance estimates in the standard setting study are discussed. (Contains 10 references.) (SLD)

This study investigated the effect on examinees' ability estimate under item response theory (IRT) when they are presented an item, have ample time to answer the item, but decide not to respond to the item. Simulation data were modeled on an empirical data set of 25,546 examinees that was calibrated using the 3-parameter logistic model. The study was conducted in three phases. Phase 1 was an exploratory study comparing the various estimation methods under different conditions. Methods compared were the Biweight, expected a posteriori (EAP), and maximum likelihood estimation (MLE) methods. Phase 2, based on Phase 1 results, examined a modified EAP approach. The third phase examined the use of R. Bock's 1972 nominal response (NR) model for handling omission data. Results seem to indicate that omits should not be treated as incorrect. It also appears that ignoring omits can have a greater impact on thetas using certain estimation approaches (e.g., MLE) than with others. To the extent that a model with a pseudo-guessing parameter more accurately describes the data than a model without one, then the use of the NR model may not produce results comparable to those seen here. (Contains 5 tables, 10 figures, and 14 references.) (SLD)

Many school districts are developing assessments that incorporate both selected response and constructed response formats. Scores on these assessments can be used for a variety of purposes ranging from subject remediation to promotion decisions. These policy decisions are informed by recommendations for Minimum Passing Scores (MPSs) from standard setting studies. This paper presents a model for setting MPSs on mixed assessments using a combination of two standard setting methodologies. For multiple-choice items, an Angoff standard setting approach is used. For constructed response items, a paper selection strategy is used. The model is described in detail and an illustrative case is presented for a large metropolitan school district in the Midwest. (Contains 4 tables and 11 references.) (Author/SLD)

Judgmental standard setting methods, such as the Angoff method (W. Angoff, 1971), use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers (L. Shepard, 1994; J. Impara, 1997) have called into question the ability of the judges to make accurate item performance estimates for target subgroups of candidates, such as minimally competent candidates. The purpose of this study was to examine both the reliability and accuracy of item performance estimates from an Angoff standard setting application using as a model the framework suggested by M. Kane (1994). Results from the standard setting example involving 29 judges for an international certification program in financial management provide evidence that item performance estimates were both valid and reliable. Factors that might have influenced this high degree of reliability and validity in the item performance estimates in a standard setting study are discussed. (Contains five tables and nine references.) (Author/SLD)

This paper reports on a standard-setting method designed for complex performance assessments with multiple performance categories. The method studied, the Analytical Judgment Method, involves panelists' making analytical classification decisions for each of the test's components individually. It also allows for discussion and reconsideration of panelists' initial classification decisions before moving on to the next test component. The method was used in two sites, Georgia and Michigan, on two different occasions using the 1996 eighth grade National Assessment of Educational Progress Science Assessment. The Georgia administration focused on the comparability of results from the Analytical Judgment Method when two different classification strategies, a sorting strategy and a direct classification strategy, were used. In Michigan, 2 versions of the rating scale were used, 1 with 12 classification categories and 1 with 7. Different performance descriptors were used in the two states. Sixteen teacher panelists in Georgia divided into 2 panels evaluated 50 papers per panel. In Michigan, 2 groups of 4 panelists each evaluated 50 papers. Results did not vary much as a function of the analytic strategy or whether a long or short classification scale was used. Discussion by panelists affected the results only when a seven-point scale was used, and the result was higher performance standards after discussion. The sorting method resulted in lower performance standards for Basic and Proficient levels, but not for Advanced. Results suggest that the decision regarding the method of analyzing the data and the choice of classification scale may be made as a matter of convenience or personal preference. (Contains nine tables, one figure, and nine references.) (SLD)

Setting performance standards on constructed-response assessments involving polytomously scored exercises presents a challenge for measurement practitioners. Some standard setting methods designed for use with multiple-choice, dichotomously scored assessments entail aggregating item performance estimates across a panel of experts. For these items, the experts are asked to predict the probability that a minimally competent candidate will correctly answer each of the items in the test. When working with constructed-response, polytomously scored assessments, panelists are often asked to predict the score that would be obtained by a minimally competent candidate and these expected score values are aggregated to determine the passing score. The resultant cutscore often has been found in practice to be unrealistically high. This study investigates the effectiveness of an adjustment technique to reduce the possible inflation of cutscores. Candidates are asked to estimate the proportion of minimally competent candidates who will answer the item (or pass the examination) correctly, and proportions are used as weights in computing the adjusted minimum passing score. The study applied the adjusted extended Angoff approach to a high school writing assessment involving 23 teachers. Application of the adjustment procedure was less than successful for a variety of reasons. The adjustment was minimal and panelists felt that it was unnecessary. In addition, the ramifications of revealing Round 2 results in order to gather these adjustments had negative consequences. Research is needed to study other possible adjustment strategies. (Contains one table and six references.) (SLD)

In the traditional Angoff Standard Setting Method, experts are instructed to predict the possibility that a randomly selected, hypothetical minimally competent candidate will be able to answer each multiple choice question in the test correctly. These item performance estimates are averaged across panelists and aggregated to determine the minimum passing score for the test. Some applications have used a modification of this method where panelists are instructed to provide their item performance estimates in deciles, with each decile representing a 10-point probability range. The purpose of the study was to investigate the validity of this approach, in terms of comparability of results to that which would occur from the traditional, open-ended administration procedures. Differences were found between the minimum passing scores across the two methods. A variation that gathered restricted item performance estimates for the initial round and reverted to the full probability scale for round 2 was shown to reduce these differences. Discussion focuses on situations where this variation to the modified Angoff method may be particularly attractive. (Contains one table and four references.) (Author/SLD)

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to estimate how well a randomly selected hypothetical examinee who is representative of a well-defined target group, usually a minimally competent candidate (MCC), will perform on each item on a test. Item performance estimates are difficult for panelists to make accurately; however, the validity of these performance standards rests on the ability of the panelist to conceptualize the skills and abilities of the MCC accurately and make accurate performance estimates. This study investigated the utility of a strategy to aid in the conceptualization of the MCC. Panelists were asked to envision the MCC and the typical student, and then make performance estimates, first for the typical student and then for the MCC. Results showed that the strategy resulted in significantly lower performance standards than did the traditional approach. Validity data were used to evaluate the accuracy of the judgments resulting from the experimental and traditional approaches. More research is needed to clarify the utility of concept-focusing strategy on the judgmental performance standards. (Contains 2 tables and 10 references.) (Author/SLD)

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to predict the probability that a randomly selected, hypothetical minimally competent candidate (MCC) will correctly answer test questions. Item performance predictions are difficult for panelists to do accurately; however, the validity of these performance standards rests on the ability of panelists to conceptualize the skills and abilities of the MCC accurately and to make accurate performance predictions. This study investigated the utility of a strategy to aid in the conceptualization of the MCC. Panelists were asked to envision the typical candidate and the MCC, and then to make item performance predictions for the typical student first and then the MCC. Results with 24 panelists predicting performance of students on a high school mathematics examination showed that the strategy resulted in significantly lower performance standards than did the traditional approach. Validity data were used to evaluate the accuracy of judgments resulting from the experimental and traditional approaches. More research is needed to clarify the utility of a concept focusing strategy on the judgmental performance standards. (Contains six references.) (Author/SLD)

This study investigated the intrajudge consistency of Angoff-based item performance estimates. The examination used was a certification examination in an emergency medicine specialty. Ten expert panelists rated the same 24 items twice during an operational standard setting study. Results indicate that the panelists were highly consistent, in terms of average absolute difference in item performance estimates (mean=0.073, standard deviation=0.066) and in resultant cutscores (16.37 versus 16.25). Features of the standard setting study and psychometric properties of the test were identified as possible contributors to this high level of intrajudge consistency. (Contains one table and eight references.) (Author/SLD)

The number of performance-based assessments is increasing rapidly, but to date there is no established procedure for setting standards on these assessments. This paper describes several extensions to the Angoff procedure to accommodate the characteristics of a performance-based assessment and presents the results of research in applying this procedure. The extensions included a revised task for panelists that involved the specification of expected scores for just barely certifiable candidates on polytomously scored exercises and weights to reflect the relative importance of scoring dimensions at the exercise level and of the exercises themselves. Results obtained from using the new procedure with 12 panelists (teachers of adolescents with experience in pilot tests of the assessment) were mixed. On one hand, panelists seemed to find the new procedure straightforward to apply and were able to reach a high level of agreement among themselves about the standards. Confidence levels in the resulting standards were high. However, when given the choice, panelists definitely preferred a standard-setting procedure that was at least partly conjunctive as opposed to the extended Angoff procedure that was completely compensatory in nature. Ten tables and one figure present study findings. (Contains 7 references.) (SLD)

In April 1991, the Board of the National Council on Measurement in Education (NCME) voted to accept a grant from the Kellogg Foundation to: (1) measure teacher competency levels in the educational assessment of students; (2) identify a topic for the development of a training prototype; and (3) prepare a prototype training module on the selected topic. Efforts toward meeting these goals are summarized. Goal 1 was achieved through a national survey of teacher competencies in the area of educational assessment of students responded to by 555 teachers (48 percent response rate) and 286 administrators (51 percent response). On the basis of that survey and other relevant data, the topic "Interpreting and Communicating Test Results" was selected for development of a training prototype. Efforts toward Goal 3 are in progress. The training modules will consist of several components, designed to be delivered in either a 1-day or 2-day inservice program or through a series of 2-hour training sessions. The training modules will consist of a context-orienting vignette, course training materials, practice exercises, hands-on experiences, small group discussion questions, and homework problems. This prototype could serve as a model for future development efforts for training teachers in educational assessment. (Author/SLD)

A two-stage testing strategy is one method of adapting the difficulty of a test to an individual's ability level in an effort to achieve more precise measurement. A routing test provides an initial estimate of ability level, and a second-stage measurement test then evaluates the examinee further. The measurement accuracy and efficiency of item response theory (IRT) based two-stage testing was investigated in comparison with an individualized computerized adaptive test (CAT). Eighteen simulated two-stage tests and three fixed-length CATs differing in the number of test items administered were compared. Abilities were generated for a sample of 1,600 simulees. Results indicate that the statistical characteristics of the routing test have a major influence on measurement precision in ability estimation. Overall, it was apparent that a fixed-length CAT is superior to the two-stage tests of equivalent length in terms of measurement accuracy and efficiency. IRT-based two-stage tests using rectangular distribution of item difficulties in the routing test and an odd number of second-stage tests produced more accurate theta estimates than did the other two-stage test configurations studied. IRT-based two-stage tests may sometimes be practical alternatives to CAT, considering its limitations. Two tables and 18 graphs present analysis results. (SLD)