Search

Using self-reported but empirically verified repeater groups, we analyzed vast amounts of real test data across a wide range of administrations from a graduate admissions examination that was administered in a non-English language to investigate repeater effects on score equating using the nonequivalent groups with anchor test (NEAT) design. Both linear and nonlinear equating models were considered in deriving the equating functions for various study groups. We evaluated scaled score differences between equating in the total group, the repeater group, and the first-timer group using statistics of simple differences and subpopulation invariance measures developed and used widely in the last 10 years. Standard errors of statistics summarizing scaled score differences were estimated using a simulation approach to provide statistical criteria for evaluating the significance of equating differences. In addition, we used scaled score differences that were critical to admissions screening as criteria for evaluating the practical significance of equating differences. To put the investigation of repeater effects in proper perspective, we analyzed the repeater data for an in-depth understanding of repeater performance trends. Overall, we found no significant effects of repeater performance on score equating for the exam being studied. Although many of the equating differences were practically significant, most of the practically significant differences were not statistically significant. However, further research with larger repeater samples is recommended to help explain the practical significance of equating differences consistently observed in this study for the repeater group. Potential problems associated with small repeater study sample sizes, issues of the practical criterion for evaluating the significance of equating differences, and study limitations are also discussed. (Contains 8 tables, 25 figures and 8 notes.)

Scores on the multiple-choice sections of alternate forms are equated through anchor-test equating for the Advanced Placement Program (AP) examinations. There is no linkage of free-response sections since different free-response items are given yearly. However, the free-response and multiple-choice sections are combined to produce a composite. Therefore, to derive new-form cut scores on the composite score scale that are comparable to the old-form cut scores, the multiple-choice score of the AP examinations are linked to the composite score through single-group linking design. This study investigates whether the multiple-choice to composite linking functions remain invariant over subgroups by region for two AP examinations using 3 years of test data. The region groups are of interest because the AP program administers different free-response sections to different time-zone regions as a precaution for security reasons. The study focused on: (1) how invariant cut scores are across regions; and (2) whether the small sample size for some regional groups presents particular problems for assessing linking invariance. Both equipercentile and linear linking methods were applied. The equatability index proposed by N. Dorans and P. Holland (2000) is used to evaluate the invariance of the linking functions, and the cross-classification approach is used to evaluate the invariance of the composite cut scores. Overall, the curvilinear linkings across regions seem to hold up reasonably well. Nevertheless, more examinations are needed to decide whether the linkings in the small regions contain enough data to support the use of weak models such as equipercentile scaling. The strong smoothing associated with the linear linking model, as expected, reduces the variability of linkages across regions. However, more work is needed to see if less strong models can be found for improving consistency over the unsmoothed linkings without incurring the bias apparent with the linear linkings. (Author/SLD)

This study investigated whether equating accuracy improves with an anchor test that is more representative of its corresponding total test and whether such content effect depends on the particular equating method used. Scoring outcomes of a professional examination for a medical specialty were used. A total of 1,092 examinees took one form, and 1,149 took the other form. Item-sampling schemes were introduced to manipulate the content homogeneity and representation of anchor items so that their effects on equating accuracy could be studied with varied equating methods. Multiple criteria were proposed to evaluate equating accuracy. Overall, the study found that all the equating methods employed had accurate results to a moderate degree. They all produced more accurate results when the anchor items were more representative of their total test or when the content coverage of a test concentrated on fewer topics. In general, Item Response Theory-based equating was found to be more accurate than classical equating, but the small differences may not be substantial. To produce equating outcomes that are precise and less prone to bias, various equating methods should be considered for equating test forms for high-stakes examinations. (Contains 4 tables, 3 figures, and 22 references.) (SLD)

This paper provides a historical review of the changes and improvements made in estimating numerical cutpoints for the National Assessment of Educational Progress (NAEP). While reviewing the various methodologies used for collecting judgment data, the paper discusses: (1) the incorporation of Item Response Theory for setting standards; (2) methodologies for converting judgment data into achievement level cutpoints; (3) combining cutpoints estimated for dichotomously scored multiple-choice items and polytomously scored constructed response items; and (4) deriving group cutpoints from the estimates of individual cutpoints for items and judges. In addition to the eight formal achievement level setting (ALS) processes that have been used, pilot studies and field trials have evaluated other approaches before the formal standard-setting process. In general, methodologies can be categorized into item-by-term rating (analytical) and holistic approaches. This review illustrates the importance and significance of the application of Item Response Theory in setting NAEP achievement levels. (Contains 1 table and 25 references.) (SLD)

The Achievement-Levels Setting (ALS) process for the National Assessment of Educational Progress (NAEP) resulted in numerical cutscores on the NAEP score scale representing the performance standards for three achievement levels: Basic, Proficient, and Advanced. This paper focuses on an important, but less researched, aspect of the standard setting process using an item-rating approach--the patterns and changes in panelists' item ratings. Instead of analyzing the cutpoints based on item ratings, this paper presents analyses of intrarater consistency in item ratings within and across rounds of ratings, reasonableness of panelists' item ratings, and changes in ratings, in relation to cutpoints, across rounds, using the NAEP Civics examination as an example. For the 1998 achievement level setting process, the NAEP incorporated "Reckase" charts to provide extensive and easy-to-understand feedback regarding the consistency of the panelists' own ratings in the Item Response Theory context. Although the impact of Reckase charts on improving intrarater consistency could not be examined directly, Reckase generally indicated that they found the charts to be useful and informative. The analysis findings summarized in this paper are regarded as evidence of the procedural validity for the 1998 ALS process and the safeguard for the validity of the 1998 ALS outcomes. Generally, panelists were able to improve their item ratings over time. The intervention aimed at improving intrarater consistency seemed to be effective, but it was not dominating in driving panelists' subsequent item ratings. (Contains 3 tables, 4 figures, and 13 references.) (SLD)

Using an anchor-item design of test equating, the effects of three equating methods (Tucker linear and two three-parameter item-response-theory-based (3PL-IRT) methods), and the content representativeness of anchor items on the accuracy of equating were examined; and an innovative way of evaluating equating accuracy appropriate for the particular item-sampling design of the study was introduced. Data analyzed were test results from 2 forms of a professional competency examination with 197 and 203 items respectively. There were 145 anchor items embedded in both forms, and the 2 examinee groups were not randomly formed. From the two test forms, four pairs of shortened test forms were created to differ in the content representativeness of their anchor items. The total raw score on the original anchor items was regarded as a "pseudo true score," which was used as a criterion for evaluating equating accuracy. Overall, the three equating methods appeared to yield moderately accurate equating results on every test, but the outcomes of the IRT-based methods seemed to be more accurate than the outcomes of the Tucker method. The accuracy of equating depended on the content representativeness of the anchor items, no matter which method was used to equate test forms. The 3PL-IRT model seemed appropriate for equating the test form with negative skewed score distribution. One appendix presents the item sampling schemes and the other contains tables of correlation analyses on anchor and nonanchor items. (Contains 6 tables, 2 figures, and 58 references.) (SLD)

International comparisons in educational research can be difficult to accomplish because the findings of individual countries may not be comparable due to study design or inherent country features that cannot be manipulated. Quantitative meta-analysis techniques have great potential in improving international comparisons. In this paper, participant countries/regions in an international study were treated as study populations, and meta-analytic techniques were used to synthesize study outcomes across countries. Homogeneity tests were conducted to determine whether there was a common population parameter across countries, outliers were identified empirically, moderator effects die to country characteristics were studied, and homogeneous country outcomes were combined by a variance-weighting method to yield an optimal parameter estimate. The study of interest in this paper was gender differences in students' perceptions about whether girls or boys will do better in mathematics. Data are a subset of data from the Third International Mathematics and Science Study (TIMSS), in which seventh- and eighth-graders from 25 countries participated. Multiple regression models were used to country-level analyses, and effect sizes were computed. Overall, the meta-analytic techniques were satisfactory in analyzing the TIMSS data, illustrating the potential of meta analysis in improving the validity of international comparison studies. Meta analysis appears to be capable of detecting substantial differences in country outcomes and effective in offering strategies to deal with the situation. Appendixes present tables of coding schemes and outcomes of moderators, independent variables and questionnaire items, and a summary of meta analytic results. (Contains 7 tables, 8 figures, and 20 references.) (SLD)

The influence of anchor length on the accuracy of test equating was studied using Tucker's linear method and two Item-Response-Theory (IRT) based methods, focusing on whether equating accuracy improved with more anchor items, whether the anchor effect depended on the equating method used, and the adequacy of the inclusion of the guessing parameter for a test that had a negatively skewed distribution of scores. Data were from 2 forms of a minimum competency examination that contained 197 and 203 items respectively. Three pairs of shorted forms were created by the simple random sampling of items, and the pairs were equated separately. The total score on the 145 anchor items was used as a criterion, a pseudo true score, to evaluate result accuracy. True score estimates were obtained that were correlated to the pseudo true score. Overall, results yielded by all three equating methods were moderately accurate, and no matter which equating method was used, the results tended to be more accurate when there were more anchor items. In addition, inclusion of a guessing parameter was justified. Six appendixes present the sampling scheme for the reduced forms, item correlations and descriptive test statistics for the reduced and full forms, and the two equating methods. (Contains 1 figure, 4 tables, 18 appendix tables, and 46 references.) (SLD)