Search

The purpose of this study was to compare and evaluate three online pretest item calibration/scaling methods in terms of item parameter recovery when the item responses to the pretest items in the pool would be sparse. The three methods considered were the marginal maximum likelihood estimate with one EM cycle (OEM) method, the marginal maximum likelihood estimate with multiple EM cycles (MEM) method, and Stocking's Method B. The three methods were evaluated using simulations of data from computerized adaptive tests (CAT). The MEM method produced the smallest average total error in recovering the 240 pretest item characteristic curves. Stocking's Method B yielded the second smallest average total error in parameter estimation. In terms of scale maintenance, the MEM method and Stocking's Method B performed well in keeping with the scale of the pretest items on the same scale as that of the true parameters. With the OEM method, the scale of the pretest item parameter estimates deviated from that of the true parameters. (Contains 1 figure, 4 tables, and 14 references.) (Author/SLD)

The requirement of large sample sizes for calibrating items based on item response theory (IRT) models is not easily met in many practical pretesting situations. Although classical item statistics could be estimated with much smaller samples, the values may not be comparable across different groups of examinees. This study extended the authors' AERA 2000 by further exploring the standardization method and comparing its effectiveness with the one-parameter (1PL) and three-parameter (3PL) logist IRT models in adjusting pretest item statistics with small sample sizes, using more realistic data than the previous study. Based on the realistic data generated from a 50-dimensional multidimensional IRT model, the 3PL model performed better than the 1PL or standardization method in recovering both the population p-values and point biserial correlations. The standardization method outperformed the 1PL model in recovering the population point biserial, correlations but not in recovering the population p-values. The performance of the methods was also evaluated using the real pretest data of a high-stakes test. In terms of recovering the p-values and point biserial correlations for the real data, the 1PL model produced the most satisfactory results. The 3PL model performed worst in terms of recovering the p-values for the real data, and the standardization method performed worst in recovering the point biserial correlations for the real data. Due to the very limited number of conditions studied, one must be cautious about making conclusions about the standardization method relative to IRT methods based on these studies. The standardization method appears to be a viable alternative to IRT methods that may be simpler to implement, although these results do not suggest that it will produce more accurate results. (Contains 8 tables and 13 references.) (Author/SLD)

This paper discusses the evaluation and appropriate uses of concordances defined as linkages between scores on two tests that do not measure the same underlying construct. A conceptualization of score equivalence in terms of a latent variable model for test scores is presented. Two factors involved in evaluating the quality of a linkage of two scores are discussed: (1) the initial comparability of the scores; and (2) whether comparability is desired for scores of individual examinees or for score distributions. It is concluded that the only appropriate use of concordant scores of individuals is in situations where empirical evidence exists that the inferences made using concordant scores and the inferences made using actual scores are not too different. The use of concordant score distributions is likely to be less problematic than the use of concordant scores of individuals. Still, evidence should exist that a concordant score distribution does not differ greatly from the distribution of the actual scores in any population in which the concordance is used. Examples are presented to illustrate the evaluation and appropriate uses of concordances. (Contains 2 figures, 2 tables, and 14 references.) (Author/SLD)

The purpose of this study was to compare and evaluate five online pretest item calibration/scaling methods in computerized adaptive testing (CAT): (1) the marginal maximum likelihood estimate with one-EM cycle (OEM); (2) the marginal maximum likelihood estimate with multiple EM cycles (MEM); (3) Stocking's Method A (M. Stocking, 1988); (4) Stocking's Method B (M. Stocking, 1988); and (5) the BILOG/Prior method. The five methods were evaluated in terms of item parameter recovery under three different sample size conditions (300, 1,000, and 3,000). The MEM method appears to be the best choice among the methods used in this study because it produced the smallest parameter estimation errors for all sample size conditions. Stocking's Method B also worked very well, but it requires anchor items, which would make test lengths longer. The BILOG/Prior method did not seem to work with small sample sizes. Until more appropriate ways of handling the sparse data with BILOG are devised, the BILOG/Prior method may not be a reasonable choice. Because Stocking's Method A has the largest weighted total error, as well as a theoretical weakness (i.e., treating estimated ability as true ability), there appears to be little reason to use it. The MEM method should be preferred to the OEM method unless amount of time involved in iterative computation is a great concern. Otherwise, the OEM method and the MEM method are mathematically similar, and the OEM method produces larger errors than the MEM method. (Contains 2 tables, 3 figures, and 16 references.) (Author/SLD)

This paper describes procedures for estimating various indices of classification consistency and accuracy for multiple category classifications using data from a single test administration. The estimates of the classification consistency and accuracy indices are compared under three different psychometric models: the two-parameter beta binomial, four-parameter beta binomial, and three-parameter logistic item response theory (IRT) models. Using real data sets, the estimation procedures are illustrated, and the characteristics of the estimated indices are examined. This paper also examines the behavior of the estimated indices as a function of the latent variable. The IRT model tends to provide better fits to the data used in this study, and shows larger estimated consistency and accuracy. Although the results are not substantially different across different models, all three components of the models (i.e., the estimated true score distributions, fitted observed score distributions, and estimated conditional error variances) appear to have a great influence on estimates of the indices. Choosing a model in practice should be based on various considerations including the degree of the model fit to the data, suitability of the model assumptions, and the computational feasibility. (Contains 7 tables, 16 figures, and 37 references.) (Author/SLD)

This study presents and evaluates a method of standardization that may be used by test practitioners to standardize classical item statistics when sample sizes are small. The effectiveness of this standardization approach was compared through simulation with the one-parameter logistic (1PL) and three parameter logistic (3PL) models based on the criteria of the Pearson product-moment correlation, the mean square error, variance, and squared bias. In light of estimating the item difficulty values, the differences of the performance between the 3PL and standardization methods were small, but the differences between the 1PL and these two methods were large. For the estimation of point biserial correlations, the 3PL model seemed to perform better than the standardization method, and the standardization method performed better than the 1PL model. Although the standardization method did not outperform the 3PL model for the design considered in this study, it could be promising when smaller sample sizes are used. This method may be recommended for use in conjunction with the item response models for the test development when the pretesting sample sizes are small. By employing the classical measurement framework to obtain pretest item statistics, the problem of inaccurate item response theory parameter estimates when limited calibration sample sizes are available can be avoided. (Contains 4 tables and 16 references.) (Author/SLD)

Item response theory (IRT) item parameters can be estimated using data from a common item equating design either separately for each form or concurrently across forms. This paper reports the results of a simulation study of separate versus concurrent item parameter estimation. Using simulated data from a test with 60 dichotomous items, 4 factors were considered: (1) program (MULTILOG versus BILOG-MG); (2) sample size per form (3,000 versus 1,000); (3) number of common items (20 versus 10); and (4) equivalent versus nonequivalent groups taking the 2 forms (no mean difference versus a mean difference of 1 standard deviation). In addition, four methods of item parameter scaling were used in the separate estimation condition: two item characteristic curve methods (Stocking-Lord and Haebara), and two moment methods (Mean/Mean and Mean/Sigma). Although concurrent estimation resulted in less error than separate estimation more times than not, the results of this study, together with other research on this topic, are not sufficient to recommend completely avoiding separate estimation in favor of concurrent estimation. Two appendixes contain MUTILOG and BILOG-MG control files. (Contains 1 table, 11 figures, and 10 references.) (Author/SLD)

This paper focuses on methodological issues in applying equipercentile equating methods to pairs of tests that do not meet the assumptions of equating. This situation is referred to as a concordance situation, as opposed to an equating situation, and the end result is a concordance table that gives "comparable" scores between the tests. Interpretation of results is more ambiguous for a concordance situation than an equating situation, due to weaker assumptions, so that inappropriate use of results may be a problem. Examples are presented of specific issues that arose in applying equipercentile methods to create concordance tables between American College Test (ACT) composite and Scholastic Assessment Test (SAT) verbal plus mathematics scores for both a large group of 14 institutions and 2 states and individual institutions within that pooled group. Issues include data collection and cleaning, applying the equipercentile method, choosing smoothing methods, computing concordance standard errors, and adjusting for sample size. The examples illustrate that it cannot be assumed that procedures that are appropriate for an equating situation are also appropriate for a concordance situation. Each pair of tests to be linked must be considered a unique situation. (Contains 5 tables, 9 figures, and 16 references.) (SLD)

Equating a test form to itself through a chain of equatings, commonly referred to as circular equating, has been widely used as a criterion to evaluate the adequacy of equating. This paper uses both analytical methods and simulation methods to show that this criterion is in general invalid in serving this purpose. For the random groups design done in the same year, it is shown analytically that circular equating will always result in the identity function (i.e., the perfect result) even with the presence of random and systematic equating errors. For the random groups design done in the different years, a heuristic argument is provided that circular equating will generally deviate from the identity function by some random sampling error. A simulation study for this design also showed that the expected values of the circular equating may deviate slightly from the identity function but those deviations do not reflect the systematic error (bias) embedded in the equating. For the common-item nonequivalent groups design, a simulation study was done to show that circular equating again can not reflect the systematic error in equating. More effective ways of assessing random and systematic equating errors are recommended. (Contains 4 tables, 4 figures, and 13 references.) (Author)

This report presents an investigation by the American College Testing Program (ACT) of an alternative design for the National Assessment of Educational Progress (NAEP). The proposed design greatly simplifies the data collection and analysis procedures needed to produce assessment results and has the potential to produce results that are more timely and easier to interpret. The plan calls for developing individual NAEP forms, where each individual form represents, as closely as possible, the assessment questions from the domain of knowledge being measured by an NAEP construct. Sets of these forms could be administered, in random order, to students in the schools. This would replace the balanced incomplete block design (BIB) currently used. Assessments constructed under the BIB design do not closely represent, at least for the 1996 science assessment, the content framework. Enhanced procedures are also suggested for developing precise content and statistical specifications for individual forms and procedures for pretesting items. The basic scores that ACT suggests using for producing group assessment results are calculated by weighting item scores from multiple-choice and constructed-response items, where the weights are determined, a priori, by content specialists. These weights should relate more closely to the weighting intended by content specialists than do current NAEP weights. Scaling, equating, and score distribution estimation methods are described that rely on less stringent psychometric and statistical assumptions than do current procedures. Issues in sampling that include sample size requirements, sample design, and estimating standard errors are also examined, as are procedures for reporting score distributions that reflect group performance on content domains. The feasibility of using such domain scores to measure trends and to facilitate setting NAEP standards is explored. ACT recommends focusing on the design of assessments and the data collection methods rather than on complex analysis procedures. (Contains 18 tables, 9 figures, and 41 references.) (SLD)

Loglinear and logit models that have been suggested for studying differential item functioning (DIF) are reviewed, and loglinear formulations of the logit models are given. A polynomial loglinear model for assessing DIF is introduced that incorporates scores on the matching variable and item responses. The polynomial loglinear model contains far fewer parameters than loglinear models which treat the matching variable and item response as nominal. Compared to logit models that have been presented for investigating DIF, the polynomial loglinear model is easier to generalize to the case of more than two groups and more than two response categories, and can model more complex forms of DIF. The use of DIF methodology to investigate whether common items in the common-item equating design are functioning differently across test forms is discussed. Examples are given of using the polynomial loglinear model to investigate DIF for a test containing dichotomous and polytomous items and for investigating DIF for common items in a common-item nonequivalent groups equating design. (Contains 6 tables, 12 figures, and 20 references.) (Author/SLD)

The effectiveness of smoothing in reducing random errors in equipercentile equating of a short writing assessment with two raters, two prompts, with scores ranging from zero to five was examined. Thirteen methods were examined: no equating, three presmoothing, three postsmoothing, three combination presmoothing and postsmoothing, mean equating, linear equating, and unsmoothed equipercentile. The data for the study resulted from simulations of a writing assessment with one and two raters used for a large testing program. Mean equating appears to have less error with small samples than the other methods. A combination of presmoothing and postsmoothing appears to have less error using a small sample with two raters. For the larger sample size, presmoothing with degree three appears to have less error than the other methods. Equating can be problematic with performance assessments that have small score ranged, however, it can be done and reduces error relative to no equating. (Contains 2 tables, 4 figures, and 11 references.) (Author/SLD)

This paper presents a detailed description of maximum parameter estimation for item response models using the general EM algorithm. In this paper the models are specified using a univariate discrete latent ability variable. When the latent ability variable is discrete the distribution of the observed item responses is a finite mixture, and the EM algorithm for finite mixtures can be used. Maximum likelihood estimates of the item parameters and of the discrete probabilities of the latent ability distribution are given using the EM algorithm for finite mixtures. Results are presented in general for both dichotomous and polytomous item response models. The relation between the EM estimates and the Bock Aitken marginal maximum likelihood estimates is discussed. Estimates for the item parameters will depend on the specific form of the item response functions, and will usually require iterative numerical procedures. The EM algorithm is the same as the Bock-Aitken algorithm (R. D. Bock and M. Aitken, 1981) for marginal maximum likelihood estimation of the item parameters. (Contains 28 references.) (Author/SLD)

This paper discusses loglinear models for assessing differential item functioning (DIF). Loglinear and logit models that have been suggested for studying DIF are reviewed, and loglinear formulations of the logit models are given. A polynomial loglinear model for assessing DIF is introduced. Two examples using the polynomial loglinear model for investigating DIF are discussed. One example investigates DIF for a test consisting of both dichotomous and polytomous items. Another example illustrates the use of DIF techniques in investigating whether common items are functioning differently on two forms of a test in the common item nonequivalent groups equating design. (Contains 17 references, 6 tables, and 12 figures.) (Author)

Three methods of estimating test score distributions that may improve on using the observed frequencies (OBFs) as estimates of a population test score distribution are considered: the kernel method (KM); the polynomial method (PM); and the four-parameter beta binomial method (FPBBM). The assumption each method makes about the smoothness of the true distribution and computational details of the methods are described. The methods are compared in a simulation study in which 500 samples of size 500, 1,000, 2,000, and 5,000 are taken from three population distributions. For each of the 6,000 samples (3 population distributions by 4 sample sizes by 500 samples), the variable kernel, fixed kernel, FPBBM, and PM estimates of the true distribution are computed. The three population distributions are defined using observed raw score distributions on three tests for which a large number of examinees is available. Methods based on smoothness assumptions performed better than using the OBFs, and differences among the methods' performance were small compared to differences between the performance of the worst method and use of OBFs. The FPBBM performed best across all conditions, although the PM performed as well as the FPBBM for sample sizes of 5,000. The PM generally performed better than the KM except for one population for which the test score distribution was relatively flat. Four tables, 15 graphs, and a 13-item list of references are included. (Author/SLD)