Search

Item response theory (IRT) models are commonly used in educational and psychological testing to assess the (latent) ability of examinees and the effectiveness of the test items in measuring this underlying trait. The focus of this paper is on the assessment of item fit for unidimensional IRT models for dichotomous items using a Bayesian method. This paper will illustrate and compare the effectiveness of several discrepancy measures, used within the posterior predictive model check procedure, in detecting misfitted items. The effectiveness of the different discrepancy measures are illustrated in a simulation study using artificially altered simulated data. Using the best discrepancy measure among those studied, this method was applied to real data coming from a mathematics placement exam.

This study presents new findings on the utility of S - X2 as an item fit index for dichotomous item response theory models. Results are based on a simulation study in which item responses were generated and calibrated for 100 tests under each of 27 conditions. The item fit indices S - X2 and Q1 - X2 were calculated for each item. ROC curves were constructed based on the hit and false alarm rates of the two indices. Examination of these curves indicated that in general, the performance of S - X2 improved with test length and sample size. The performance of S - X2 was superior to that of Q1 - X2 under most but not all conditions. Results from this study imply that S - X2 may be a useful tool in detecting the misfit of one item contained in an otherwise well-fitted test, lending additional support to the utility of the index for use with dichotomous item response theory models. Index Terms: item response theory, S - X2, Q1 - X, model = data fit, item fit index.

Dodeen (2004) studied the correlation between the item parameters of the three-parameter logistic model and two item fit statistics, and found some linear relationships (e.g., a positive correlation between item discrimination parameters and item fit statistics) that have the potential for influencing the work of practitioners who employ item response theory. This paper examines the same type of linear relationships as studied in Dodeen. However, this paper adds to the literature by employing item fit statistics not considered in Dodeen, which have been recently suggested and whose Type I error rates have been demonstrated to be generally close to the nominal level. Detailed simulations show that if one uses certain of the recently suggested item fit statistics, there is no need to worry about any linear relationships between the item parameters and item fit statistics.

Two types of global testing procedures for item fit to the Rasch model were evaluated using simulation studies. The first type incorporates three tests based on first-order statistics: van den Wollenberg's Q(1) test, Glas's R(1) test, and Andersen's LR test. The second type incorporates three tests based on second-order statistics: van den Wollenberg's Q(2) test, Glas's R(2) test, and a non-parametric test proposed by Ponocny. The Type I error rates and the power against the violation of parallel item response curves, unidimensionality and local independence were analysed in relation to sample size and test length. In general, the outcomes indicate a satisfactory performance of all tests, except the Q(2) test which exhibits an inflated Type I error rate. Further, it was found that both types of tests have power against all three types of model violation. A possible explanation is the interdependencies among the assumptions underlying the model.

A new item-fit index is proposed that is both a descriptive measure of deviance of single items and an index for statistical inference. This index is based on the assumptions of the dichotomous and polytomous Rasch models for items with ordered categories and, in particular, is a standardization of the conditional likelihood of the item pattern that does not depend on the item parameters. This approach is compared with other methods for determining item fit. In contrast to many other item-fit indexes, this index is not based on response-score residuals. Results of a simulation study illustrating the performance of the index are provided. An asymptotically normally distributed Z statistic is derived and an empirical example demonstrates the sensitivity of the index with respect to item and person heterogeneity. Index terms: appropriateness measurement, item discrimination, item fit, partial credit model, Rasch model.

This study compares item reduction analysis based on classical test theory (maximizing Cronbach's alpha - approach A), with analysis based on the Rasch Partial Credit Model item-fit (approach B), as applied to children and adolescents' health-related quality of life (HRQoL) items. The reliability and structural, cross-cultural and known-group validity of the measures were examined.Within the European KIDSCREEN project, 3019 children and adolescents (8-18 years) from seven European countries answered 19 HRQoL items of the Physical Well-being dimension of a preliminary KIDSCREEN instrument. The Cronbach's alpha and corrected item total correlation (approach A) were compared with infit mean squares and the Q-index item-fit derived according to a partial credit model (approach B). Cross-cultural differential item functioning (DIF ordinal logistic regression approach), structural validity (confirmatory factor analysis and residual correlation) and relative validity (RV) for socio-demographic and health-related factors were calculated for approaches (A) and (B).Approach (A) led to the retention of 13 items, compared with 11 items with approach (B). The item overlap was 69% for (A) and 78% for (B). The correlation coefficient of the summated ratings was 0.93. The Cronbach's alpha was similar for both versions [0.86 (A); 0.85 (B)]. Both approaches selected some items that are not strictly unidimensional and items displaying DIF. RV ratios favoured (A) with regard to socio-demographic aspects. Approach (B) was superior in RV with regard to health-related aspects.Both types of item reduction analysis should be accompanied by additional analyses. Neither of the two approaches was universally superior with regard to cultural, structural and known-group validity. However, the results support the usability of the Rasch method for developing new HRQoL measures for children and adolescents.

This study reports the results of an empirical investigation of the distributional properties of the item fit statistics that are commonly used in Rasch model calibration programs as indices of the fit of the responses to individual items to the measurement model. There are two aspects of this study: an investigation of the distributional properties of the item fit statistics when the data fit the model, and the power of the item fit statistics to detect measurement disturbances. This study is based on simulated data to control for the presence of confounding factors, such as multidimensionality, differences in the slopes of item characteristic curves, and guessing. The results indicate that, when that data fit the model, the distributional properties of the item fit statistics, corrected for use of estimated person parameters, are close to hypothesized mean and standard deviation and that it is possible to construct reasonable Type I error rates that can be used as a frame of reference when investigating the fit of actual data to the Rasch model. The item fit statistics also have the power to detect measurement disturbances of reasonable magnitude.

Historically, rule-of-thumb critical values have been employed for interpreting fit statistics that depict anomalous person and item response patterns in applications of the Rasch model. Unfortunately, prior research has shown that these values are not appropriate in many contexts. This article introduces a bootstrap procedure for identifying reasonable critical values for Rasch fit statistics and compares the results of that procedure to applications of rule-of-thumb critical values for three example datasets. The results indicate that rule-of-thumb values may over- or under-identify the number of misfitting items or persons.

This study investigates the properties of the total and between fit statistics commonly used in Rasch model calibration and analysis programs to assess the fit of items to various Rasch models. There are two aspects of the study: an examination of the distributional properties of the item between fit statistics when the data fit the dichotomous model and a comparison of the power of the item between and total fit statistics to detect various types of simulated measurement disturbances in dichotomous data. The results indicate that the distributional properties of the between fit statistic are approximate unit normals and stable over a wide variety of testing conditions. In the presence of measurement disturbances, the between fit statistic appears to be more sensitive to systematic measurement disturbances and the total fit statistic is more sensitive to random measurement disturbances.

Computer-based testing (CBT), as computerized adaptive testing (CAT), is based on the availability of a large pool of calibrated test items. Usually, the calibration process consists of two stages.

The purpose of this research is twofold. First is to extend the work of Smith (1992, 1996) and Smith and Miao (1991, 1994) in comparing item fit statistics and principal component analysis as tools for assessing the unidimensionality requirement of Rasch models. Second is to demonstrate methods to explore how violations of the unidimensionality requirement influence person measurement. For the first study, rating scale data were simulated to represent varying degrees of multidimensionality and the proportion of items contributing to each component. The second study used responses to a 24 item Attention Deficit Hyperactivity Disorder scale obtained from 317 college undergraduates. The simulation study reveals both an iterative item fit approach and principal component analysis of standardized residuals are effective in detecting items simulated to contribute to multidimensionality. The methods presented in Study 2 demonstrate the potential impact of multidimensionality on norm and criterion-reference person measure interpretations. The results provide researchers with quantitative information to help assist with the qualitative judgment as to whether the impact of multidimensionality is severe enough to warrant removing items from the analysis.

This study examined five Rasch-model-based item-fit indices: unweighted and weighted standardized indices (denoted UWz and Wz), standardized likelihood index (denoted Lz), and Extended Caution Indices (denoted ECI2z and ECI4z), in terms of their distributional properties and the power of detecting item bias or Differential Item Functioning (DIF). The results indicated that although these five standardized item-fit indices did not depart significantly from a normal distribution, it appeared that the Type I error rates were not reasonable. For the power of five standardized item-fit indices to detect DIF, the results showed that all indices did perform poorly across various conditions. These findings lead to the conclusion that all indices used in this study are inadequate fit measures for detecting DIF.

A Monte Carlo study was conducted using simulated dichotomous data to determine the effects of guessing on Rasch item fit statistics (weighted total, unweighted total, and unweighted between fit statistics) and the Logit Residual Index (LRI). The data were simulated using 100 items, 100 persons, three levels of guessing (0%, 25%, and 50%), and two item difficulty distributions (normal and uniform). The results of the study indicated that no significant differences were found between the mean Rasch item fit statistics for each distribution type as the probability of guessing the correct answer increased. The mean item scores differed significantly with uniformly distributed item difficulties, but not normally distributed item difficulties. The LRI was more sensitive to large positive item misfit values associated with the unweighted total fit statistic than to similar values associated with the weighted total fit or unweighted between fit statistics. The greatest magnitude of change in LRI values (negative) was observed when the unweighted total fit statistic had large positive values greater than 2.4. The LRI statistic was most useful in identifying the linear trend in the residuals for each item, thereby indicating differences in ability groups, i.e. differential item functioning.

This study investigated the behavior of several per son and item fit statistics commonly used to test and obtain fit to the one-parameter item response model. Using simulated data for 500 persons and 15 items, the sensitivity of the total-t, mean-square residual, and between-t fit statistics to guessing, heterogeneity in discrimination parameters, and multidimensionality was examined. Additionally, 25 misfitting persons and a misfitting item were generated to test the power of the three fit statistics to detect deviations in a subset of observations. Neither the total-t nor the mean-square residual were able to detect deviation from any of the models fitted. Use of these statistics appears to be un warranted. The between-t was a useful indicator of guessing and heterogeneity in discrimination parame ters, but was unable to detect multidimensionality. These results show that use of person and item fit statistics to test and obtain overall fit to the one- parameter model can lead to acceptance of the model even when it is grossly inappropriate. Assessments of model fit based on this strategy are inadequate. Alter native methods must be sought.

Detecting item fit for common dichotomous item response theory (IRT) models has always been an issue of enormous interest, but there exists no unanimously agreed upon item fit diagnostic for the models. This paper employs the posterior predictive model checking method (Guttman, 1967; Rubin, 1981, 1984), a popular Bayesian model checking tool, to examine item fit for common dichotomous item response theory models. An item fit plot, comparing the observed and predicted proportion correct scores of examinee groups (with groups based on raw scores), promises to be useful in real applications. This paper also suggests how to obtain posterior predictive p-values for the χ2-type test statistics of Orlando and Thissen (2000) comparing the observed and predicted proportion correct scores for different raw score groups. A number of simulation studies and real data applications examine the effectiveness of the suggested item fit diagnostics. The suggested p-values seem to have low Type I error rate, low false alarm rate, and adequate power, indicating that researchers might find the methods more acceptable than the existing item fit measures.