Search

Evaluating scale comparability in international large-scale assessments depends on measurement invariance (MI). The root mean square deviation (RMSD) is a standard method for establishing MI in several programs, such as the Programme for International Student Assessment and the Programme for the International Assessment of Adult Competencies. Previous research showed that the RMSD was unable to detect departures from MI when the latent trait distribution was far from item difficulty. In this study, we developed three alternative approaches to the original RMSD: equal, item information, and b-norm weighted RMSDs. Specifically, we considered the item-centered normalized weight distributions to compute the item characteristic curve difference in the RMSD procedure more efficiently. We further compared all methods' performance via a simulation study and the item information and b-norm weighted RMSDs showed the most promising results. An empirical example is demonstrated, and implications for researchers are discussed.

The generalized S-X[superscript 2]-test is a test of item fit for items with polytomous responses format. The test is based on a comparison of the observed and expected number of responses in strata defined by the test score. In this article, we make four contributions. We demonstrate that the performance of the generalized S-X[superscript 2]-test depends on how sparse cells are pooled. We propose alternative implementations of the test within the framework of limited information testing. We derive the distribution of the S-X[superscript 2]-residuals that can be used for post hoc analyses. We suggest a diagnostic plot that visualizes the form of the misfit. The performance of the alternative implementations is investigated in a simulation study. The simulation study suggests that the alternative implementations are capable of controlling the Type-I error rate well and have high power. An empirical application concludes this article.

Orlando and Thissen (2000) introduced the "S - X[superscript 2]" item-fit index for testing goodness-of-fit with dichotomous item response theory (IRT) models. This study considers and evaluates an alternative approach for computing "S - X[superscript 2]" values and other factors associated with collapsing tables of observed and expected numbers (OE tables), which can affect flagging items. Results suggest that collapsing OE tables requires careful consideration of a trade-off between power and empirical type I error rate. Concurrent collapsing of score categories would be preferred over separate collapsing for its procedural simplicity, minimal effect of choice of a minimum cell value on empirical type I error rates, and reasonable type I error rates even for the most sparse condition in the study. For separate collapsing, a smaller minimum cell value is recommended as OE tables possess more sparseness (e.g., longer test lengths and smaller sample sizes) if inflated type I error rates are more of a concern in detecting items for misfit based on the "S - X[superscript 2]" index. If it is more important to identify misfit items, the study results recommend using a larger minimum cell value for collapsing.

Item-level fit analysis not only serves as a complementary check to global fit analysis, it is also essential in scale development because the fit results will guide item revision and/or deletion (Liu & Maydeu-Olivares, 2014). During data collection, missing response data may likely happen due to various reasons. Chi-square-based item fit indices (e.g., Yen's Q[subscript 1] , McKinley and Mill's G[superscript 2], Orlando and Thissen's S-X[superscript 2] and S-G[superscript 2]) are the most widely used statistics to assess item-level fit. However, the role of total scores with complete data used in S-X[superscript 2] and S-G[superscript 2] is different from that with incomplete data. As a result, S-X[superscript 2] and S-G[superscript 2] cannot handle incomplete data directly. To this end, we propose several modified versions of S-X[superscript 2] and S-G[superscript 2] to evaluate item-level fit when response data are incomplete, named as M[subscript impute]-X[superscript 2] and M[subscript impute]-G[superscript 2]. Instead of using observed total scores for grouping, the new indices rely on imputed total scores by either a single imputation method or three multiple imputation methods (i.e., two-way with normally distributed errors, corrected item-mean substitution with normally distributed errors and response function imputation). The new indices are equivalent to S-X[superscript 2] and S-G[superscript 2] when response data are complete. Their performances are evaluated and compared via simulation studies, the manipulated factors include test length, sources of misfit, misfit proportion and missing proportion. The results from simulation studies are consistent with those of Orlando and Thissen (2000, 2003), and different indices are recommended under different conditions.

In Programme for International Student Assessment (PISA), item response theory (IRT) scaling is used to examine the psychometric properties of items and scales and to provide comparable test scores across participating countries and over time. To balance the comparability of IRT item parameter estimations across countries with the best possible model fit, a partial invariance approach is used in PISA. In this approach, international or common item parameters are estimated for the majority of items, while unique or country-specific item parameters are allowed for item-country combinations where a misfit to the common parameters can be identified. The goal of the current study is to establish item fit statistic thresholds for identifying such misfits. We investigated the impact of various thresholds on scale and score estimation. To evaluate the impact of various item fit thresholds, we systematically examined the number of unique item parameters and country performance distributions and compared the overall model fit statistics using data from PISA 2015 and 2018. Results showed that RMSD = 0.10 provides the best fitting model while still establishing stable parameter estimations and sufficient comparability across groups. The applications and implications of the results are discussed.

Testing whether items fit the assumptions of an item response theory model is an important step in evaluating a test. In the literature, numerous item fit statistics exist, many of which show severe limitations. The current study investigates the root mean squared deviation (RMSD) item fit statistic, which is used for evaluating item fit in various large-scale assessment studies. The three research questions of this study are (1) whether the empirical RMSD is an unbiased estimator of the population RMSD; (2) if this is not the case, whether this bias can be corrected; and (3) whether the test statistic provides an adequate significance test to detect misfitting items. Using simulation studies, it was found that the empirical RMSD is not an unbiased estimator of the population RMSD, and nonparametric bootstrapping falls short of entirely eliminating this bias. Using parametric bootstrapping, however, the RMSD can be used as a test statistic that outperforms the other approaches--infit and outfit, S -- X[superscript 2]--with respect to both Type I error rate and power. The empirical application showed that parametric bootstrapping of the RMSD results in rather conservative item fit decisions, which suggests more lenient cut-off criteria.

We used simulation techniques to assess the item-level and familywise Type I error control and power of an IRT item-fit statistic, the "S-X"[superscript 2]. Previous research indicated that the "S-X"[superscript 2] has good Type I error control and decent power, but no previous research examined familywise Type I error control. We varied percentage of misfitting items, sample size, and test length, and computed familywise Type I error with no correction, a Bonferroni correction, and a Benjamini-Hochberg correction. The "S-X"[superscript 2] controlled item-level and familywise Type I errors when corrections were applied to conditions with no misfitting items. In the presence of misfitting items, the "S-X"[superscript 2] exhibited inflated item-level and familywise false hit rates in many conditions, even with familywise Type I error corrections. Lastly, power was low and negatively impacted when either of the familywise Type I error corrections was applied. We suggest using the "S-X"[superscript 2] with no familywise Type I error control in conjunction with other methods of assessing item fit (e.g., visual analysis).

Dodeen (2004) studied the correlation between the item parameters of the three-parameter logistic model and two item fit statistics, and found some linear relationships (e.g., a positive correlation between item discrimination parameters and item fit statistics) that have the potential for influencing the work of practitioners who employ item response theory. This paper examines the same type of linear relationships as studied in Dodeen. However, this paper adds to the literature by employing item fit statistics not considered in Dodeen, which have been recently suggested and whose Type I error rates have been demonstrated to be generally close to the nominal level. Detailed simulations show that if one uses certain of the recently suggested item fit statistics, there is no need to worry about any linear relationships between the item parameters and item fit statistics.

This paper presents a non-parametric imputation technique, named random forest, from the machine learning field. The random forest procedure has two main tuning parameters: the number of trees grown in the prediction and the number of predictors used. Fifty experimental conditions were created in the imputation procedure, with different combinations of predictors (from 1 to 10) and number of trees (10, 50, 100, 500 and 1000). We examined how each experimental condition affected the items fit and the difficulty of an inductive reasoning test to the dichotomous Rasch model. The results point that using random forest to impute missing values is a reliable technique to be used in psychological researches, since it led to statistically significant differences in the infit's median only in 4% of the experimental conditions investigated, compared to the original data set result. However, researchers should be aware that in 32% of the experimental conditions, the imputation procedure significantly increased the median of the estimated items' difficulty, compared to the original data set.

Drawing valid inferences from item response theory (IRT) models is contingent upon a good fit of the data to the model. Violations of model-data fit have numerous consequences, limiting the usefulness and applicability of the model. This instructional module provides an overview of methods used for evaluating the fit of IRT models. Upon completing this module, the reader will have an understanding of traditional and Bayesian approaches for evaluating model-data fit of IRT models, the relative advantages of each approach, and the software available to implement each method.

Empirical information regarding performance of model-fit procedures has been a persistent need in measurement practice. Statistical procedures for evaluating item fit were applied to real test examples that consist of both dichotomously and polytomously scored items. The item fit statistics used in this study included the PARSCALE's G[squared], Orlando and Thissen's (2000) S - [chi][squared] and S - G[squared], and Stone's (2000) [chi][squared*] and G[squared*]. The results of this study indicated that the fit of an individual item was affected by the choice of model-fit analyses. The performance of fit indices appeared to vary depending on item response theory (IRT) model mixtures used for calibration, sample size, and test length. In terms of consistency among the fit indices, the statistics based on the same approach (e.g., S - [chi][squared] and S - G[squared] ) showed considerably higher agreement in detecting misfitting items than the statistics based on different approaches (e.g., S - [chi][squared] and [chi][squared*]). Consistent and inconsistent findings compared to previous research are discussed along with practical implications. (Contains 3 tables and 1 figure.)

A novel approach to item-fit analysis based on an asymptotic test is proposed. The new test statistic, χ w 2 , compares pseudo-observed and expected item mean scores over a set of ability bins. The item mean scores are computed as weighted means with weights based on test-takers' a posteriori density of ability within the bin. This article explores the properties of χ w 2 in case of dichotomously scored items for unidimensional IRT models. Monte Carlo experiments were conducted to analyze the performance of χ w 2 . Type I error of χ w 2   was acceptably close to the nominal level and it had greater power than Orlando and Thissen's S - x 2 . Under some conditions, power of χ w 2 also exceeded the one reported for the computationally more demanding Stone's χ 2 ∗ ., Competing Interests: Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article., (© The Author(s) 2022.)

The S- X statistic (Orlando & Thissen, 2000) is popular among researchers and practitioners who are interested in the assessment of item fit. However, the statistic suffers from the Chernoff-Lehmann problem (Chernoff & Lehmann, 1954) and hence does not have a known asymptotic null distribution. This paper suggests a modified version of the S- 2 statistic (Orlando & Thissen, 2000) is popular among researchers and practitioners who are interested in the assessment of item fit. However, the statistic suffers from the Chernoff-Lehmann problem (Chernoff & Lehmann, 1954) and hence does not have a known asymptotic null distribution. This paper suggests a modified version of the S- X 2 statistic that is based on the modified Rao-Robson χ 2 statistic (Rao & Robson, 1974). A simulation study and a real data analyses demonstrate that the use of the modified statistic instead of the S- X 2 statistic would lead to fewer items being flagged for misfit., Competing Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article., (© The Author(s) 2022.)

This research examines the utility of the s-x[superscript 2] statistic proposed by Orlando and Thissen (2000) in evaluating item fit for multidimensional item response models. Monte Carlo simulation was conducted to investigate both the Type I error and statistical power of this fit statistic in analyzing two kinds of multidimensional test structures: approximate simple structure and complex structure. Overall, results show that this statistic is capable of evaluating item fit in the application of multidimensional item response models. It is important to identify the structure of multidimensional tests before this fit statistic is applied. For tests with an approximate simple structure, the sampling distribution can be approximated by a standard chi-square distribution. But for tests with a complex structure, that approximation is more complicated. As regards power in detecting the model nonfitting items, the performance of this statistic in multidimensional tests is comparable to that in unidimensional tests. (Contains 1 figure and 6 tables.)

Orlando and Thissen (2000) developed an item fit statistic for binary item response theory (IRT) models known as S-X[superscript 2]. This article generalizes their statistic to polytomous unfolding models. Four alternative formulations of S-X[superscript 2] are developed for the generalized graded unfolding model (GGUM). The GGUM is a unidimensional IRT model for unfolding polytomous responses. It yields single-peaked, non-monotonic item characteristic curves that predict a higher item score to the extent that an individual is located close to an item on the underlying latent continuum. A simulation was performed to examine the characteristics of these new item fit indices under the GGUM, as well as a traditional likelihood ratio x[superscript 2] test (G[superscript 2]). All variants of S-X[superscript 2] exhibited reasonable Type I error rates, but that for G[superscript 2] was more erratic. The new indices exhibited variable power to detect misfit. Two new item fit tests are recommended for use based on simulation results. (Contains 1 figure and 4 tables.)

Orlando and Thissen's S-X[superscript 2] item fit index has performed better than traditional item fit statistics such as Yen' s Q[subscript 1] and McKinley and Mill' s G[superscript 2] for dichotomous item response theory (IRT) models. This study extends the utility of S-X[superscript 2] to polytomous IRT models, including the generalized partial credit model, partial credit model, and rating scale model. The performance of the generalized S-X[superscript 2] in assessing item model fit was studied in terms of empirical Type I error rates and power and compared to G[superscript 2]. The results suggest that the generalized S-X[superscript 2] is promising for polytomous items in educational and psychological testing programs. (Contains 2 figures, 3 tables, and 52 formulas.)

Examined five Rasch-model-based item-fit indices in terms of their distributional properties and the power of detecting item bias or differential item functioning. Results indicate that, although these five standardized item-fit indices did not depart significantly from a normal distribution, the Type I error rates were not reasonable. (Author/SLD)

Extended work in comparing item fit statistics and principal component analysis as tools for assessing the unidimensionality requirements of Rasch models and demonstrated methods to explore how violations of the unidimensionality requirement influence person measurement through 2 studies involving simulated rating scale data and responses to a scale by 317 undergraduates. Results can help researchers determine the impact of multidimensionality. (SLD)

A Monte Carlo study was conducted using simulated dichotomous data to determine the effects of guessing on Rasch item fit statistics and the Logit Residual Index. Results indicate that no significant differences were found between the mean Rasch item fit statistics for each distribution type as the probability of guessing the correct answer increased. (SLD)

Studied the Type I error rate and the power of a number of new and existing tests of fit to the three-parameter logistic model. Simulation study results show that all the tests except one have a Type I error rate acceptably close to the nominal significant level and good power to detect the model violations at which they are targeted. (SLD)

Results of a simulation study imply that S X squared may be a useful tool in detecting the misfit of one item contained in an otherwise well-fitted test, lending additional support to the utility of the index for use with dichotomous item response theory models. (SLD)

Introduces new goodness-of-fit indices for dichotomous item response theory (IRT) models. Indices are based on the likelihoods of number-correct scores derived from the IRT model, and they provide a direct comparison of the modeled and observed frequencies for correct and incorrect responses for each number-correct score. Assesses these indices through simulation. (Author/SLD)