Search

A study was conducted to implement the use of a standardized effect size and corresponding classification guidelines for polytomous data with the POLYSIBTEST procedure and compare those guidelines with prior recommendations. Two simulation studies were included. The first identifies new unstandardized test heuristics for classifying moderate and large differential item functioning (DIF) for polytomous response data with three to seven response options. These are provided for researchers studying polytomous data using POLYSIBTEST software that has been published previously. The second simulation study provides one pair of standardized effect size heuristics that can be employed with items having any number of response options and compares true-positive and false-positive rates for the standardized effect size proposed by Weese with one proposed by Zwick et al. and two unstandardized classification procedures (Gierl; Golia). All four procedures retained false-positive rates generally below the level of significance at both moderate and large DIF levels. However, Weese's standardized effect size was not affected by sample size and provided slightly higher true-positive rates than the Zwick et al. and Golia's recommendations, while flagging substantially fewer items that might be characterized as having negligible DIF when compared with Gierl's suggested criterion. The proposed effect size allows for easier use and interpretation by practitioners as it can be applied to items with any number of response options and is interpreted as a difference in standard deviation units.

The property of item parameter invariance in item response theory (IRT) plays a pivotal role in the applications of IRT such as test equating. The scope of parameter invariance when using estimates from finite biased samples in the applications of IRT does not appear to be clearly documented in the IRT literature. This article provides information on the extent to which item parameter invariance is observed in samples with the Rasch and 2-parameter model calibrations through simulations, where the behaviors of item parameter estimates were examined under 12 different types of convenient sampling scenarios. The results indicated that the property of item invariance in IRT for dichotomously scored data could hold for the sample item parameter estimates, regardless of biased samples, when the model holds in the data, the number of items in a test is not small, and the sample size is large.

The demand for diagnostic feedback has triggered extensive research on cognitive diagnostic models (CDMs), such as the deterministic input, noisy output "and" gate (DINA) model. This study explored two Q-matrix specifications with the DINA model in a statewide large-scale mathematics assessment. The first Q-matrix was developed based on five predefined content reporting categories, and the second was based on the post hoc coding of 15 attributes by test-development experts. Total raw scores correlated strongly with the number of skills mastered, using both Q-matrices. Correlations between the DINA-model item statistics and those from the item response theory analyses were moderate to strong, but were always lower for the 15-skill model. Results highlighted the trade-off between finer-grained modeling and less precise model estimation.

Current methods that simultaneously model differential testlet functioning (DTLF) and differential item functioning (DIF) constrain the variances of latent ability and testlet effects to be equal between the focal and the reference groups. Such a constraint can be stringent and unrealistic with real data. In this study, we propose a multigroup one-parameter testlet model (M1PLTM) that relaxes this equal variance assumption and also allows simultaneous modeling of DTLF and DIF. M1PLTM is a more generalized version of the extant models. The results of a simulation study show that parameters of M1PLTM could be accurately recovered via a Markov chain Monte Carlo (MCMC) method; when data were generated based on M1PLTM, falsely assuming equal variances of latent ability and testlet effects resulted in inferior parameter recovery.

The property of item parameter invariance in item response theory (IRT) plays a pivotal role in the applications of IRT such as test equating. The scope of parameter invariance when using estimates from finite biased samples in the applications of IRT does not appear to be clearly documented in the IRT literature. This article provides information on the extent to which item parameter invariance is observed in samples with the Rasch and 2-parameter model calibrations through simulations, where the behaviors of item parameter estimates were examined under 12 different types of convenient sampling scenarios. The results indicated that the property of item invariance in IRT for dichotomously scored data could hold for the sample item parameter estimates, regardless of biased samples, when the model holds in the data, the number of items in a test is not small, and the sample size is large. [ABSTRACT FROM AUTHOR]