Development of an Effect Size to Classify the Magnitude of DIF in Dichotomous and Polytomous Items

A standardized effect size for the SIBTEST/POLYSIBTEST procedure is proposed, allowing for Differential Item Functioning (DIF) to be classified with a single set of DIF heuristics regardless of whether data are dichotomous or polytomous. This proposed standardized effect size accounts for both variability in responses and whether participants are included in the SIBTEST/POLYSIBTEST calculations. First, a new set of unstandardized effect size heuristics are established for dichotomous data that are more aligned with Educational Testing Service (ETS) standards using two and three parameter logistic (2PL and 3PL) models. Second, a standardized effect size is proposed and compared to other DIF heuristics with SIBTEST in a variety of data conditions, with a set of standardized heuristics proposed for dichotomous data. Using the heuristics for the standardized effect size, the application is extended to polytomous data with comparisons made to other proposed DIF heuristics with POLYSIBTEST when there are three to seven item response options. The proposed standardized effect size has practical importance due to there not being an agreed upon way of classifying the magnitude of DIF present in polytomous items using POLYSIBTEST. The standardized effect size also provides a single set of DIF heuristics for classifying DIF in 2PL, 3PL, and graded response models; therefore, making it easier when researchers and practitioners are assessing items for DIF using a variety of scale formats.