Search

Test developers and psychometricians have historically examined measurement bias and differential item functioning (DIF) across a single categorical variable (e.g., gender), independently of other variables (e.g., race, age, etc.). This is problematic when more complex forms of measurement bias may adversely affect test responses and, ultimately, bias test scores. Complex forms of measurement bias include conditional effects, interactions, and mediation of background information on test responses. I propose a multidimensional, person-specific perspective of measurement bias to explain how complex sources of bias can manifest in the assessment of human knowledge, skills, and abilities. I also describe a data-driven approach for identifying key sources of bias among many possibilities--namely, a machine learning method commonly known as regularization.

Testing for differential item functioning (DIF) has undergone rapid statistical developments recently. Moderated nonlinear factor analysis (MNLFA) allows for simultaneous testing of DIF among multiple categorical and continuous covariates (e.g., sex, age, ethnicity, etc.), and regularization has shown promising results for identifying DIF among many covariates. However, computationally inefficient estimation methods have hampered practical use of the regularized MNFLA method. We develop a penalized expectation–maximization (EM) algorithm with soft- and firm-thresholding to more efficiently estimate regularized MNLFA parameters. Simulation and empirical results show that, compared to previous implementations of regularized MNFLA, the penalized EM algorithm is faster, more flexible, and more statistically principled. This method also yields similar recovery of DIF relative to previous implementations, suggesting that regularized DIF detection remains a preferred approach over traditional methods of identifying DIF. [ABSTRACT FROM AUTHOR]

Test-retest reliability is often estimated using naturally occurring data from test repeaters. In settings such as admissions testing, test takers choose if and when to retake an assessment. This self-selection can bias estimates of test-retest reliability because individuals who choose to retest are typically unrepresentative of the broader testing population and because differences among test takers in learning or practice effects may increase with time between test administrations. We develop a set of methods for estimating test-retest reliability from observational data that can mitigate these sources of bias, which include sample weighting, polynomial regression, and Bayesian model averaging. We demonstrate the value of using these methods for reducing bias and improving precision of estimated reliability using empirical and simulated data, both of which are based on more than 40,000 repeaters of a high-stakes English language proficiency test. Finally, these methods generalize to settings in which only a single, error-prone measurement is taken repeatedly over time and where self-selection and/or changes to the underlying construct may be at play. [ABSTRACT FROM AUTHOR]

Differential item functioning (DIF) is a pernicious statistical issue that can mask true group differences on a target latent construct. A considerable amount of research has focused on evaluating methods for testing DIF, such as using likelihood ratio tests in item response theory (IRT). Most of this research has focused on the asymptotic properties of DIF testing, in part because many latent variable methods require large samples to obtain stable parameter estimates. Much less research has evaluated these methods in small sample sizes despite the fact that many social and behavioral scientists frequently encounter small samples in practice. In this article, we examine the extent to which model complexity—the number of model parameters estimated simultaneously—affects the recovery of DIF in small samples. We compare three models that vary in complexity: logistic regression with sum scores, the 1-parameter logistic IRT model, and the 2-parameter logistic IRT model. We expected that logistic regression with sum scores and the 1-parameter logistic IRT model would more accurately estimate DIF because these models yielded more stable estimates despite being misspecified. Indeed, a simulation study and empirical example of adolescent substance use show that, even when data are generated from / assumed to be a 2-parameter logistic IRT, using parsimonious models in small samples leads to more powerful tests of DIF while adequately controlling for Type I error. We also provide evidence for minimum sample sizes needed to detect DIF, and we evaluate whether applying corrections for multiple testing is advisable. Finally, we provide recommendations for applied researchers who conduct DIF analyses in small samples. [ABSTRACT FROM AUTHOR]

Determining whether measures are equally valid for all individuals is a core component of psychometric analysis. Traditionally, the evaluation of measurement invariance (MI) involves comparing independent groups defined by a single categorical covariate (e.g., men and women) to determine if there are any items that display differential item functioning (DIF). More recently, Moderated Nonlinear Factor Analysis (MNLFA) has been advanced as an approach for evaluating MI/DIF simultaneously over multiple background variables, categorical and continuous. Unfortunately, conventional procedures for detecting DIF do not scale well to the more complex MNLFA. The current manuscript therefore proposes a regularization approach to MNLFA estimation that penalizes the likelihood for DIF parameters (i.e., rewarding sparse DIF). This procedure avoids the pitfalls of sequential inference tests, is automated for end users, and is shown to perform well in both a small-scale simulation and an empirical validation study. [ABSTRACT FROM AUTHOR]

The Eating Disorder Inventory-Drive for Thinness (EDI-DT) subscale is commonly used in research and as an eating disorder screening measure, but extant evidence is scant regarding its psychometric properties. University and community samples often are treated as interchangeable in terms of research conclusions. Given established demographic differences between these two populations, the present study tested measurement invariance of the EDI-DT across these two sample types. Two large samples of university students (n = 537; 50% female, 67% White; n = 584; 52% female, 67% White) and community participants (n = 535; 57% female, 81% White; n = 533; 63% female, 82% White) completed the EDI-DT online. Multiple group confirmatory factor analyses tested configural, metric, scalar, and strict invariance by sample type. The EDI-DT subscale was not invariant across university and community samples. Post-hoc-regularized multiple nonlinear factor analyses suggested potential item bias associated with sample type, age, and body mass index on six of the seven items. Item bias, however, appeared to be associated with minimal clinical impact. Collectively, results suggest that the EDI-DT may be functionally invariant and appropriate for use with broad populations. Public Significance Statement: The Eating Disorder Inventory-Drive for Thinness (EDI-DT) subscale is commonly used in research and clinical work. Though how individuals interpret the items on the EDI-DT differs by age and body mass index (BMI), this did not appear to impact clinical conclusions. Researchers and clinicians can use the EDI-DT with populations of diverse age, BMI, and sex. [ABSTRACT FROM AUTHOR]

Alcohol outcomes expectancies (AOEs) are robust predictors of alcohol initiation and escalation of drinking behavior among adolescents. Although measurement invariance is a prerequisite for inferring valid comparisons of AOEs across groups (e.g., age), empirical evidence is lacking. In a secondary data analysis study, we employed regularized moderated nonlinear factor analysis (MNLFA) to simultaneously test differential item functioning (DIF) across age, sex, race, ethnicity, socioeconomic status (SES), and alcohol initiation for a 22-item, two-factor measure of positive and negative AOEs among adolescents (analytic n = 936 drawn from a parent study of 1023 adolescents). Evidence of DIF was minimal, with no DIF for the negative AOE factor and DIF for only two items of the positive AOE factor. The item "feel grown up" exhibited DIF by age, and the item "feel romantic" exhibited DIF by SES. After accounting for DIF, the positive AOE latent factor mean differed by SES, age, and alcohol initiation, and exhibited lower variability by alcohol initiation. The negative AOE latent factor mean differed by sex and SES, with greater variability by SES and age and lower variability by alcohol initiation. Group-differences findings for age and alcohol initiation are consistent with prior work, and differences by sex and SES are a new contribution to the literature that should prompt additional research to ensure replicability. The present study demonstrates the utility of the MNLFA technique for examining comprehensive measurement invariance, particularly for applied researchers who seek to examine substantive research questions while accounting for any DIF present in the scales used., (Copyright © 2021. Published by Elsevier Ltd.)

It is common in addictions research for statistical analyses to include interaction effects to test moderation hypotheses. Far less commonly do researchers consider the possibility that a given predictor may exert a nonlinear effect on the outcome. This lack of attention to the possible nonlinear effects of individual predictors is problematic because it may result in identification of entirely spurious interactions with other, correlated predictors. Given the commonplace practice of testing interactions, and the rarity of testing nonlinear effects, we speculate that some of the significant interactions reported in the literature may actually be spurious, reflecting only the misspecification of nonlinear effects. We outline the mathematical reasons for this problem using the relatively simple case of a quadratic regression model. Within this context, prior research by Busemeyer and Jones (1983) clearly demonstrated that quadratic effects of individual predictors can masquerade as interaction effects between correlated predictors. Furthermore, the explosive growth of mediation, moderation, and moderated mediation analyses in behavioral research makes this issue especially relevant for researchers of addiction. In this article, we (1) call further attention to the potential problems of omitting nonlinear effects in linear regression, (2) extend these findings to the more complex moderated mediation model, and (3) provide practical recommendations for applied researchers for differentiating nonlinear from interactive effects., (Copyright © 2018 Elsevier Ltd. All rights reserved.)