Search

Survey score interpretations are often plagued by sources of construct-irrelevant variation, such as response styles. In this study, we propose the use of an IRTree Model to account for response styles by making use of self-report items and anchoring vignettes. Specifically, we investigate how the IRTree approach with anchoring vignettes compares to traditional approaches that either do not include anchoring vignettes or do not account for response styles. We analyze secondary data using four different models: 1) total score; 2) graded response model; 3) IRTree without the consideration of anchoring vignettes, and 4) IRTree considering anchoring vignettes. We found significant differences in trait estimates from models that account for response styles compared to those that do not. Additionally, we found differences in trait estimates between the IRTree Models when considering anchoring vignettes and when not. Model comparisons suggest that trait differences are due to adjusting for acquiescence response style.

Response styles introduce construct-irrelevant variance as a result of respondents systematically responding to Likert-type items regardless of content. Methods to account for response styles through data analysis as well as approaches to mitigating the effects of response styles during data collection have been well-documented. Recent approaches to modeling Likert responses, such as the IRTree model, rely on the response process individuals take when answering item responses. In this study, we advocate for the use of IRTrees to analyze Likert items in addition to using the hypothesized response process to design new items. Combining these two approaches facilitates answering Likert item design questions that have plagued researchers. These include the interpretation of a middle response option, the optimal number of response options, and how to label the response options. We present 7 research questions that could be answered using this new approach, outline methods of data collection and analysis for each, and present results from an empirical example to address one of these seven questions.

In this study, we examined the results and interpretations produced from two different IRTree models--one using paths consisting of only dichotomous decisions, and one using paths consisting of both dichotomous and polytomous decisions. We used data from two versions of an impulsivity measure. In the first version, all the response options had labels; in the second version, only the endpoints were labeled. Based on past research, we hypothesized that the endpoints would be selected more frequently in the endpoint-only labeled condition, and the midpoint response option would be selected more frequently in the fully labeled condition. Results from the two models (dichotomous and polytomous) were similar and indicated that our hypotheses were partially supported--specifically, there was no consistent pattern in terms of which condition saw a higher frequency of midpoint response selection. However, our hypotheses regarding extreme responding in the endpoint-only labeling condition were supported.

We conducted a comprehensive survey of the educational measurement profession to assist recruiting to the measurement profession in the United States. Specifically, in April and May 2019, we surveyed measurement professionals and graduate students by targeting six areas to assist recruitment efforts: demographics of the field, compensation for professionals, graduate student funding, where to work, what work life entails, and mentoring opportunities. We evidence a profession trending toward being more diverse, professionals having a median annual salary between $110,000 and $130,000 (U.S.) dollars, varying graduate student funding opportunities, and flexibility in work location and tasks performed. For each of the six target areas, we present detailed results from the survey as well as implications for practice. We conclude the manuscript by discussing a change needed in the field that would assist in recruitment efforts.

Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing remediation has high-impact consequences for students. Unfortunately, tests that classify individuals are subjected to false positive and false negative misclassifications. When determining a standard, standard setting panelists implicitly consider the negative consequences of the decisions made from test use. We propose the conscious weight method and subconscious weight method to bring more objectivity to the standard setting process. To do this, these methods quantify the relative harm of the negative consequences of false positive and false negative misclassification.

Data simulation and Monte Carlo simulation studies are important skills for researchers and practitioners of educational and psychological measurement, but there are few resources on the topic specific to item response theory. Even fewer resources exist on the statistical software techniques to implement simulation studies. This article presents an introduction to the questions that simulation studies can, and cannot answer, along with the primary steps of the simulation study. We primarily focus on the parts of the simulation for which statistical software is used, presenting syntax for simulating 2PL data, and discuss how to extend the basic syntax to more complex models. We focus on SAS software in the article to build upon an existing syntax base, as most universities have SAS licenses. Two small simulations serve as didactic illustrations and to demonstrate parameter recovery of the SAS IRT procedure and an evaluation of the procedure's item-fit options.

When rating performance assessments, raters may ascribe different scores for the same performance when rubric application does not align with the intended application of the scoring criteria. Given performance assessment score interpretation assumes raters apply rubrics as rubric developers intended, misalignment between raters' scoring processes and the intended scoring processes may lead to invalid inferences from these scores. In an effort to standardize raters' scoring processes, an alternative scoring method was used. With this method, rubric developers' intended scoring processes are made explicit by requiring raters to respond to a series of selected-response statements resembling a decision tree. To determine if raters scored essays as intended using a traditional rubric and the alternative scoring method, an IRT model with a tree-like structure (IRTree) was specified to depict the intended scoring processes and fit to data from each scoring method. Results suggest raters using the alternative method may be better able to rate as intended and thus the alternative method may be a viable alternative to traditional rubric scoring. Implications of the IRTree model are discussed.

Interest in Bayesian analysis of item response theory (IRT) models has grown tremendously due to the appeal of the paradigm among psychometricians, advantages of these methods when analyzing complex models, and availability of general-purpose software. Possible models include models which reflect multidimensionality due to designed test structure, construct-irrelevant variance, and mixed item formats. Using Markov Chain Monte Carlo methods, models can be estimated and evaluated for model-fit. In addition to discussing Bayesian analysis, analyses of three IRT models designed to account for extreme response style are illustrated: IRTree, multidimensional nominal response model (MNRM), and modified generalized partial credit model (MPCM). While results indicated there may be little impact of model choice on substantive trait estimates for the data set studied herein, model-fit results favored the MNRM and MPCM over the IRTree Model.

In this digital ITEMS module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. Data visualizations are commonly used by measurement professionals to communicate results to examinees, the public, educators, and other stakeholders. To do so effectively, it is important that these visualizations communicate data efficiently and accurately. These visualizations can achieve graphical excellence when they simultaneously display data effectively, efficiently, and accurately. Unfortunately, measurement and statistical software default graphics typically fail to uphold these standards and are therefore not suitable for publication or presentation to the public. To illustrate best practices, the instructors provide an introduction to the graphical template language in SAS and show how elementary components can be used to make efficient, effective, and accurate graphics for a variety of audiences. The module contains audio-narrated slides, embedded illustrative videos, quiz questions with diagnostic feedback, a glossary, sample SAS code, and other learning resources.