Search

Many large-scale performance assessments include score resolution procedures for resolving discrepancies in rater judgments. The goal of score resolution is conceptually similar to person fit analyses: To identify students for whom observed scores may not accurately reflect their achievement. Previously, researchers have observed that rater-agreement methods and person fit analyses result in similar conclusions about which students' achievement estimates warrant additional investigation, and that score resolution generally improves person fit. We consider the implications of using person fit analysis as an initial step to identify performances for score resolution, and of using fit indices to identify raters to provide additional ratings. We simulated student responses to multiple-choice items and a writing task. We simulated various types of person misfit in the writing task and identified the persons who needed resolution using a person-fit index. Results indicate larger improvements in person fit after resolution when the person fit approach was used compared to a rater-agreement approach. With the fit-informed approach, person fit improved for of the misfitting students; with the rater-agreement approach, person fit improved for around one third of these students. We consider the implications of our findings for mixed-format assessments.

Researchers apply individual person fit analyses as a procedure for checking model-data fit for individual test-takers. When a test-taker "misfits," it means that the inferences from their test score regarding what they know and can do may not be accurate. One problem in applying individual person fit procedures in practice is the question of how much misfit it takes to make the test score an untrustworthy estimate of achievement. In this paper, we argue that if a person's responses generally follow a monotonic pattern, the resulting test score is "good enough" to be interpreted and used. We present an approach that applies statistical procedures from the Rasch and Mokken measurement perspectives to examine individual person fit based on this "good enough" criterion in real data from a performance assessment. We discuss how these perspectives may facilitate thinking about applying individual person fit procedures in practice.

Scoring procedures for many rater-mediated performance assessments include score resolution procedures in which a third rater adjudicates discrepancies between two raters' ratings of the same performance. There are numerous approaches for calculating resolved scores that involve different combinations of the original and third ratings. Using data from two content-area writing assessments, we examined the impact of applying no resolution and applying three resolution techniques: TQ, Parity, and the Operational resolution method used in the assessment program on the values and psychometric quality of student achievement estimates. We observed substantial differences in person fit, a model-based measure of the psychometric quality of student achievement estimates, across methods. Our results highlight the importance of examining person fit as part of evaluating the psychometric quality of rater-mediated performance assessments in which score resolution procedures are employed.

Individual person fit analyses provide important information regarding the validity of test score inferences for an "individual" test taker. In this study, we use data from an undergraduate statistics test (N = 1135) to illustrate a two-step method that researchers and practitioners can use to examine individual person fit. First, person fit is examined numerically with several indices based on the Rasch model (i.e., Infit, Outfit, and Between-Subset statistics). Second, person "mis"fit is presented graphically with person response functions, and these person response functions are interpreted using a heuristic. Individual person fit analysis holds promise for improving score interpretation in that it may detect potential threats to validity of score inferences for some test takers. Individual person fit analysis may also highlight particular subsets of items (on which a test taker performs unexpectedly) that can be used to further contextualize her or his test performance.

In this article, A. Adrienne Walker introduces the concept of person fit to education stakeholders as a source of evidence to inform the trustworthiness of a test score for interpretation and use (validity). Person fit analyses are used in educational measurement research to explore the degree to which a person's test score can be interpreted as a reasonable estimate of her achievement level. Although much academic literature has been produced on person fit, the findings and implications of this research remain largely unknown to education stakeholders. Walker argues that the practice of using test scores based on misfitting response patterns is problematic, and by including individual person fit in quality procedures, assessment research, practice, and policy can improve the way educational assessments serve instruction.

The idea that test scores may not be valid representations of what students know, can do, and should learn next is well known. Person fit provides an important aspect of validity evidence. Person fit analyses at the individual student level are not typically conducted and person fit information is not communicated to educational stakeholders. In this study, we focus on a promising method for detecting and conveying person fit for large-scale educational assessments. This method uses multilevel logistic regression (MLR) to model the slopes of the person response functions, a potential source of person misfit for IRT models. We apply the method to a representative sample of students who took the writing section of the SAT (N = 19,341). The findings suggest that the MLR approach is useful for providing supplemental evidence of model-data fit in large-scale educational test settings. MLR can be useful for detecting general misfit at global and individual levels. However, as with other model-data fit indices, the MLR approach is limited in providing information regarding only some types of person misfit.

"Game-Based Assessments: A Promising Way to Create Idiographic Perspectives" (Adrienne Walker and George Englehard) comments on: "How Task Features Impact Evidence from Assessments Embedded in Simulations and Games" by Russell G. Almond, Yoon Jeon Kim, Gertrudes Velasquez, and Valerie J. Shute. Here, Walker and Englehard write that they believe assessments embedded within simulations and games represent the cutting edge of next generation assessments and that Almond, Kim, Velasquez, and Shute (Focus article) suggest that a careful consideration of the principles of evidence-centered assessment design can improve game-based assessments. Student information can be gathered, analyzed, and systematically used to substantiate claims and inferences about student academic performance in ways that are fun, engaging, and perhaps even unknown to the students who are being assessed. The purpose of the Focus article is to describe how game "tasks" can be modified so that certain psychometric properties, such as task difficulty and task discrimination, can be identified to support the construct validity of game-based assessments. One of the key ideas of the Focus article is the potential of computerized assessments to collect and, therefore, potentially measure important aspects of student achievement that cannot be easily measured by traditional assessment formats. For example, constructs such as student creativity and persistence are not typically measured in traditional assessments. However, within the format of a game-based assessment, where vast amounts of data on the student are recorded, it is conceivable to measure both cognitive and affective student characteristics. Obstacles regarding the appropriate measurement of cognitive and affective constructs for game-based assessment exist, and they can be addressed to ensure valid measurement of student achievement. Walker and Englehard note that the potential to collect data on more than one dimension of student performance has exciting implications for assessment practice. This commentary suggests that game-based assessments using evidence-centered design provide an opportunity to focus on students as multidimensional individuals acting within a particular context. A more complex and nuanced picture of a student's academic achievement may be drawn from information collected with computerized game-based assessment. Further, it suggests that the pursuit of construct validation for measuring student performance can be theoretically grounded in both nomothetic and idiographic scientific inquiry.

Person fit statistics provide equivocal interpretations regarding aberrant responses. This study uses person response functions (PRF) to supplement the interpretation of person fit statistics. Sixty-three multiple-choice items were administered to a sample of persons (N= 31) who used guessing strategies to answer them. After answering each item, participants indicated which guessing strategy they used. The data were analyzed with a Rasch (1960) model, where the item calibrations were anchored to values obtained when the items were appropriately administered. The participants showed poor model-data fit as expected. Further examination of person misfit using person response functions suggests that PRF can provide information about absolute person fit to a model, whereas fit statistics provide information about relative fit, given the other persons in the testing group. PRF can also provide information about where and how person responses misfit the model. This additional information can assist practitioners in using and interpreting individual scores appropriately. [ABSTRACT FROM AUTHOR]