1. A DIF Analysis of Item-Level Mode Effects for Computerized and Paper-and-Pencil Tests.
- Author
-
Schwarz, Richard D., Rich, Changhua, and Podrabsky, Tracy
- Abstract
This paper studied the usefulness of differential item functioning (DIF) methodology for examining potential mode effects. Although the goal was not to validate the comparability of the assessments per se, it is of interest to speculate why some formats could give rise to differential performance. Data were obtained from two instruments on which validation studies were conducted to determine if scores were comparable across modes of administration. The first instrument was a norm-referenced aptitude test called InView, administered to students in grades 4 to 9 in computer and paper modes for this study. Additional data were obtained from a Test of Adult Basic Education (TABE), also norm-referenced. Test takers were asked about their levels of computer experience and preferences. Two DIF statistics, the Linn-Harnisch procedure (1981) and the nonparametric Standardized Mean Difference were used to assess DIF. Results show that some levels of InView had substantial numbers of items that were flagged for the online comparisons with the standardization study, while others had relatively few. The TABE had two flagged items, and differences between computer-based and standardization groups were throughout the ability range with more apparent differences in the lower portion of the distribution. In general, students were neutral to mode or preferred computer-administered tests. Findings show that DIF methodology presents a well-studied method for examining group differences at the item level that can be used to examine mode of administration differences at the item level. Appendixes contain the student surveys and derivations of the DIF methods. (Contains 10 figures, 4 tables, and 10 references.) (SLD)
- Published
- 2003