An Investigation of Differential Mode Effects When Comparing Paper and Online ACT Testing. ACT Research & Policy. Technical Brief

In recent ACT mode comparability studies, students testing on laptop or desktop computers earned slightly higher scores on average than students who tested on paper, especially on the ACT® reading and English tests (Li et al., 2017). Equating procedures adjust for such "mode effects" to make ACT scores comparable regardless of testing mode. However, it remains possible that the mode effects are different for different groups of students. For example, differences in performance between paper and online testing may be different for groups with different levels of comfort taking tests on computers. Thus, a general mode adjustment may be inappropriate. The purpose of this study was to explore the possibility of differential mode effects by gender, race/ethnicity, and ability using data from three recent mode comparability studies (Steedle, Pashley, & Cho, 2020). Results indicated that mode effects in English, reading, math, and science did not vary significantly between genders or race/ethnicity groups. Analyses detected significant interactions between mode effects and ability because mode effects tended to be smaller for lower ability examinees. Fortunately, equating processes appropriately adjust scores for differential mode effects by ability.