A Comparison of Scoring Modalities for Performance Assessment: A Case of Constructed Response Items.

This study was designed to investigate several scoring modalities in terms of the accuracy of scores and the efficiency of scoring procedures. Five scoring modalities (SM) are conceptualized by considering tasks and raters as sources of error where "p,""t," and "r" represent person, task, and rater, respectively. These are: (1) SM1 [p x t x r]; (2) SM2 [p x (t:r)]; (3) SM3 [p x (r:t)]; (4) SM4 [(p:r) x t]; and (5) SM5 [r:(p x t)]. SMI and SM5 seem to be somewhat unrealistic for practical use in large-scale performance assessments. SM2 would be least preferable among the five scoring modalities conceptualized in this study for large-scale performance assessments. SM3 could be the best option when a high level of score accuracy is required and a high level of rater variation might be expected. If considering both score accuracy and efficiency of scoring procedures, SM4 should be considered for use in large-scale performance assessments. (Contains 2 figures, 3 tables, and 10 references.) (Author/SLD)