Controlling for Rater Effects When Comparing Survey Items with Incomplete Likert Data. ACT Research Report Series.

This study was concerned with ranking items with Likert scale data when the items are subject to systematic (nonrandom) patterns of nonresponse. Researchers applied the rating scale model of D. Andrich (1978) to data from a survey in which the item response rate varied from less than 1% to over 90%. Data were from a section of the Student Opinion Survey for 7,133 students at 10 colleges. The person parameter in the rating scale model measured each rater's tendency to give higher or lower ratings in a fashion that was consistent across items. This tendency was named "pleasability" according to the Likert scale used in the study. Survey respondents differed substantially in pleasability. Groups responding to different items differed in average pleasability. Item parameter estimates in the rating scale model corrected for the differential exposure of survey items to pleasability. Compared to available case means, the rank order of items by their parameter estimates in the rating scale model had higher internal order consistency. (Contains 1 figure, 7 tables, and 26 references.) (Author/SLD)