Who Reads Ratings: Knowledge, Attitude, and Practice of Users of Student Ratings of Instruction.

What users of student ratings of instruction know about ratings and what they need to know to make valid and reliable use of ratings data were studied. An item bank of 153 multiple-choice items was developed from sources of information about student ratings. The full item bank had five subsets: (1) knowledge of ratings concepts; (2) knowledge of quantitative issues for interpreting/applying ratings; (3) simulated practice exercises; (4) attitudes toward ratings issues; and (5) demographic variables associated with rating users. Items were validated by a panel of 24 experts to develop the shorter questionnaires mailed to faculty at three institutions and members of a professional group for a total of 779 respondents. Participation was 15% for one institution and 20% for each of the other groups, except that the participation of the 23 experts in the field of ratings use was 100%. The magnitude of the difference between the expert scores and the scores of ratings users confirmed that users may not know all that they need about using student ratings. Those with a negative attitude were likely to use student ratings less often and less well. Positive attitudes were associated with higher knowledge scores. Many users of ratings appear insufficiently aware of the issues currently set forth in ratings literature to be able to make decisions for summative or formative purposes. Five data tables are included. Appendix A is the 69-item ratings survey, and Appendix B lists the names and academic affiliations of the experts from the first validation. (SLD)