Search

Trained raters used descriptive graphic rating scales (i.e., continuous score scales) to evaluate students' work. The scales incorporated different combinations of design features (i.e., presence or absence of a defined midpoint or hatchmarks or both). The study's purposes were to (a) decide how many points each continuous scale could be considered to have and (b) determine whether different design features affected reliability. The data were analyzed using Facets (Linacre, 1996), a Rasch-based rating scale analysis computer program. In addition, intraclass correlational analyses were run. All the continuous scales could be considered to have at least 5 points, whereas some of the scales could be viewed as having as many as 7 to 10 points. If the scales were considered to have 5 points, then student separation reliabilities ranged from .79 to .91, whereas the intraclass correlation coefficients ranged from .73 to .85. Scales with defined midpoints did not have higher student separation reliabilities or intraclass correlations than scales with no defined midpoints. Similarly, scales with more hatchmarks were no more or less reliable than scales with fewer hatchmarks. [ABSTRACT FROM AUTHOR]

Focuses on the development of empirical and theoretical foundations for establishing and monitoring the validity and reliability of portfolio assessments in education. Relation between the appropriateness of rater reasoning and the psychometric features of ratings; Sufficiency of quantitative information to detect rating problems; Impetus to the discrepancy in rating judgement.