Assessing the Reliability of a Chemical Engineering Problem-solving Rubric when Using Multiple Raters.

This evidence-based practice paper discusses the preliminary validation of a project modified version of the Promoting Problem Solving Proficiency in First Year Engineering (PROCESS). The full rating plan required four raters to use the PROCESS to assess the problem-solving ability of ~70 engineering students randomly selected from two undergraduate cohorts at two Midwest universities. The many-facet Rasch measurement model has the psychometric properties to determine if there are any characteristics other than problem-solving that influence the scores assigned to students, such as rater bias or differential item functioning. Prior to implementing the full rating plan, the analysis examined how raters interacted with the six items on the modified PROCESS when scoring a random selection of 20 students' solutions to one textbook homework problem. Follow up inter-rater reliability meetings enabled rater discussion of rationale for discrepancies observed in the ratings. Differences in conceptions of the latent construct of problem-solving were resolved by recourse to the theoretical framework that informed the development of the PROCESS. This iterative process resulted in substantial increases in construct validity and measurement reliability when raters completed another round of assessment. Evidence indicated that raters increased their understanding of how rating scale categories related to levels of the latent construct. This paper describes the impacts and benefits this method of psychometric evaluation of rater-mediated assessments hold for the implementation of the full rating plan of student outcomes, as well as for the field of engineering education more broadly. [ABSTRACT FROM AUTHOR]