Investigating the Consistency between Students' and Teachers' Ratings for the Assessment of Problem-Solving Skills with Many-Facet Rasch Measurement Model

Purpose: To assess students' problem-solving skills, this study aims to investigate the consistency between self- and peer-ratings in consideration of the teachers' ratings in the process. Method: This study was a descriptive study which examines the mathematical problem-solving skills with the MFRM model concerning self-, peer- and teachers' ratings. The study group consisted of 57 sixth grade students studying in a secondary school in Ankara. The data collection procedure was as follows: i) the students were trained in how to use rubric during the first week, ii) they practiced the rubric and a performance task in the second week and, iii) three performance tasks were applied in the following consecutively. These tasks included non-routine problem situations and two analytical rubrics were developed. For data analysis, student, steps, rater type, and task were determined as facets and rater was defined as dummy facet, and reliability statistics related to each facet were estimated. Findings: Ratings of performance tasks obtained from three-week data collection had high reliability coefficients according to MFRM modeling. The findings showed that self-, peer- and teachers' ratings vary in terms of generosity/severity according to the weeks given the rater types. Generally, self-raters were the most generous raters, whereas teachers were the most severe raters. In addition, generosity/severity of peer-raters gets closer to generosity/severity of teachers from the first task to the third one. Implications for Research and Practice: This research strengthens the possibility that peer-rating can provide reliable rating through appropriate training and practices.