Search

Decision making processes often rely on subjective evaluations provided by human raters. In the absence of a gold standard against which check evaluation trueness, rater's evaluative performance is generally measured through rater agreement coefficients. In this study some parametric and non-parametric inferential benchmarking procedures for characterizing the extent of rater agreement—assessed via kappa-type agreement coefficients—are illustrated. A Monte Carlo simulation study has been conducted to compare the performance of each procedure in terms of weighted misclassification rate computed for all agreement categories. Moreover, in order to investigate whether the procedures overestimate or underestimate the level of agreement, misclassifications have been computed also for each specific category alone. The practical application of coefficients and inferential benchmarking procedures has been illustrated via two real data sets exemplifying different experimental conditions so as to highlight performance differences due to sample size.

In this paper, a weighted variant of the normalized pairwise angular distance metric is proposed. The inclusion of position weights aims at penalizing inversions in the top of the ranking more than inversions in the tail of the ranking. The performance of the proposed weighted distance metric for assessing ranking dissimilarity and its impact on a procedure for testing inter-group heterogeneity have been investigated via a Monte Carlo simulation study under several scenarios—differing for group size, number of ranked alternatives and system of hypotheses—and compared against those obtained for the unweighted variant.