1. Analysis of Inter-rater Agreement among Human Observers Who Judge Image Similarity
- Author
-
Bartłomiej Józef Dzieńkowski, Krzysztof Michalak, Michal Stanek, and Elżbieta Hudyma
- Subjects
Inter-rater reliability ,Variation (linguistics) ,Similarity (network science) ,Statistics ,Image pair ,Data mining ,computer.software_genre ,computer ,Measure (mathematics) ,Mathematics ,Image (mathematics) - Abstract
In this paper a problem of inter-rater agreement is discussed in the case of human observers who judge how similar pairs of images are. In such a case significant differences in judgment appear among the group of people. We have observed that for some pairs of images all values of similarity ratings are assigned by various people with approximately the same probability. To investigate this phenomenon in a more thorough mannerwe performed experiments in which inter-rater coefficients were used to measure the level of agreement for each given pair of images and for each pair of human judges. The results obtained in the experiments suggest that the variation of the level of agreement is considerable among pairs of images as well as among pairs of people. We suggest that this effect should be taken into account in design of computer systems using image similarity as a criterion.
- Published
- 2011
- Full Text
- View/download PDF