Search

Supervised machine learning utilizes large datasets, often with ground truth labels annotated by humans. While some data points are easy to classify, others are hard to classify, which reduces the inter-annotator agreement. This causes noise for the classifier and might affect the user's perception of the classifier's performance. In our research, we investigated whether the classification difficulty of a data point influences how strongly a prediction mistake reduces the "perceived accuracy". In an experimental online study, 225 participants interacted with three fictive classifiers with equal accuracy (73%). The classifiers made prediction mistakes on three different types of data points (easy, difficult, impossible). After the interaction, participants judged the classifier's accuracy. We found that not all prediction mistakes reduced the perceived accuracy equally. Furthermore, the perceived accuracy differed significantly from the calculated accuracy. To conclude, accuracy and related measures seem unsuitable to represent how users perceive the performance of classifiers., In CHI '22: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

This paper introduces CARS, an open source driving simulator tool for evaluating driver distraction. We present the map editor, the driving simulation tool and the analysis tool included in CARS. Based on initial results from initial user studies we illustrate the effectiveness of CARS in evaluating different secondary tasks (e.g., compare different UIs for navigation systems) with regard to driver distraction. (author's abstract)