Measuring the fit between human judgments and automated alerting algorithms: a study of collision detection

INTRODUCTION Motivation A common paradigm in human-automation interaction requires a human to make a judgment, in parallel with an automated system, and then to consider, accept, or reject the automated […]
Methodologies for assessing human judgment in complex domains are important for the design of both displays that inform judgment and automated systems that suggest judgments. This paper uses the n-system lens model to evaluate the impact of displays on human judgment and to explicitly assess the similarity between human judgments and a set of potential judgment algorithms for use in automated systems. First, the need for and concepts underlying judgment analysis are outlined. Then the n-system lens model and its parameters are formally described. This model is then used to examine a previously conducted study of aircraft collision detection that had been analyzed using standard analysis of variance methods. Our analysis found the same main effects as did the earlier analysis. However, n-system lens model analysis was able to provide greater insight into the information relied upon for judgments and the impact of displays on judgment. Additionally, the analysis was able to identify attributes of human judgments that were--and were not--similar to judgments produced by automated systems. Potential applications of this research include automated and design and operator training.