Search

Within a workshop on evaluation methods for automated vehicles (AVs) at the Driving Assessment 2019 symposium in Santa Fe; New Mexico, a heuristic evaluation methodology that aims at supporting the development of human–machine interfaces (HMIs) for AVs was presented. The goal of the workshop was to bring together members of the human factors community to discuss the method and to further promote the development of HMI guidelines and assessment methods for the design of HMIs of automated driving systems (ADSs). The workshop included hands-on experience of rented series production partially automated vehicles, the application of the heuristic assessment method using a checklist, and intensive discussions about possible revisions of the checklist and the method itself. The aim of the paper is to summarize the results of the workshop, which will be used to further improve the checklist method and make the process available to the scientific community. The participants all had previous experience in HMI design of driver assistance systems, as well as development and evaluation methods. They brought valuable ideas into the discussion with regard to the overall value of the tool against the background of the intended application, concrete improvements of the checklist (e.g., categorization of items; checklist items that are currently perceived as missing or redundant in the checklist), when in the design process the tool should be applied, and improvements for the usability of the checklist.

Objective: The human–machine interface (HMI) is a crucial part of every automated driving system (ADS). In the near future, it is likely that—depending on the operational design domain (ODD)—different levels of automation will be available within the same vehicle. The capabilities of a given automation level as well as the operator’s responsibilities must be communicated in an appropriate way. To date, however, there are no agreed-upon evaluation methods that can be used by human factors practitioners as well as researchers to test this. Methods: We developed an iterative test procedure that can be applied during the product development cycle of ADS. The test procedure is specifically designed to evaluate whether minimum requirements as proposed in NHTSA’s automated vehicle policy are met. Results: The proposed evaluation protocol includes (a) a method to identify relevant use cases for testing on the basis of all theoretically possible steady states and mode transitions of a given ADS; (b) an expert-based heuristic assessment to evaluate whether the HMI complies with applicable norms, standards, and best practices; and (c) an empirical evaluation of ADS HMIs using a standardized design for user studies and performance metrics. Conclusions: Each can be used as a stand-alone method or in combination to generate objective, reliable, and valid evaluations of HMIs, focusing on whether they meet minimum requirements. However, we also emphasize that other evaluation aspects such as controllability, misuse, and acceptance are not within the scope of the evaluation protocol.

In most levels of vehicle automation, drivers will not be merely occupants or passengers of automated vehicles. Especially in lower levels of automation, where the driver is still required to serve as a fallback level (SAE L3) or even as a supervisor (SAE L2), there is a need to communicate relevant system states (e.g., that the automated driving system works reliably or that there is a need for manual intervention) via the Human-Machine Interface (HMI). However, there are currently no agreed-upon guidelines that apply specifically to HMIs for automated driving. In this paper, we summarize design recommendations for visual-auditory and visual-vibrotactile HMIs derived from empirical research, applicable standards and design guidelines pertaining to in-vehicle interfaces. On this basis, we derive an initial set of principles and criteria for guiding the development and design of automated vehicle HMIs. A heuristic evaluation methodology consisting of an itemized checklist evaluation that can be used to verify that basic HMI requirements formulated in the guidelines are met is also presented. The heuristic evaluation involves an inspection of the HMI during typical use cases, judging their compliance with the proposed guidelines and documentation of identified instances of non-compliance. Taken together, the combination of the proposed guidelines and the heuristic evaluation methodology form the basis for both design and validation recommendations of automated vehicle HMIs, that can serve the industry in the important evolution of automation within vehicles.