Directly Comparing Computer and Human Performance in Language Understanding and Visual Reasoning.

Evaluation models are being developed for assessing artificial intelligence (AI) systems in terms of similar performance by groups of people. Natural language understanding and vision systems are the areas of concentration. In simplest terms, the goal is to norm a given natural language system's performance on a sample of people. The specific program under study is a natural language query system, IRUS--an interface between the user and the information desired. IRUS is designed to serve as a general purpose interface to a broad range of databases and expert systems. A pilot study is discussed, which was conducted with early elementary school and preschool students to determine the appropriate language understanding level at which to administer the IRUS test. In the vision area, common measures of visual tasks are being analyzed in terms of their appropriateness to the vision system. This is the inverse of the language exploration that began with the tasks and created the measures. A review by the vision community of approaches they would use to compare human and machine vision will determine if looking for consistent benchmarks is a feasible approach. (SLD)