JUDGE: A Laboratory Evaluation.

This paper describes an experiment performed to evaluate the JUDGE technique (Judged Utility Decision Generator). The JUDGE system is designed to dispatch aircraft on non-preplanned close air support missions, the number dispatched depending on judgments of target values made by experts at the times when targets appear. In contrast to an earlier field study employing Air Force officers as subjects, the current experiment included an extensive training period, longer scenarios, and repeated measurements. The subjects were fourteen students from the junior and senior Army ROTC classes at UCLA; they worked for two hours a day over the eight weeks of the experiment. For comparison, JUDGE was pitted against a second technique called DASC in the experiment, the name being taken from the Direct Air Support Center. This mode of operation is a hypothetical version of the system the Air Force currently uses, and is not a standard Air Porte procedure. The subjects performed in both modes against all the situations in the simulated war. In the JUDGE mode, the subjects assigned a value to each target as it appeared by comparing it with a "standard" target, which had a constant value of 100 throughout the experiment. A computer program then translated each subject's responses into dispatching decisions, and evaluated those decisions based on his value responses. Operating in the DASC mode, the subject received target reports identical to those in the JUDGE mode, but also containing a graph showing how effective various numbers of aircraft would be against the target. The subject himself assigned aircraft to the target, being permitted to allocate any even number from 0 to 16 of the aircraft remaining to him. The results clearly show the superiority of JUDGE over DASC when measured by an expected utility criterion. JUDGE performed at the 90% level when compared with the perfect possible performance. DASC reached a level of only 40%. We conclude from this that JUDGE is more effective in implementing a subject's value judgments than the subject is himself, The reliability of both systems was evaluated by measuring both the intersubject and intrasubject correlations. These two measures were substantially higher for the JUDGE system than for DASC, revealing that an exceptional amount of agreement occurs within JUDGE. Examination of the data processing task in isolation from the judgmental process in the decision environment led to the conclusion that JUDGE gains its advantage by turning over the necessary mechanical calculations to a computer. [ABSTRACT FROM AUTHOR]