Quality assessment of classification and cluster maps without ground truth knowledge

This work focuses on two challenging types of problems related to quality assessment and comparison of thematic maps generated from remote sensing (RS) images when little or no ground truth knowledge is available. These problems occur when: (1) competing thematic maps, generated from the same input RS image, assumed to be available, must be compared, but no ground truth knowledge is found to assess the accuracy of the mapping problem at hand, and (2) the generalization capability of competing classifiers must be estimated and compared when the small/unrepresentative ground truth problem affects the RS inductive learning application at hand. Specifically focused on badly posed image classification tasks, this paper presents an original data-driven (i.e., unsupervised) thematic map quality assessment (DAMA) strategy complementary (not alternative) in nature to traditional supervised map accuracy assessment techniques, driven by the expensive and error-prone digitization of ground truth knowledge. To compensate for the lack of supervised regions of interest, DAMA generates so-called multiple reference cluster maps from several blocks of the input RS image that are clustered separately. Due to the unsupervised (i.e., subjective) nature (ill-posedness) of data clustering, DAMA provides no (absolute) map accuracy measure. Rather, DAMA's map quality indexes are to be considered unsupervised (i.e., subjective) relative estimates of labeling and segmentation consistency between every competing map at hand and the set of multiple reference cluster maps. In two badly posed RS image mapping experiments, DAMA's map quality measures are proven to be: (1) useful in the relative comparison of competing mapping systems; (2) consistent with theoretical expectations; and (3) in line with mapping quality criteria adopted by expert photointerpreters. Documented limitations of DAMA are that it is intrinsically heuristic due to the subjective nature of the clustering problem, and like any evaluation measure, it cannot be injective.