Search

We developed a method to evaluate the accuracy of segmentation algorithms. Oversegmentation, undersegmentation, missing and spurious labels may all appear concurrently in machine segmented images. Segmentation algorithms make systematic errors and have different optimal operating ranges. Existing methods of segmentation evaluation do not evaluate these details. Our method, based on multiple feature recovery, reports systematic errors and indicates optimal operating ranges of features, besides measuring overall errors. A knowledge of the magnitude and type of errors can be used for tuning or selecting segmentation algorithms. Although our method was developed for CT scanning for security, it is applicable to other fields, including medical imaging, where multi-object feature recovery, non-uniform costs and a knowledge of optimal operating ranges are helpful.

Author(s): Karimi, Seemeen; Jiang, Xiaoqian; Cosman, Pamela; Martz, Harry | Abstract: BackgroundImaging systems used in aviation security include segmentation algorithms in an automatic threat recognition pipeline. The segmentation algorithms evolve in response to emerging threats and changing performance requirements. Analysis of segmentation algorithms' behavior, including the nature of errors and feature recovery, facilitates their development. However, evaluation methods from the literature provide limited characterization of the segmentation algorithms.ObjectiveTo develop segmentation evaluation methods that measure systematic errors such as oversegmentation and undersegmentation, outliers, and overall errors. The methods must measure feature recovery and allow us to prioritize segments.MethodsWe developed two complementary evaluation methods using statistical techniques and information theory. We also created a semi-automatic method to define ground truth from 3D images. We applied our methods to evaluate five segmentation algorithms developed for CT luggage screening. We validated our methods with synthetic problems and an observer evaluation.ResultsBoth methods selected the same best segmentation algorithm. Human evaluation confirmed the findings. The measurement of systematic errors and prioritization helped in understanding the behavior of each segmentation algorithm.ConclusionsOur evaluation methods allow us to measure and explain the accuracy of segmentation algorithms.