Beyond Lesion Detection: Towards Semantic Interpretation of Endoscopy Videos

Several computer-based medical systems have been proposed for automatic detection of abnormalities in a variety of medical imaging domains. The majority of these systems are based on binary supervised classification algorithms capable of discriminating abnormal from normal image patterns. However, this approach usually does not take into account that the normal content of images is diverse, including various kinds of tissues and artifacts. In the context of gastrointestinal video-endoscopy, which is addressed in this study, the semantics of the normal content include mucosal tissues, the hole of the lumen, bubbles, and debris. In this paper we investigate such a semantic interpretation of the endoscopy video content as an approach to improve lesion detection in a weakly supervised framework. This framework is based on a novel salient point detection algorithm, the bag-of-words image representation technique and multi-label classification. Advantages of the proposed method include: (a) It does not require detailed, pixel-level annotation of training images, instead image-level annotations are sufficient; (b) It enables a richer description of image content, which is beneficial for the discrimination of lesions. The annotation of the multi-labeled training images was performed using a novel annotation tool called RATStream. The results of the experiments performed in a wireless capsule endoscopy dataset with inflammatory lesions promises an improved performance for future generation diagnostic systems.