Ontology-Guided Approach to Retrieving Disease Manifestation Images for Health Image Base Construction

Building a comprehensive medical image database, in the spirit of the UMLS, can be beneficial for assisting diagnosis, patient education and self-care. However, a highly curated, comprehensive image database is difficult to collect as well as to annotate. We present an approach to combine visual object detection technologies with medical ontology to automatically mine web photos and retrieve a large number of disease manifestation images with minimal manual labeling. Comparing to a supervised approach, our ontology-guided approach reduces manual labeling effort to 1/10 on a variety of eye/ear/mouth diseases and improves the precision of retrieval by over 10% in many cases.