1. Multi-Modal Multi-Instance Learning for Retinal Disease Recognition
- Author
-
Yang Zhou, Weihong Yu, Youxin Chen, Jianchun Zhao, Dayong Ding, Xirong Li, Jie Wang, and Hailan Lin
- Subjects
FOS: Computer and information sciences ,Modalities ,Modality (human–computer interaction) ,genetic structures ,Artificial neural network ,Computer Science - Artificial Intelligence ,Computer science ,business.industry ,Computer Vision and Pattern Recognition (cs.CV) ,Computer Science - Computer Vision and Pattern Recognition ,Pattern recognition ,Multimedia (cs.MM) ,Domain (software engineering) ,Artificial Intelligence (cs.AI) ,Modal ,Data acquisition ,Artificial intelligence ,Set (psychology) ,business ,Computer Science - Multimedia - Abstract
This paper attacks an emerging challenge of multi-modal retinal disease recognition. Given a multi-modal case consisting of a color fundus photo (CFP) and an array of OCT B-scan images acquired during an eye examination, we aim to build a deep neural network that recognizes multiple vision-threatening diseases for the given case. As the diagnostic efficacy of CFP and OCT is disease-dependent, the network's ability of being both selective and interpretable is important. Moreover, as both data acquisition and manual labeling are extremely expensive in the medical domain, the network has to be relatively lightweight for learning from a limited set of labeled multi-modal samples. Prior art on retinal disease recognition focuses either on a single disease or on a single modality, leaving multi-modal fusion largely underexplored. We propose in this paper Multi-Modal Multi-Instance Learning (MM-MIL) for selectively fusing CFP and OCT modalities. Its lightweight architecture (as compared to current multi-head attention modules) makes it suited for learning from relatively small-sized datasets. For an effective use of MM-MIL, we propose to generate a pseudo sequence of CFPs by over sampling a given CFP. The benefits of this tactic include well balancing instances across modalities, increasing the resolution of the CFP input, and finding out regions of the CFP most relevant with respect to the final diagnosis. Extensive experiments on a real-world dataset consisting of 1,206 multi-modal cases from 1,193 eyes of 836 subjects demonstrate the viability of the proposed model., Accepted by ACM Multimedia 2021 (Main Track)
- Published
- 2021