Back to Search
Start Over
Multi-Modal Multi-Instance Learning for Retinal Disease Recognition
- Source :
- ACM Multimedia
- Publication Year :
- 2021
- Publisher :
- ACM, 2021.
-
Abstract
- This paper attacks an emerging challenge of multi-modal retinal disease recognition. Given a multi-modal case consisting of a color fundus photo (CFP) and an array of OCT B-scan images acquired during an eye examination, we aim to build a deep neural network that recognizes multiple vision-threatening diseases for the given case. As the diagnostic efficacy of CFP and OCT is disease-dependent, the network's ability of being both selective and interpretable is important. Moreover, as both data acquisition and manual labeling are extremely expensive in the medical domain, the network has to be relatively lightweight for learning from a limited set of labeled multi-modal samples. Prior art on retinal disease recognition focuses either on a single disease or on a single modality, leaving multi-modal fusion largely underexplored. We propose in this paper Multi-Modal Multi-Instance Learning (MM-MIL) for selectively fusing CFP and OCT modalities. Its lightweight architecture (as compared to current multi-head attention modules) makes it suited for learning from relatively small-sized datasets. For an effective use of MM-MIL, we propose to generate a pseudo sequence of CFPs by over sampling a given CFP. The benefits of this tactic include well balancing instances across modalities, increasing the resolution of the CFP input, and finding out regions of the CFP most relevant with respect to the final diagnosis. Extensive experiments on a real-world dataset consisting of 1,206 multi-modal cases from 1,193 eyes of 836 subjects demonstrate the viability of the proposed model.<br />Accepted by ACM Multimedia 2021 (Main Track)
- Subjects :
- FOS: Computer and information sciences
Modalities
Modality (human–computer interaction)
genetic structures
Artificial neural network
Computer Science - Artificial Intelligence
Computer science
business.industry
Computer Vision and Pattern Recognition (cs.CV)
Computer Science - Computer Vision and Pattern Recognition
Pattern recognition
Multimedia (cs.MM)
Domain (software engineering)
Artificial Intelligence (cs.AI)
Modal
Data acquisition
Artificial intelligence
Set (psychology)
business
Computer Science - Multimedia
Subjects
Details
- Database :
- OpenAIRE
- Journal :
- Proceedings of the 29th ACM International Conference on Multimedia
- Accession number :
- edsair.doi.dedup.....a82bfe771e0b63df6ce3485560e10fcb