Hierarchical Multimodal Fusion of Deep-Learned Lesion and Tissue Integrity Features in Brain MRIs for Distinguishing Neuromyelitis Optica from Multiple Sclerosis

Neuromyelitis optica spectrum disorder (NMOSD) is a disease of the central nervous system that is often misdiagnosed as multiple sclerosis (MS) because they share similar clinical and radiological characteristics. Two key pathological signs of NMOSD and MS that are detectable on magnetic resonance imaging (MRI) are white matter lesions and alterations in tissue integrity as measured by fractional anisotropy (FA) values on diffusion tensor images (DTIs). This paper proposes a multimodal deep learning model that discovers latent features in brain lesion masks and DTIs for distinguishing NMOSD from MS. The main technical challenge is to optimally extract and integrate features from two very heterogeneous image types (lesion masks and FA maps). Our solution is to first build two modality-specific pathways, each designed to accommodate the expected feature density and scale, then integrate them into a hierarchical multimodal fusion (HMF) model. The HMF model contains two multimodal fusion layers operating at two different scales, which in turn are joined by a multi-scale fusion layer. We hypothesize that the HMF approach would allow the automatic extraction of joint-features of heterogeneous image types to be optimized with greater efficiency and accuracy than the traditional multimodal approach of combining only the top-layer modality-specific features with a single fusion layer. The proposed model gives an average diagnostic accuracy of 81.3% (85.3% sensitivity and 75.0% specificity) on 82 NMOSD patients and 52 MS patients in a seven-fold cross-validation, which significantly outperforms the user-defined MRI features previously used in clinical studies, as well as deep-learned features using the conventional fusion approach.