1. Deep multimodal predictome for studying mental disorders
- Author
-
Rahaman, Abdur, Chen, Jiayu, Fu, Zening, Lewis, Noah, Iraji, Armin, Erp, Theo GM, and Calhoun, Vince D
- Subjects
Biological Psychology ,Psychology ,Schizophrenia ,Serious Mental Illness ,Basic Behavioral and Social Science ,Brain Disorders ,Mental Health ,Neurosciences ,Genetics ,Behavioral and Social Science ,Mental health ,Good Health and Well Being ,Humans ,Magnetic Resonance Imaging ,Neuroimaging ,Mental Disorders ,Neural Networks ,Computer ,functional network connectivity ,multimodal deep learning ,resting-state functional and structural MRI ,saliency ,schizophrenia classification ,single nucleotide polymorphism ,Cognitive Sciences ,Experimental Psychology ,Biological psychology ,Cognitive and computational psychology - Abstract
Characterizing neuropsychiatric disorders is challenging due to heterogeneity in the population. We propose combining structural and functional neuroimaging and genomic data in a multimodal classification framework to leverage their complementary information. Our objectives are two-fold (i) to improve the classification of disorders and (ii) to introspect the concepts learned to explore underlying neural and biological mechanisms linked to mental disorders. Previous multimodal studies have focused on naïve neural networks, mostly perceptron, to learn modality-wise features and often assume equal contribution from each modality. Our focus is on the development of neural networks for feature learning and implementing an adaptive control unit for the fusion phase. Our mid fusion with attention model includes a multilayer feed-forward network, an autoencoder, a bi-directional long short-term memory unit with attention as the features extractor, and a linear attention module for controlling modality-specific influence. The proposed model acquired 92% (p
- Published
- 2023