A multi-modal extraction integrated model for neuropsychiatric disorders classification.

Convolutional neural networks (CNNs) provide high-precision automatic classification of neuropsychiatric disorders based on images. However, the "black box" nature leads to poor interpretability of CNN. This study constructs an integrated model for neuropsychiatric disorders classification from multi-modal data. The proposed model consists of a novel multi-scale image features extraction neural network (MSFM) and a XGBoost. The proposed MSFM extracts the pixel context semantic information from fMRI images with different scales, which employs token and channel-mixing strategy to enhance the information communication between context semantic information. XGBoost is used to extract phenotypic feature from phenotypic records. Based on the integration of phenotypic and image features, a comparative interpretable classification of mental disorders can be achieved. The overall accuracy, sensitivity, and recall of the binary classification (healthy controls & neuropsychiatric disorders) of the integrated model are 90.23%, 91.08%, and 89.33%, respectively. The visualization of image features and the phenotypic features present consistency in the brain regions, increasing the interpretability of the MSFM. Especially, through visual statistical analysis of the test set, it was found that there are differences in the distribution of ADHD, BD, and SD in the brain regions. Our solution may provide psychiatrists with ideas for comparative examinations and diagnosis. • A multi-modal extraction integrated model is proposed for neuropsychiatric disorders classification, the integrated model consists of MSFM network and XGBoost. • The proposed MSFM extracts multi-scale pixel context semantic information from fMRI images. XGBoost is used to extract phenotypic features from phenotypic records. • The visualization of image features and phenotypic features presents consistency in the brain regions, increasing the interpretability of the MSFM. • Deep visualization analysis reveals differences between subtypes of NDs. [ABSTRACT FROM AUTHOR]