Back to Search Start Over

Multimodal fusion models for pulmonary embolism mortality prediction.

Authors :
Cahan N
Klang E
Marom EM
Soffer S
Barash Y
Burshtein E
Konen E
Greenspan H
Source :
Scientific reports [Sci Rep] 2023 May 09; Vol. 13 (1), pp. 7544. Date of Electronic Publication: 2023 May 09.
Publication Year :
2023

Abstract

Pulmonary embolism (PE) is a common, life threatening cardiovascular emergency. Risk stratification is one of the core principles of acute PE management and determines the choice of diagnostic and therapeutic strategies. In routine clinical practice, clinicians rely on the patient's electronic health record (EHR) to provide a context for their medical imaging interpretation. Most deep learning models for radiology applications only consider pixel-value information without the clinical context. Only a few integrate both clinical and imaging data. In this work, we develop and compare multimodal fusion models that can utilize multimodal data by combining both volumetric pixel data and clinical patient data for automatic risk stratification of PE. Our best performing model is an intermediate fusion model that incorporates both bilinear attention and TabNet, and can be trained in an end-to-end manner. The results show that multimodality boosts performance by up to 14% with an area under the curve (AUC) of 0.96 for assessing PE severity, with a sensitivity of 90% and specificity of 94%, thus pointing to the value of using multimodal data to automatically assess PE severity.<br /> (© 2023. The Author(s).)

Details

Language :
English
ISSN :
2045-2322
Volume :
13
Issue :
1
Database :
MEDLINE
Journal :
Scientific reports
Publication Type :
Academic Journal
Accession number :
37160926
Full Text :
https://doi.org/10.1038/s41598-023-34303-8