Search

Abstract This study investigates the effects of including patients’ clinical information on the performance of deep learning (DL) classifiers for disease location in chest X-ray images. Although current classifiers achieve high performance using chest X-ray images alone, consultations with practicing radiologists indicate that clinical data is highly informative and essential for interpreting medical images and making proper diagnoses. In this work, we propose a novel architecture consisting of two fusion methods that enable the model to simultaneously process patients’ clinical data (structured data) and chest X-rays (image data). Since these data modalities are in different dimensional spaces, we propose a spatial arrangement strategy, spatialization, to facilitate the multimodal learning process in a Mask R-CNN model. We performed an extensive experimental evaluation using MIMIC-Eye, a dataset comprising different modalities: MIMIC-CXR (chest X-ray images), MIMIC IV-ED (patients’ clinical data), and REFLACX (annotations of disease locations in chest X-rays). Results show that incorporating patients’ clinical data in a DL model together with the proposed fusion methods improves the disease localization in chest X-rays by 12% in terms of Average Precision compared to a standard Mask R-CNN using chest X-rays alone. Further ablation studies also emphasize the importance of multimodal DL architectures and the incorporation of patients’ clinical data in disease localization. In the interest of fostering scientific reproducibility, the architecture proposed within this investigation has been made publicly accessible( https://github.com/ChihchengHsieh/multimodal-abnormalities-detection ).

Disaster victim identification (DVI) entails a protracted process of evidence collection and data matching to reconcile physical remains with victim identity. Technology is critical to DVI by enabling the linkage of physical evidence to information. However, labelling physical remains and collecting data at the scene are dominated by low-technology paper-based practices. We ask, how can technology help us tag and track the victims of disaster? Our response to this question has two parts. First, we conducted a human–computer interaction led investigation into the systematic factors impacting DVI tagging and tracking processes. Through interviews with Australian DVI practitioners, we explored how technologies to improve linkage might fit with prevailing work practices and preferences; practical and social considerations; and existing systems and processes. We focused on tagging and tracking activities throughout the DVI process. Using insights from these interviews and relevant literature, we identified four critical themes: protocols and training; stress and stressors; the plurality of information capture and management systems; and practicalities and constraints. Second, these findings were iteratively discussed by the authors, who have combined expertise across electronics, data science, cybersecurity, human–computer interaction and forensic pathology. We applied the themes identified in the first part of the investigation to critically review technologies that could support DVI practitioners by enhancing DVI processes that link physical evidence to information. This resulted in an overview of candidate technologies matched with consideration of their key attributes. This study recognises the importance of considering human factors that can affect technology adoption into existing practices. Consequently, we provide a searchable table (as Supplementary information) that relates technologies to the key considerations and attributes relevant to DVI practice, for readers to apply to their own context. While this research directly contributes to DVI, it also has applications to other domains in which a physical/digital linkage is required, and particularly within high stress environments with little room for error.Key Points:Disaster victim identification (DVI) processes require us to link physical evidence and digital information. While technology could improve this linkage, experience shows that technological “solutions” are not always adopted in practice.Our study of the practices, preferences and contexts of Australian DVI practitioners suggests 10 critical considerations for these technologies.We review and evaluate 44 candidate technologies against these considerations and highlight the role of human factors in adoption.