Search

Advancements in deep learning techniques carry the potential to make significant contributions to healthcare, particularly in fields that utilize medical imaging for diagnosis, prognosis, and treatment decisions. The current state-of-the-art deep learning models for radiology applications consider only pixel-value information without data informing clinical context. Yet in practice, pertinent and accurate non-imaging data based on the clinical history and laboratory data enable physicians to interpret imaging findings in the appropriate clinical context, leading to a higher diagnostic accuracy, informative clinical decision making, and improved patient outcomes. To achieve a similar goal using deep learning, medical imaging pixel-based models must also achieve the capability to process contextual data from electronic health records (EHR) in addition to pixel data. In this paper, we describe different data fusion techniques that can be applied to combine medical imaging with EHR, and systematically review medical data fusion literature published between 2012 and 2020. We conducted a systematic search on PubMed and Scopus for original research articles leveraging deep learning for fusion of multimodality data. In total, we screened 985 studies and extracted data from 17 papers. By means of this systematic review, we present current knowledge, summarize important results and provide implementation guidelines to serve as a reference for researchers interested in the application of multimodal fusion in medical imaging.

Background: Non-T2 asthma and hypothyroidism share several inflammatory mechanisms in common. However, large-scale, real-world studies evaluating the association between asthma and hypothyroidism are lacking. The objective of this study was to evaluate the risk for asthma patients of developing hypothyroidism. Methods: In the retrospective cohort study, people with asthma were recruited from the Longitudinal Health Insurance Database in Taiwan. After excluding ineligible patients with a previous history of hypothyroidism, 1:1 propensity matching was conducted to select a non-asthma control group. Based on the multivariate Cox regression model, the adjusted hazard ratio of asthma patients developing hypothyroidism was calculated. Results: In total, 95,321 asthma patients were selected as the asthma group and the same amount of people without asthma were selected as the control group. The incidence levels of new-onset hypothyroidism in asthma and non-asthma groups were 8.13 and 6.83 per 100,000 people per year, respectively. Compared with the non-asthma group, the adjusted hazard ratio of the asthma group developing hypothyroidism was 1.217 (95% confidence interval, 1.091–1.357). Conclusions: We found having asthma to be associated with an increased risk of hypothyroidism. Clinicians should be concerned regarding the endocrinological and inflammatory interaction between the two diseases while caring for people with asthma.

Estimations of electrical conductivity are essential for biological object detection. This paper presents a distributed current source (DCS) based method to develop a bio-magnetic/eddy-current (Bio-M/EC) sensor for biological object detection. The DCS method directly calculates magnetic flux densities (MFDs) generated by the eddy-current induced in the biological object, which accounts for the effects of displacement current. The electromotive force (EMF) of the differential coil system is formulated in a closed-form solution to numerically investigate the effects of sweep frequency analysis applied on the Bio-M/EC sensor for biological object classification and tumor detection. The proposed method, along with a prototype of Bio-M/EC sensor, has been experimentally compared using sweep frequency analysis for detecting meat and bone.

Pulmonary embolism (PE) is a life-threatening clinical problem and computed tomography pulmonary angiography (CTPA) is the gold standard for diagnosis. Prompt diagnosis and immediate treatment are critical to avoid high morbidity and mortality rates, yet PE remains among the diagnoses most frequently missed or delayed. In this study, we developed a deep learning model-PENet, to automatically detect PE on volumetric CTPA scans as an end-to-end solution for this purpose. The PENet is a 77-layer 3D convolutional neural network (CNN) pretrained on the Kinetics-600 dataset and fine-tuned on a retrospective CTPA dataset collected from a single academic institution. The PENet model performance was evaluated in detecting PE on data from two different institutions: one as a hold-out dataset from the same institution as the training data and a second collected from an external institution to evaluate model generalizability to an unrelated population dataset. PENet achieved an AUROC of 0.84 [0.82-0.87] on detecting PE on the hold out internal test set and 0.85 [0.81-0.88] on external dataset. PENet also outperformed current state-of-the-art 3D CNN models. The results represent successful application of an end-to-end 3D CNN model for the complex task of PE diagnosis without requiring computationally intensive and time consuming preprocessing and demonstrates sustained performance on data from an external institution. Our model could be applied as a triage tool to automatically identify clinically important PEs allowing for prioritization for diagnostic radiology interpretation and improved care pathways via more efficient diagnosis.

222 Background: Prognostication in localized prostate cancer is reliant on non-specific tools, an issue that leads to the over- and under-treatment of patients. Various tissue-based molecular biomarkers have attempted to fill this unmet need, but most lack prospective randomized trial validation. Herein, we train and validate prognostic biomarkers in localized prostate cancer using five phase III randomized trials, by leveraging multi-modal deep learning on digital histopathology. Methods: Histopathology image data was generated from pre-treatment biopsy slides in five NRG Oncology phase III randomized radiotherapy prostate cancer trials (RTOG 9202, 9408, 9413, 9910, and 0126). The trials were randomly split into training (80%) and validation (20%) cohorts. A multi-modal artificial intelligence (MMAI) architecture was developed to take clinicopathologic and image-based (histopathology) data as input and predict binary outcomes. Using this architecture, various models were trained to predict relevant clinical endpoints: biochemical recurrence (BCR), distant metastasis (DM), prostate cancer-specific survival (PCaSS), and overall survival (OS). These models were then validated for measures of prognostic discrimination using the time-based area under the curve (AUC) method. Results: Clinicopathologic and histopathology image data was available for 5,654 of 7,957 eligible patients (71.1%), yielding 16.1 TB of data from 16,204 histopathology slides of pretreatment biopsy samples. After training the models, locking them, and evaluating them on the validation cohort, we found that the MMAI prognostic model had superior discrimination compared to the NCCN model (PSA, T-stage, and Gleason score) for 5-year DM (AUC of 0.84 vs 0.73), 5-year BCR (AUC of 0.69 vs 0.58), 10-year PCaSS (AUC of 0.79 vs 0.66), and 10-year OS (AUC of 0.65 vs 0.58). Within each of the individual trials in the validation cohort, the MMAI-model had superior performance compared to NCCN risk groups for all clinical endpoints. Conclusions: This represents the first ever development and validation of prognostic biomarkers in localized prostate cancer using multiple large phase III clinical trials. We have successfully validated that our MMAI-prognostic biomarkers are superior to standard clinical and pathologic variables in identifying future BCR, DM, PCaSS, and OS. This massively scalable technology is feasible and can help personalize the management of prostate cancer patients. Funding: This project was supported by grants U10CA180868 (NRG Oncology Operations), U10CA180822 (NRG Oncology SDMC), UG1CA189867 (NCORP), U24CA196067 (NRG Specimen Bank) from the National Cancer Institute (NCI).

223 Background: The current standard of care for men with intermediate- and high-risk localized prostate cancer treated with radiotherapy (RT) is the addition of androgen deprivation therapy (ADT). Presently, there are no validated predictive biomarkers to guide ADT use or duration in such men. Herein, we train and validate the first predictive biomarker for ADT use in prostate cancer using multiple phase III NRG Oncology randomized trials. Methods: Pre-treatment biopsy slides were digitized from five phase III NRG Oncology randomized trials of men receiving RT with or without ADT. The training set to develop the artificial intelligence (AI)-derived predictive biomarker included NRG/RTOG 9202, 9413, 9910, and 0126, and was trained to predict distant metastasis (DM). A multimodal deep learning architecture was developed to learn from both clinicopathologic and digital imaging histopathology data and identify differential outcomes by treatment type. After the model was locked, an independent biostatistician performed validation on NRG/RTOG 9408, a phase III randomized trial of RT +/- 4 months of ADT. The DM rates were calculated using cumulative incidence functions in biomarker positive and negative groups, and biomarker-treatment interaction was assessed using Fine-Gray regression such that death without DM was treated as a competing event. Results: Clinical and histopathological data was available for 5,654 of 7,957 eligible patients (71.1%). The training cohort included 3,935 patients and had a median follow-up of 13.6 years (IQR [10.2, 17.7]). After the AI-derived predictive ADT classifier was trained, it was validated in NRG/RTOG 9408 (n = 1719, median follow-up 17.6 years, IQR [15.0, 19.7]). In the NRG/RTOG 9408 validation cohort that had digital histopathology data, ADT significantly improved DM (HR 0.62, 95% CI [0.44, 0.87], p = 0.006), consistent with the published trial results. The biomarker-treatment interaction was significant (p-value = 0.0021). In patients with AI-biomarker positive disease (n = 673, 39%), ADT had a greater benefit compared to RT alone (HR 0.33, 95% CI [0.19, 0.57], p < 0.001). In the biomarker negative subgroup (n = 1046, 61%), the addition of ADT did not improve outcomes over RT alone (HR 1.00, 95% CI [0.64, 1.57], p = 0.99). The 15-year DM rate difference between RT versus RT+ADT in the biomarker negative group was 0.3%, vs biomarker positive group 9.4%. Conclusions: We have successfully validated in a phase III randomized trial the first predictive biomarker of ADT benefit with RT in localized intermediate risk prostate cancer using a novel AI-derived digital pathology-based platform. This AI-derived predictive biomarker demonstrates that a majority of patients treated with RT on NRG/RTOG 9408 did not require ADT and could have avoided the associated costs and side effects of this treatment.

This paper presents a distributed current source method for modeling weak magnetic and eddy-current (EC) fields induced in biological objects for electromagnetic stimulation and sensing applications. Unlike metallic objects with negligible displacement currents, the permittivity must be accounted for in biological objects. An axial symmetrical 2D EC field induced in a biological object is formulated in state space representation. Since EC fields cannot be directly measured, the solutions to three different variables that provide a means to infer the EC effects are derived using the distributed current source method, which are the magnetic flux density generated by the induced EC at a point, the lumped-parameter magnetic flux passing through a sensing coil, and its electromotive force. Illustrated in the context of two applications, the solutions are numerically evaluated by comparing results with that simulated by a commercial finite-element analysis.

Background: Pulmonary embolism (PE) is a life-threatening clinical problem and computed tomography pulmonary angiography (CTPA) is the gold standard for diagnosis. Prompt recognition of the diagnosis and immediate treatment is critical for controlling morbidity and mortality rates while PE remains among the diagnoses most frequently missed or delayed, in part due to lack of radiologist availability, diagnostic error, and delayed imaging results communication to clinical providers. Methods: We developed a deep learning model - PENet, to detect PE on volumetric CTPA scans without any manual intervention. The PENet is a 77-layer 3D convolutional neural network that has been pre-trained on the Kinetics-600 dataset and fine-tuned only on a retrospective CTPA dataset collected from a single academic institution. The PENet model was evaluated in detecting PE on the hold-out test set (198 CTPA studies) from the same institution as well as from the external test set (227 CTPA studies) collected from an external institution. Findings: PENet achieved an AUROC of 0.83 on automatically detecting central PE on the hold out test set and 0.84 on external institution's dataset. The results represent successful application of 3D convolutional neural network models for the complex task of PE diagnosis and demonstrates sustained performance on data from an external institution. Interpretation: Our model could be applied as a triage tool in ED to automatically identify clinically important central PEs allowing for prioritization for diagnostic radiology interpretation and improved care pathways via more efficient diagnosis. Funding Statement: Research reported in this publication was supported by the National Library of Medicine of the National Institutes of Health under Award Number R01LM012966, Stanford Child Health Research Institute (Stanford NIHNCATS-CTSA Grant #UL1 TR001085), and Philips Healthcare. Declaration of Interests: The authors state: "No conflict of interest, financial or other, exists." Ethics Approval Statement: All the patient data used in this study are de-identified and the study was approved by the Institutional Review Board (IRB).

This paper presents a discussion of practical experience related to the study of a cogeneration system where one of the four steam units occurs a failure of the low-pressure blades during peak load times of the summer months in Taiwan in the year 2007. This study investigates various scenarios consisting of shutting down the damaged unit for repairs and having continued operation of the unit by removing the low-pressure blades and replacing the stationary blade ring with buffer boards. Based on the simulation results, the numerical model has reflected strong agreement with the critical decisions made to operate the damaged unit continuously in a time of the blade failure.