Search

Advancements in deep learning techniques carry the potential to make significant contributions to healthcare, particularly in fields that utilize medical imaging for diagnosis, prognosis, and treatment decisions. The current state-of-the-art deep learning models for radiology applications consider only pixel-value information without data informing clinical context. Yet in practice, pertinent and accurate non-imaging data based on the clinical history and laboratory data enable physicians to interpret imaging findings in the appropriate clinical context, leading to a higher diagnostic accuracy, informative clinical decision making, and improved patient outcomes. To achieve a similar goal using deep learning, medical imaging pixel-based models must also achieve the capability to process contextual data from electronic health records (EHR) in addition to pixel data. In this paper, we describe different data fusion techniques that can be applied to combine medical imaging with EHR, and systematically review medical data fusion literature published between 2012 and 2020. We conducted a systematic search on PubMed and Scopus for original research articles leveraging deep learning for fusion of multimodality data. In total, we screened 985 studies and extracted data from 17 papers. By means of this systematic review, we present current knowledge, summarize important results and provide implementation guidelines to serve as a reference for researchers interested in the application of multimodal fusion in medical imaging.

Over the past few years, the processing of motor imagery (MI) electroencephalography (EEG) signals has been attracted for developing brain-computer interface (BCI) applications, since feature extraction and classification of these signals are extremely difficult due to the inherent complexity and tendency to artifact properties of them. The BCI systems can provide a direct interaction pathway/channel between the brain and a peripheral device, hence the MI EEG-based BCI systems seem crucial to control external devices for patients suffering from motor disabilities. The current study presents a semi-supervised model based on three-stage feature extraction and machine learning algorithms for MI EEG signal classification in order to improve the classification accuracy with smaller number of deep features for distinguishing right- and left-hand MI tasks. Stockwell transform is employed at the first phase of the proposed feature extraction method to generate two-dimensional time–frequency maps (TFMs) from one-dimensional EEG signals. Next, the convolutional neural network (CNN) is applied to find deep feature sets from TFMs. Then, the semi-supervised discriminant analysis (SDA) is utilized to minimize the number of descriptors. Finally, the performance of five classifiers, including support vector machine, discriminant analysis, k-nearest neighbor, decision tree, random forest, and the fusion of them are compared. The hyperparameters of SDA and mentioned classifiers are optimized by Bayesian optimization to maximize the accuracy. The presented model is validated using BCI competition II dataset III and BCI competition IV dataset 2b. The performance metrics of the proposed method indicate its efficiency for classifying MI EEG signals.

Lung Cancer screening trials have demonstrated significant mortality reduction. Low-Dose Computed Tomography (LDCT) screening can frequently discover many small nodules in at risk participants. However classification of these, sub-cm nodules as cancerous or benign is a challenging task even for expert clinicians. We use machine learning (ML) and deep learning (CNN) techniques to differentiate, sub-cm cancerous and benign nodules. Data for this study is drawn from a screening study (PanCan) from which we selected 612 distinct nodules (140 cancerous, and ~size matched 472 benign). Both methods demonstrated a ~80% accuracy, whereas currently used measures (size) had a 68% accuracy.

X-ray tensor tomography (XTT) is a novel imaging modality for the three-dimensional reconstruction of X-ray scattering tensors from dark-field images obtained in a grating interferometry setup. The two-dimensional dark-field images measured in XTT are degraded by noise effects, such as detector readout noise and insufficient photon statistics, and consequently, the three-dimensional volumes reconstructed from this data exhibit noise artifacts. In this paper, we investigate the best way to incorporate a denoising technique into the XTT reconstruction pipeline, i.e., the popular total variation denoising technique. We propose two different schemes of including denoising in the reconstruction process, one using a column block-parallel iterative scheme and one using a whole-system approach. In addition, we compare the results when using a simple denoising approach applied either before or after reconstruction. The effectiveness is evaluated qualitatively and quantitatively based on datasets from an industrial sample and a clinical sample. The results clearly demonstrate the superiority of including denoising in the reconstruction process, along with slight advantages of the whole-system approach.

Lung Cancer screening trials have demonstrated significant mortality reduction. Low-Dose Computed Tomography (LDCT) screening can frequently discover many small nodules in at risk participants. However classification of these, sub-cm nodules as cancerous or benign is a challenging task even for expert clinicians. In this work we use machine learning (ML) techniques to differentiate, cancerous (clinically confirmed) and benign nodules (>5 years of follow-up). Data for this study is drawn from a screening study (PanCan) from which we selected 613 distinct nodules (141 cancerous, and ~size matched 472 benign). We analyzed texture and shape features (~170) that are extracted from the nodule with and without perimeter transition pixels to control for perimeter effects. Features are also extracted from the ring of parenchyma surrounding the nodule to account for tumor effects on surrounding tissue. From the equivalent location in the opposite lung, parenchymal characteristics were extracted which we use to normalize the nodule texture features in an effort to reduce scanner bias. Our preliminary results for machine learning classification have shown model accuracies of up to ~80% (feature selection and classification algorithm dependent). Radiomic feature data can be combined with patient demographic variables such as age, sex, and smoking status to further improve our models, reaching ~84% classification accuracy. When normalized by the opposite lung, 58% of texture features showed improved classification ability. A single feature from the ring of parenchyma surrounding the nodule achieved 73.5% accuracy. Others have shown that nodule area/volume is a good classifier; for this data set it gave 68% accuracy. Data exploration to study classification accuracies among different subsets of patients reveals a number of interesting trends. We have noted for instance that in patients without emphysema, tumors can be classified with much higher accuracy (>90%) than in those who suffer from it. This and other results detailing classification among specific patient groupings may prove relevant in any potential future clinical implementation of these results. Citation Format: Rohan Abraham, Ian Janzen, Saeed Seyyedi, Sukhinder Khattra, John Mayo, Ren Yuan, Renelle Meyers, Stephen Lam, Calum MacAulay. Machine learning CADx process for classification of lung nodules below the Lung-RADS 4A threshold in LDCT scans [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-053.

Liver CT perfusion (CTP) is used in the detection, staging, and treatment response analysis of hepatic diseases. Unfortunately, CTP radiation exposures is significant, limiting more widespread use. Traditional CTP data processing reconstructs individual temporal samples, ignoring a large amount of shared anatomical information between temporal samples, suggesting opportunities for improved data processing. We adopt a prior-image-based reconstruction approach called Reconstruction of Difference (RoD) to enable low-exposure CTP acquisition. RoD differs from many algorithms by directly estimating the attenuation changes between the current patient state and a prior CT volume. We propose to use a high-fidelity unenhanced baseline CT image to integrate prior anatomical knowledge into subsequent data reconstructions. Using simulation studies based on a 4D digital anthropomorphic phantom with realistic time-attenuation curves, we compare RoD with conventional filtered-backprojection, penalized-likelihood estimation, and prior image penalized-likelihood estimation. We evaluate each method in comparisons of reconstructions at individual time points, accuracy of estimated time-attenuation curves, and in an analysis of common perfusion metric maps including hepatic arterial perfusion, hepatic portal perfusion, perfusion index, and time-to-peak. Results suggest that RoD enables significant exposure reductions, outperforming standard and more sophisticated model-based reconstruction, making RoD a potentially important tool to enable low-dose liver CTP.

Digital breast tomosynthesis (DBT) is an innovative imaging modality that provides 3D reconstructed images of breast to detect the breast cancer. Projections obtained with an X-ray source moving in a limited angle interval are used to reconstruct 3D image of breast. Several reconstruction algorithms are available for DBT imaging. Filtered back projection algorithm has traditionally been used to reconstruct images from projections. Iterative reconstruction algorithms such as algebraic reconstruction technique (ART) were later developed. Recently, compressed sensing based methods have been proposed in tomosynthesis imaging problem. We have developed an object-oriented simulator for 3D digital breast tomosynthesis (DBT) imaging system using C++ programming language. The simulator is capable of implementing different iterative and compressed sensing based reconstruction methods on 3D digital tomosynthesis data sets and phantom models. A user friendly graphical user interface (GUI) helps users to select and run the desired methods on the designed phantom models or real data sets. The simulator has been tested on a phantom study that simulates breast tomosynthesis imaging problem. Results obtained with various methods including algebraic reconstruction technique (ART) and total variation regularized reconstruction techniques (ART+TV) are presented. Reconstruction results of the methods are compared both visually and quantitatively by evaluating performances of the methods using mean structural similarity (MSSIM) values.

X-ray Tensor Tomography (XTT) is a recently developed imaging modality that provides reconstruction of X-ray scattering tensors from dark-field projections obtained in a grating interferometry setup. In this work we present a novel component-based total variation (TV) regularized reconstruction technique for XTT data. First results show promising qualitative improvements of the reconstructed tensors as well as reduced noise and reduced streak artifacts.

Attenuation based X-ray micro computed tomography (XCT) provides three-dimensional images with micrometer resolution. However, there is a trade-off between the smallest size of the structures that can be resolved and the measurable sample size. In this letter, we present an imaging method using a compact laboratory setup that reveals information about micrometer-sized structures within samples that are several orders of magnitudes larger. We combine the anisotropic dark-field signal obtained in a grating interferometer and advanced tomographic reconstruction methods to reconstruct a six dimensional scattering tensor at every spatial location in three dimensions. The scattering tensor, thus obtained, encodes information about the orientation of micron-sized structures such as fibres in composite materials or dentinal tubules in human teeth. The sparse acquisition schemes presented in this letter enable the measurement of the full scattering tensor at every spatial location and can be easily incorporated in a practical, commercially feasible laboratory setup using conventional X-ray tubes, thus allowing for widespread industrial applications.

This paper presents a compressed sensing based reconstruction method for 3D digital breast tomosynthesis (DBT) imaging. Algebraic reconstruction technique (ART) has been in use in DBT imaging by minimizing the isotropic total variation (TV) of the reconstructed image. The resolution in DBT differs in sagittal and axial directions which should be encountered during the TV minimization. In this study we develop a 3D anisotropic TV (ATV) minimization by considering the different resolutions in different directions. A customized 3D Shepp-logan phantom was generated to mimic a real DBT image by considering the overlapping tissue and directional resolution issues. Results of the ART, ART+3D TV and ART+3D ATV are compared using structural similarity (SSIM) diagram.

We present an object-oriented simulator for 3D digital breast tomosynthesis (DBT) system using C++ programming language. The simulator is capable of implementing different iterative reconstruction and total variation (TV) regularization methods on the real world phantom models. A user friendly graphical user interface (GUI) helps users to select and run the desired methods on the designed phantom models. Results from a phantom study that mimics a real tomosynthesis problem, including the images of the layer of the interest (LOI) and structural similarity (SSIM) diagrams are presented.

Digital breast tomosynthesis (DBT) is a promising imaging modality that provides a 3D reconstructed object from 2D projections acquired over a limited angular range. The diagnostic quality of tomosynthesis depends on multiple variables, each of which needs to be optimized. This study investigates the effects of number of angular projections and the total angular span of those projections on the reconstructed image quality of breast tomosynthesis system. 5 different acquisition sets are used considering the available DBT systems.

In tomosynthesis imaging, out-of-focus slice blur problem arises due to incomplete sampling problem. Several approaches have been proposed to deal with this problem. Algebraic reconstruction technique (ART) is one of the most commonly used methods. Total variation (TV) minimization has recently been applied to improve performance of the classical approaches. Though it is able to provide improved results, its sensitivity to the regularization parameter is still an important issue. Former studies addressed largely 2-D tomosynthesis image reconstruction problem. In this study, a 3-D phantom model was used to understand the effect of total variation minimization on a 3-D image reconstruction problem. The significance of selecting an appropriate regularization parameter of TV not addressed in the prior studies was also investigated by means of comparing root mean square error (RMSE) and contrast to noise ratio (CNR) values.