Search

Recent proposals of deep learning-based beamformers for ultrasound imaging (US) have attracted significant attention as computational efficient alternatives to adaptive and compressive beamformers. Moreover, deep beamformers are versatile in that image post-processing algorithms can be readily combined. Unfortunately, with the existing technology, a large number of beamformers need to be trained and stored for different probes, organs, depth ranges, operating frequency, and desired target 'styles', demanding significant resources such as training data, etc. To address this problem, here we propose a switchable and tunable deep beamformer that can switch between various types of outputs such as DAS, MVBF, DMAS, GCF, etc., and also adjust noise removal levels at the inference phase, by using a simple switch or tunable nozzle. This novel mechanism is implemented through Adaptive Instance Normalization (AdaIN) layers, so that distinct outputs can be generated using a single generator by merely changing the AdaIN codes. Experimental results using B-mode focused ultrasound confirm the flexibility and efficacy of the proposed method for various applications.

Metal artifact reduction (MAR) is one of the most important research topics in computed tomography (CT). With the advance of deep learning approaches for image reconstruction, various deep learning methods have been suggested for metal artifact reduction, among which supervised learning methods are most popular. However, matched metal-artifact-free and metal artifact corrupted image pairs are difficult to obtain in real CT acquisition. Recently, a promising unsupervised learning for MAR was proposed using feature disentanglement, but the resulting network architecture is so complicated that it is difficult to handle large size clinical images. To address this, here we propose a simple and effective unsupervised learning method for MAR. The proposed method is based on a novel β -cycleGAN architecture derived from the optimal transport theory for appropriate feature space disentanglement. Moreover, by adding the convolutional block attention module (CBAM) layers in the generator, we show that the metal artifacts can be more focused so that it can be effectively removed. Experimental results confirm that we can achieve improved metal artifact reduction that preserves the detailed texture of the original image.

Plaque erosion, a distinct histopathological and clinical entity, accounts for over 30% of acute coronary syndromes (ACS). Optical coherence tomography allows in vivo diagnosis of plaque erosion. Local flow perturbation with activation of Toll-like receptor 2 and CD8+ T cells and subsequent desquamation of endothelium and neutrophil extracellular trap formation contribute to mechanisms of plaque erosion. Compared with ACS patients with plaque rupture, those with plaque erosion are younger, have fewer traditional cardiovascular risk factors, have lower plaque burden, and are more likely to present with non-ST-segment elevation ACS. Early evidence suggests that in patients with ACS caused by plaque erosion, antithrombotic therapy without stenting may be a safe and effective option. Future randomized trials are needed to validate these findings. Clinical studies to develop noninvasive point-of-care biomarkers that distinguish plaque rupture from erosion, and to test novel therapies that target molecular pathways involved in plaque erosion are needed.

Multispectral satellite imaging sensors acquire various spectral band images and have a unique spectroscopic property in each band. Unfortunately, image artifacts from imaging sensor noise often affect the quality of scenes and have a negative impact on applications for satellite imagery. Recently, deep learning approaches have been extensively explored to remove noise in satellite imagery. Most deep learning denoising methods, however, follow a supervised learning scheme, which requires matched noisy image and clean image pairs that are difficult to collect in real situations. In this article, we propose a novel unsupervised multispectral denoising method for satellite imagery using a wavelet directional cycle-consistent adversarial network (WavCycleGAN). The proposed method is based on an unsupervised learning scheme using adversarial loss and cycle-consistency loss to overcome the lack of paired data. Moreover, in contrast to the standard image-domain cycleGAN, we introduce a wavelet directional learning scheme for effective denoising without sacrificing high-frequency components such as edges and detailed information. Experimental results for the removal of vertical stripes and wave noise in satellite imaging sensors demonstrate that the proposed method effectively removes noise and preserves important high-frequency features of satellite images.

Recently, deep learning approaches have been successfully used for ultrasound (US) image artifact removal. However, paired high-quality images for supervised training are difficult to obtain in many practical situations. Inspired by the recent theory of unsupervised learning using optimal transport driven CycleGAN (OT-CycleGAN), here, we investigate the applicability of unsupervised deep learning for US artifact removal problems without matched reference data. Two types of OT-CycleGAN approaches are employed: one with the partial knowledge of the image degradation physics and the other with the lack of such knowledge. Various US artifact removal problems are then addressed using the two types of OT-CycleGAN. Experimental results for various unsupervised US artifact removal tasks confirmed that our unsupervised learning method delivers results comparable to supervised learning in many practical applications.

Optical diffraction tomography measures the three-dimensional refractive index map of a specimen and visualizes biochemical phenomena at the nanoscale in a non-destructive manner. One major drawback of optical diffraction tomography is poor axial resolution due to limited access to the three-dimensional optical transfer function. This missing cone problem has been addressed through regularization algorithms that use a priori information, such as non-negativity and sample smoothness. However, the iterative nature of these algorithms and their parameter dependency make real-time visualization impossible. In this article, we propose and experimentally demonstrate a deep neural network, which we term DeepRegularizer, that rapidly improves the resolution of a three-dimensional refractive index map. Trained with pairs of datasets (a raw refractive index tomogram and a resolution-enhanced refractive index tomogram via the iterative total variation algorithm), the three-dimensional U-net-based convolutional neural network learns a transformation between the two tomogram domains. The feasibility and generalizability of our network are demonstrated using bacterial cells and a human leukaemic cell line, and by validating the model across different samples. DeepRegularizer offers more than an order of magnitude faster regularization performance compared to the conventional iterative method. We envision that the proposed data-driven approach can bypass the high time complexity of various image reconstructions in other imaging modalities.

(1) To compare low-contrast detectability of a deep learning–based denoising algorithm (DLA) with ADMIRE and FBP, and (2) to compare image quality parameters of DLA with those of reconstruction methods from two different CT vendors (ADMIRE, IMR, and FBP). Using abdominal CT images of 100 patients reconstructed via ADMIRE and FBP, we trained DLA by feeding FBP images as input and ADMIRE images as the ground truth. To measure the low-contrast detectability, the randomized repeat scans of Catphan® phantom were performed under various conditions of radiation exposures. Twelve radiologists evaluated the presence/absence of a target on a five-point confidence scale. The multi-reader multi-case area under the receiver operating characteristic curve (AUC) was calculated, and non-inferiority tests were performed. Using American College of Radiology CT accreditation phantom, contrast-to-noise ratio, target transfer function, noise magnitude, and detectability index (d’) of DLA, ADMIRE, IMR, and FBPs were computed. The AUC of DLA in low-contrast detectability was non-inferior to that of ADMIRE (p < .001) and superior to that of FBP (p < .001). DLA improved the image quality in terms of all physical measurements compared to FBPs from both CT vendors and showed profiles of physical measurements similar to those of ADMIRE. The low-contrast detectability of the proposed deep learning–based denoising algorithm was non-inferior to that of ADMIRE and superior to that of FBP. The DLA could successfully improve image quality compared with FBP while showing the similar physical profiles of ADMIRE. • Low-contrast detectability in the images denoised using the deep learning algorithm was non-inferior to that in the images reconstructed using standard algorithms. • The proposed deep learning algorithm showed similar profiles of physical measurements to advanced iterative reconstruction algorithm (ADMIRE).

Energy-dispersive X-ray spectroscopy (EDX) is often performed simultaneously with high-angle annular dark-field scanning transmission electron microscopy (STEM) for nanoscale physico-chemical analysis. However, high-quality STEM-EDX tomographic imaging is still challenging due to fundamental limitations such as sample degradation with prolonged scan time and the low probability of X-ray generation. To address this, we propose an unsupervised deep learning method for high-quality 3D EDX tomography of core–shell nanocrystals, which can be usually permanently dammaged by prolonged electron beam. The proposed deep learning STEM-EDX tomography method was used to accurately reconstruct Au nanoparticles and InP/ZnSe/ZnS core–shell quantum dots, used in commercial display devices. Furthermore, the shape and thickness uniformity of the reconstructed ZnSe/ZnS shell closely correlates with optical properties of the quantum dots, such as quantum efficiency and chemical stability. Advanced electron microscopy and spectroscopy techniques can reveal useful structural and chemical details at the nanoscale. An unsupervised deep learning approach helps to reconstruct 3D images and observe the relationship between optical and structural properties of semiconductor nanocrystals, of interest in optoelectronic applications.

Time-resolved MR angiography (tMRA) has been widely used for dynamic contrast enhanced MRI (DCE-MRI) due to its highly accelerated acquisition. In tMRA, the periphery of the ${\textit k}$ -space data are sparsely sampled so that neighbouring frames can be merged to construct one temporal frame. However, this view-sharing scheme fundamentally limits the temporal resolution, and it is not possible to change the view-sharing number to achieve different spatio-temporal resolution trade-offs. Although many deep learning approaches have been recently proposed for MR reconstruction from sparse samples, the existing approaches usually require matched fully sampled ${\textit k}$ -space reference data for supervised training, which is not suitable for tMRA due to the lack of high spatio-temporal resolution ground-truth images. To address this problem, here we propose a novel unpaired training scheme for deep learning using optimal transport driven cycle-consistent generative adversarial network (cycleGAN). In contrast to the conventional cycleGAN with two pairs of generator and discriminator, the new architecture requires just a single pair of generator and discriminator, which makes the training much simpler but still improves the performance. Reconstruction results using in vivo tMRA and simulation data set confirm that the proposed method can immediately generate high quality reconstruction results at various choices of view-sharing numbers, allowing us to exploit better trade-off between spatial and temporal resolution in time-resolved MR angiography.

Optical diffraction tomography (ODT) produces a three-dimensional distribution of the refractive index (RI) by measuring scattering fields at various angles. Although the distribution of the RI is highly informative, due to the missing cone problem stemming from the limited-angle acquisition of holograms, reconstructions have very poor resolution along the axial direction compared to the horizontal imaging plane. To solve this issue, we present a novel unsupervised deep learning framework that learns the probability distribution of missing projection views through an optimal transport-driven CycleGAN. The experimental results show that missing cone artifacts in ODT data can be significantly resolved by the proposed method.

Deep-learning-based tomographic imaging is an important application of artificial intelligence and a new frontier of machine learning. Deep learning has been widely used in computer vision and image analysis, which deal with existing images, improve these images, and produce features from them. Since 2016, deep learning techniques have been actively researched for tomographic imaging, especially in the context of biomedicine, with impressive results and great potential. Tomographic reconstruction produces images of multi-dimensional structures from externally measured ‘encoded’ data in the form of various tomographic transforms (integrals, harmonics, echoes and so on). In this Review, we provide a general background, highlight representative results with an emphasis on medical imaging, and discuss key issues that need to be addressed in this emerging field. In particular, tomographic imaging is an integral part of modern medicine, and will play a key role in personalized, preventive and precision medicine and make it intelligent, inexpensive and indiscriminate. The popularity of deep learning is leading to new areas in biomedical applications. Wang and colleagues summarize in this Review the recent development and future directions of deep neural networks for superior image quality in the tomographic imaging field.

Conebeam CT using a circular trajectory is quite often used for various applications due to its relative simple geometry. For conebeam geometry, Feldkamp, Davis and Kress algorithm is regarded as the standard reconstruction method, but this algorithm suffers from so-called conebeam artifacts as the cone angle increases. Various model-based iterative reconstruction methods have been developed to reduce the cone-beam artifacts, but these algorithms usually require multiple applications of computational expensive forward and backprojections. In this paper, we develop a novel deep learning approach for accurate conebeam artifact removal. In particular, our deep network, designed on the differentiated backprojection domain, performs a data-driven inversion of an ill-posed deconvolution problem associated with the Hilbert transform. The reconstruction results along the coronal and sagittal directions are then combined using a spectral blending technique to minimize the spectral leakage. Experimental results show that our method outperforms the existing iterative methods despite significantly reduced runtime complexity., Comment: This paper is accepted for IEEE Trans. Medical Imaging

The nineteen papers in this special section focus on domain enriched learning for medical imaging. In recent years, learning based methods have emerged to complement traditional model and feature based methods for a variety of medical imaging problems such as image formation, classification and segmentation, quality enhancement etc. In the case of deep neural networks, many solutions have achieved unprecedented performance gains and have defined a new state of the art. Despite the progress, compelling open challenges remain. One such key challenge is that many learning frameworks (notably deep learning) are purely data-driven approaches and their performance depends strongly on the quantity and quality of training image data available. When training is limited or noisy, the performance drops sharply. Deep neural networks based approaches additionally face the challenge of often not being straightforward to interpret. Fortunately, exciting recent progress has emerged in enriching learning frameworks with domain knowledge and signal structure. As a couple of representative examples: in image reconstruction problems, this may involve using statistical/structural image priors; for image segmentation, shape and anatomical knowledge (conveyed by an expert) may be leveraged, etc. This special issue brings together contributions that combine signal, image priors and other flavors of domain knowledge with machine learning methods for solving many diverse medical imaging problems.

In ultrasound (US) imaging, various types of adaptive beamforming techniques have been investigated to improve the resolution and contrast-to-noise ratio of the delay and sum (DAS) beamformers. Unfortunately, the performance of these adaptive beamforming approaches degrade when the underlying model is not sufficiently accurate and the number of channels decreases. To address this problem, here we propose a deep learning-based beamformer to generate significantly improved images over widely varying measurement conditions and channel subsampling patterns. In particular, our deep neural network is designed to directly process full or sub-sampled radio-frequency (RF) data acquired at various subsampling rates and detector configurations so that it can generate high quality ultrasound images using a single beamformer. The origin of such input-dependent adaptivity is also theoretically analyzed. Experimental results using B-mode focused ultrasound confirm the efficacy of the proposed methods., This is a significantly extended version of the original paper in arXiv:1901.01706. This paper is accepted for IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control

Under the global pandemic of COVID-19, the use of artificial intelligence to analyze chest X-ray (CXR) image for COVID-19 diagnosis and patient triage is becoming important. Unfortunately, due to the emergent nature of the COVID-19 pandemic, a systematic collection of the CXR data set for deep neural network training is difficult. To address this problem, here we propose a patch-based convolutional neural network approach with a relatively small number of trainable parameters for COVID-19 diagnosis. The proposed method is inspired by our statistical analysis of the potential imaging biomarkers of the CXR radiographs. Experimental results show that our method achieves state-of-the-art performance and provides clinically interpretable saliency maps, which are useful for COVID-19 diagnosis and patient triage., Accepted for IEEE Trans. on Medical Imaging Special Issue on Imaging-based Diagnosis of COVID-19

A unique advantage of magnetic resonance imaging (MRI) is its mechanism for generating various image contrasts depending on tissue-specific parameters, which provides useful clinical information. Unfortunately, a complete set of MR contrasts is often difficult to obtain in a real clinical environment. Recently, there have been claims that generative models such as generative adversarial networks (GANs) can synthesize MR contrasts that are not acquired. However, the poor scalability of existing GAN-based image synthesis poses a fundamental challenge to understanding the nature of MR contrasts: which contrasts matter, and which cannot be synthesized by generative models? Here, we show that these questions can be addressed systematically by learning the joint manifold of multiple MR contrasts using collaborative generative adversarial networks. Our experimental results show that the exogenous contrast provided by contrast agents is not replaceable, but endogenous contrasts such as T1 and T2 can be synthesized from other contrasts. These findings provide important guidance for the acquisition-protocol design of MR in clinical environments. Magnetic resonance scans use different contrast agents to generate different images, each giving specific clinical information. Lee et al. use a collaborative generative model to synthesize some magnetic resonance contrasts from others, providing guidance for how clinical imaging times can be reduced.

Recent state-of-the-art image segmentation algorithms are mostly based on deep neural networks, thanks to their high performance and fast computation time. However, these methods are usually trained in a supervised manner, which requires large number of high quality ground-truth segmentation masks. On the other hand, classical image segmentation approaches such as level-set methods are formulated in a self-supervised manner by minimizing energy functions such as Mumford-Shah functional, so they are still useful to help generation of segmentation masks without labels. Unfortunately, these algorithms are usually computationally expensive and often have limitation in semantic segmentation. In this paper, we propose a novel loss function based on Mumford-Shah functional that can be used in deep-learning based image segmentation without or with small labeled data. This loss function is based on the observation that the softmax layer of deep neural networks has striking similarity to the characteristic function in the Mumford-Shah functional. We show that the new loss function enables semi-supervised and unsupervised segmentation. In addition, our loss function can be also used as a regularized function to enhance supervised semantic segmentation algorithms. Experimental results on multiple datasets demonstrate the effectiveness of the proposed method., Comment: Accepted for IEEE Transactions on Image Processing

To improve the performance of classical generative adversarial network (GAN), Wasserstein generative adversarial networks (W-GAN) was developed as a Kantorovich dual formulation of the optimal transport (OT) problem using Wasserstein-1 distance. However, it was not clear how cycleGAN-type generative models can be derived from the optimal transport theory. Here we show that a novel cycleGAN architecture can be derived as a Kantorovich dual OT formulation if a penalized least square (PLS) cost with deep learning-based inverse path penalty is used as a transportation cost. One of the most important advantages of this formulation is that depending on the knowledge of the forward problem, distinct variations of cycleGAN architecture can be derived: for example, one with two pairs of generators and discriminators, and the other with only a single pair of generator and discriminator. Even for the two generator cases, we show that the structural knowledge of the forward operator can lead to a simpler generator architecture which significantly simplifies the neural network training. The new cycleGAN formulation, what we call the OT-cycleGAN, have been applied for various biomedical imaging problems, such as accelerated magnetic resonance imaging (MRI), super-resolution microscopy, and low-dose x-ray computed tomography (CT). Experimental results confirm the efficacy and flexibility of the theory., Comment: accepted for publication in the SIAM Journal on Imaging Sciences

Recently, deep learning approaches for MR motion artifact correction have been extensively studied. Although these approaches have shown high performance and lower computational complexity compared to classical methods, most of them require supervised training using paired artifact-free and artifact-corrupted images, which may prohibit its use in many important clinical applications. For example, transient severe motion (TSM) due to acute transient dyspnea in Gd-EOB-DTPA-enhanced MR is difficult to control and model for paired data generation. To address this issue, here we propose a novel unpaired deep learning scheme that does not require matched motion-free and motion artifact images. Specifically, the first step of our method is ${k}$ -space random subsampling along the phase encoding direction that can remove some outliers probabilistically. In the second step, the neural network reconstructs fully sampled resolution image from a downsampled ${k}$ -space data, and motion artifacts can be reduced in this step. Last, the aggregation step through averaging can further improve the results from the reconstruction network. We verify that our method can be applied for artifact correction from simulated motion as well as real motion from TSM successfully from both single and multi-coil data with and without ${k}$ -space raw data, outperforming existing state-of-the-art deep learning methods.

Developing a robust algorithm to diagnose and quantify the severity of the novel coronavirus disease 2019 (COVID-19) using Chest X-ray (CXR) requires a large number of well-curated COVID-19 datasets, which is difficult to collect under the global COVID-19 pandemic. On the other hand, CXR data with other findings are abundant. This situation is ideally suited for the Vision Transformer (ViT) architecture, where a lot of unlabeled data can be used through structural modeling by the self-attention mechanism. However, the use of existing ViT may not be optimal, as the feature embedding by direct patch flattening or ResNet backbone in the standard ViT is not intended for CXR. To address this problem, here we propose a novel Multi-task ViT that leverages low-level CXR feature corpus obtained from a backbone network that extracts common CXR findings. Specifically, the backbone network is first trained with large public datasets to detect common abnormal findings such as consolidation, opacity, edema, etc. Then, the embedded features from the backbone network are used as corpora for a versatile Transformer model for both the diagnosis and the severity quantification of COVID-19. We evaluate our model on various external test datasets from totally different institutions to evaluate the generalization capability. The experimental results confirm that our model can achieve state-of-the-art performance in both diagnosis and severity quantification tasks with outstanding generalization capability, which are sine qua non of widespread deployment., Graphical abstract Image, graphical abstract

In ultrasound (US) imaging, various adaptive beamforming methods have been proposed to improve the resolution and contrast-to-noise ratio of the delay and sum (DAS) beamformers. Unfortunately, they often require computationally expensive calculations and their performance degrades when the underlying model is not sufficiently accurate. Moreover, ultrasound images usually require various type of post filtration such as deblurring and despeckling, etc., which further increase the complexity of the system. Deep learning-based solutions provides a quick remedy to these issue; however, in the current technology, a separate beamformer should be trained and stored for each application, demanding significant scanner resources. To address this problem, here we propose a switchable deep beamformer that can produce various types of output such as DAS, speckle removal, deconvolution, etc., using a single network with a simple switch. In particular, the switch is implemented through Adaptive Instance Normalization (AdaIN) layers, so that distinct outputs can be generated by merely changing the AdaIN code. Experimental results using B-mode focused ultrasound confirm the efficacy of the proposed methods.

Echocardiography is a pivotal imaging tool for emergency medicine. Unfortunately, it suffers from poor image quality due to the intrinsic limitations of sonography systems. Towards this end, a better quality can be achieved at the cost of reduced frame rate by increasing the number of transmit/receive events and utilizing computationally expensive noise suppression algorithms. However, this visual quality and temporal resolution trade-off is a bottleneck for many echocardiography applications. Conventional acceleration methods, such as multi-line acquisition (MLA), work only for limited acceleration factors and produce blocking artifacts at a high frame rate. Accordingly, various machine learning algorithms have been designed to reduce blocking artifacts in MLA. These algorithms require access to either high-quality raw RF data or time-delayed baseband IQ data. Unfortunately, in many lower-end commercial systems, such data are not accessible. On the other hand, ultrasound images are badly affected by speckle noises which significantly reduces the image quality. We propose an image domain unsupervised deep learning framework using cycleGAN architecture for high quality accelerated echocardiography that simultaneously reduces the blocking artifacts and the speckle noise. The method is evaluated on real in-vivo and phantom data and achieves notable performance gain.

The articles in this special section focus on computational magnetic resonance imaging (MRI) using compressed sensing applications. Presents recent developments in computational MRI. These developments are pushing the frontier of computational imaging beyond CS. Similar to CS, most of these algorithms rely on image representation in one form or another. However, the common recent thread is the departure from handcrafted image representations to learning-based image representations. These learned representations are seamlessly combined with clever measurement strategies to significantly advance the state of the art in a number of areas. Several exciting applications including significantly improved spatial and temporal resolution, a considerable reduction in scan time, measurement of biophysical parameters directly from highly undersampled data, and direct measurement of very high-dimensional data are reviewed in this special issue of SPM. This issue describes key ideas underlying the computational approaches used in MRI. These approaches range from CS algorithms that rely on fixed transforms or dictionaries, to adaptive or shallow-learning algorithms that adapt the image representation to the data to recent deep-learning methods that learn a highly nonlinear representation from exemplar data. The articles provide insight into the capabilities of the current algorithms, their limitations, and their utility in challenging MRI problems.

Purpose Nyquist ghost artifacts in echo planar imaging (EPI) are originated from phase mismatch between the even and odd echoes. However, conventional correction methods using reference scans often produce erroneous results especially in high-field MRI due to the nonlinear and time-varying local magnetic field changes. Recently, it was shown that the problem of ghost correction can be reformulated as k-space interpolation problem that can be solved using structured low-rank Hankel matrix approaches. Another recent work showed that data driven Hankel matrix decomposition can be reformulated to exhibit similar structures as deep convolutional neural network. By synergistically combining these findings, we propose a k-space deep learning approach that immediately corrects the phase mismatch without a reference scan in both accelerated and non-accelerated EPI acquisitions. Theory and methods To take advantage of the even and odd-phase directional redundancy, the k-space data are divided into 2 channels configured with even and odd phase encodings. The redundancies between coils are also exploited by stacking the multi-coil k-space data into additional input channels. Then, our k-space ghost correction network is trained to learn the interpolation kernel to estimate the missing virtual k-space data. For the accelerated EPI data, the same neural network is trained to directly estimate the interpolation kernels for missing k-space data from both ghost and subsampling. Results Reconstruction results using 3T and 7T in vivo data showed that the proposed method outperformed the image quality compared to the existing methods, and the computing time is much faster. Conclusions The proposed k-space deep learning for EPI ghost correction is highly robust and fast, and can be combined with acceleration, so that it can be used as a promising correction tool for high-field MRI without changing the current acquisition protocol.

In portable, three dimensional, and ultra-fast ultrasound imaging systems, there is an increasing demand for the reconstruction of high quality images from a limited number of radio-frequency (RF) measurements due to receiver (Rx) or transmit (Xmit) event sub-sampling. However, due to the presence of side lobe artifacts from RF sub-sampling, the standard beamformer often produces blurry images with less contrast, which are unsuitable for diagnostic purposes. Existing compressed sensing approaches often require either hardware changes or computationally expensive algorithms, but their quality improvements are limited. To address this problem, here we propose a novel deep learning approach that directly interpolates the missing RF data by utilizing redundancy in the Rx-Xmit plane. Our extensive experimental results using sub-sampled RF data from a multi-line acquisition B-mode system confirm that the proposed method can effectively reduce the data rate without sacrificing image quality., The title has been changed. This version will appear in IEEE Trans. on Medical Imaging

This editorial accompanies the launch of BMC Biomedical Engineering, a new open access, peer-reviewed journal within the BMC series, which seeks to publish articles on all aspects of biomedical engineering. As one of the first engineering journals within the BMC series portfolio, it will support and complement existing biomedical communities, but at the same time, it will provide an open access home for engineering research. By publishing original research, methodology, database, software and review articles, BMC Biomedical Engineering will disseminate quality research, with a focus on studies that further the understanding of human disease and that contribute towards the improvement of human health.

Background and Aims Endoscopic differential diagnoses of gastric mucosal lesions (benign gastric ulcer, early gastric cancer [EGC], and advanced gastric cancer) remain challenging. We aimed to develop and validate convolutional neural network-based artificial intelligence (AI) models: lesion detection (AI-LD), differential diagnosis (AI-DDx), and invasion-depth (AI-ID, pT1a vs. pT1b among EGC) models. Methods This study included 1,366 consecutive patients with gastric mucosal lesions from 2 referral centers in Korea. One representative endoscopic image from each patient was used. Histological diagnoses were set as the criterion standard. The performances of the AI-DDx (training/internal/external validation set, n=1009/112/245) and AI-ID (training/internal/external validation set, n=620/68/155) were compared with visual diagnoses by independent endoscopists (stratified by novice [ 5 years of experience]) and endoscopic ultrasonography (EUS) results, respectively. Results The AI-DDx showed good diagnostic performance for both internal (area under of the receiver operating characteristic curve [AUROC]=0.86) and external validation (AUROC=0.86). The performance of the AI-DDx was better than that of the novice (AUROC=0.82, P=0.01) and intermediate endoscopists (AUROC=0.84, P=0.02), but was comparable to the experts (AUROC=0.89, P=0.12) in the external validation set. The AI-ID showed fair performances in both internal (AUROC=0.78) and external validation sets (AUROC=0.73), which were significantly better than EUS results performed by experts (internal validation: AUROC=0.62, external validation: AUROC=0.56; both P Conclusion The AI-DDx was comparable with experts and outperformed novice and intermediate endoscopists for the differential diagnosis of gastric mucosal lesion. The AI-ID performed better than EUS for the invasion-depth evaluation ( https://aiscopeseoul.com/ ).

Point-of-Care Ultrasound (POCUS) imaging can help efficient resource utilization by reducing the secondary care referrals, and work as an extension in physical examination. Recently, many methods were proposed to reduce the size and power consumption of the system while improving the visual quality, but hand-held POCUS devices still have inferior image contrast and spatial resolution compared to the high-end ultrasound systems. To address this, here we propose an efficient solution for contrast and resolution enhancement of hand-held POCUS images using unsupervised deep learning. In contrast to the existing CycleGAN approaches that have difficulty in improving both contrast and image resolutions, the proposed method mitigate the problem by decomposing the contrast transfer and resolution improvement through CycleGAN and self-supervised learning. Experimental results confirmed that our method is superior than the conventional approaches.

Background: Endoscopic differential diagnoses of gastric mucosal lesions (benign gastric ulcer, early gastric cancer [EGC], and advanced gastric cancer) remain challenging. We aimed to develop and validate convolutional neural network-based artificial intelligence (AI) models: lesion detection (AI-LD), differential diagnosis (AI-DDx), and invasion-depth (AI-ID, pT1a vs. pT1b among EGC) models. Methods: This study included 1,366 consecutive patients with gastric mucosal lesions from two referral centers in Korea. One representative endoscopic image from each patient was used. Histological diagnoses were set as the gold standard. The performances of the AI-DDx (training/internal/external validation set, n=1009/112/245) and AI-ID (training/internal/external validation set, n=620/68/155) were compared with visual diagnoses by independent endoscopists (stratified by novice [ 5 years of experience]) in a prospective manner and by endoscopic ultrasonography (EUS), respectively. Findings: The AI-DDx showed good diagnostic performance for both internal (area under of the receiver operating characteristic curve [AUROC]=0.86) and external validation (AUROC=0.86). The performance of the AI-DDx was better than that of the novice (AUROC=0.82, P=0.01) and intermediate endoscopists (AUROC=0.84, P=0.02), but was comparable to the experts (AUROC=0.89, P=0.12) in the external validation set. The AI-ID showed fair performances in both internal (AUROC=0.78) and external validation sets (AUROC=0.73), which were significantly better than EUS results performed by experts (internal validation: AUROC=0.62, external validation: AUROC=0.56; both P

In electrocardiography (ECG) gated cardiac CT angiography (CCTA), multiple images covering the entire cardiac cycle are taken continuously, so reduction of the accumulated radiation dose could be an important issue for patient safety. Although ECG-gated dose modulation (so-called ECG pulsing) is used to acquire many phases of CT images at a low dose, the reduction of the radiation dose introduces noise into the image reconstruction. To address this, we developed a high performance unsupervised deep learning method using noise disentanglement that can effectively learn the noise patterns even from extreme low dose CT images. For noise disentanglement, we use a wavelet transform to extract the high-frequency signals that contain the most noise. Since matched low-dose and high-dose cardiac CT data are impossible to obtain in practice, our neural network was trained in an unsupervised manner using cycleGAN for the extracted high frequency signals from the low-dose and unpaired high-dose CT images. Once the network is trained, denoised images are obtained by subtracting the estimated noise components from the input images. Image quality evaluation of the denoised images from only 4% dose CT images was performed by experienced radiologists for several anatomical structures. Visual grading analysis was conducted according to the sharpness level, noise level, and structural visibility. Also, the signal-to-noise ratio was calculated. The evaluation results showed that the quality of the images produced by the proposed method is much improved compared to low-dose CT images and to the baseline cycleGAN results. The proposed noise-disentangled cycleGAN with wavelet transform effectively removed noise from extreme low-dose CT images compared to the existing baseline algorithms. It can be an important denoising platform for low-dose CT.

In recent years, artificial intelligence (AI) technologies have greatly advanced and become a reality in many areas of our daily lives. In the health care field, numerous efforts are being made to implement the AI technology for practical medical treatments. With the rapid developments in machine learning algorithms and improvements in hardware performances, the AI technology is expected to play an important role in effectively analyzing and utilizing extensive amounts of health and medical data. However, the AI technology has various unique characteristics that are different from the existing health care technologies. Subsequently, there are a number of areas that need to be supplemented within the current health care system for the AI to be utilized more effectively and frequently in health care. In addition, the number of medical practitioners and public that accept AI in the health care is still low; moreover, there are various concerns regarding the safety and reliability of AI technology implementations. Therefore, this paper aims to introduce the current research and application status of AI technology in health care and discuss the issues that need to be resolved., Graphical Abstract

High quality US imaging demand large number of measurements that can increase the cost, size and power requirements. Therefore, low-powered, portable and 3D ultrasound imaging system require reconstruction algorithms that can produce high quality images using fewer receive measurements. Number of model specific methods has been proposed which doesn't work under perturbation. For instance, compressive deconvolution ultrasound which provide a reasonable quality with limited measurements however, it has its own down-sides such as high computation cost and accurate estimation of point spread function (PSF). An other major limitation of conventional methods is that they require RF or base-band signal which is difficult to obtain from portable US systems. To deal with the aforementioned issues, in this study we designed a novel deep deconvolution model for image domain-based deconvolution. The proposed deep deconvolution (DeepDeconv) model can be trained in an unsupervised fashion, alleviate the need of paired high and low quality images. The model was evaluated on both the phantom and in-vivo scans for various sampling configurations. The proposed DeepDeconv significantly enhance the details of anatomical structures and using unsupervised learning on average it achieved 2.14dB, 4.96dB and 0.01 units gain in CR, PSNR and SSIM values respectively, which are comparable to the supervised method.

Recently, deep learning approaches for accelerated MRI have been extensively studied thanks to their high performance reconstruction in spite of significantly reduced runtime complexity. These neural networks are usually trained in a supervised manner, so matched pairs of subsampled and fully sampled k-space data are required. Unfortunately, it is often difficult to acquire matched fully sampled k-space data, since the acquisition of fully sampled k-space data requires long scan time and often leads to the change of the acquisition protocol. Therefore, unpaired deep learning without matched label data has become a very important research topic. In this paper, we propose an unpaired deep learning approach using a optimal transport driven cycle-consistent generative adversarial network (OT-cycleGAN) that employs a single pair of generator and discriminator. The proposed OT-cycleGAN architecture is rigorously derived from a dual formulation of the optimal transport formulation using a specially designed penalized least squares cost. The experimental results show that our method can reconstruct high resolution MR images from accelerated k- space data from both single and multiple coil acquisition, without requiring matched reference data., Accepted for IEEE Transactions on Computational Imaging

Image registration is a fundamental task in medical image analysis. Recently, many deep learning based image registration methods have been extensively investigated due to their comparable performance with the state-of-the-art classical approaches despite the ultra-fast computational time. However, the existing deep learning methods still have limitations in the preservation of original topology during the deformation with registration vector fields. To address this issues, here we present a cycle-consistent deformable image registration, dubbed CycleMorph. The cycle consistency enhances image registration performance by providing an implicit regularization to preserve topology during the deformation. The proposed method is so flexible that it can be applied for both 2D and 3D registration problems for various applications, and can be easily extended to multi-scale implementation to deal with the memory issues in large volume registration. Experimental results on various datasets from medical and non-medical applications demonstrate that the proposed method provides effective and accurate registration on diverse image pairs within a few seconds. Qualitative and quantitative evaluations on deformation fields also verify the effectiveness of the cycle consistency of the proposed method.

Background Pharmacokinetic (PK) parameters obtained from dynamic contrast agent-enhanced (DCE) MRI evaluates the microcirculation permeability of astrocytomas, but the unreliability from arterial input function (AIF) remains a challenge. Purpose To develop a deep learning model that improves the reliability of AIF for DCE MRI and to validate the reliability and diagnostic performance of PK parameters by using improved AIF in grading astrocytomas. Materials and Methods This retrospective study included 386 patients (mean age, 52 years ± 16 [standard deviation]; 226 men) with astrocytomas diagnosed with histopathologic analysis who underwent dynamic susceptibility contrast (DSC)-enhanced and DCE MRI preoperatively from April 2010 to January 2018. The AIF was obtained from each sequence: AIF obtained from DSC-enhanced MRI (AIF

Time-of-flight magnetic resonance angiography (TOF-MRA) is one of the most widely used non-contrast MR imaging methods to visualize blood vessels, but due to the 3-D volume acquisition highly accelerated acquisition is necessary. Accordingly, high quality reconstruction from undersampled TOF-MRA is an important research topic for deep learning. However, most existing deep learning works require matched reference data for supervised training, which are often difficult to obtain. By extending the recent theoretical understanding of cycleGAN from the optimal transport theory, here we propose a novel two-stage unsupervised deep learning approach, which is composed of the multi-coil reconstruction network along the coronal plane followed by a multi-planar refinement network along the axial plane. Specifically, the first network is trained in the square-root of sum of squares (SSoS) domain to achieve high quality parallel image reconstruction, whereas the second refinement network is designed to efficiently learn the characteristics of highly-activated blood flow using double-headed projection discriminator. Extensive experiments demonstrate that the proposed learning process without matched reference exceeds performance of state-of-the-art compressed sensing (CS)-based method and provides comparable or even better results than supervised learning approaches.

Graph neural networks (GNN) rely on graph operations that include neural network training for various graph related tasks. Recently, several attempts have been made to apply the GNNs to functional magnetic resonance image (fMRI) data. Despite recent progresses, a common limitation is its difficulty to explain the classification results in a neuroscientifically explainable way. Here, we develop a framework for analyzing the fMRI data using the Graph Isomorphism Network (GIN), which was recently proposed as a powerful GNN for graph classification. One of the important contributions of this paper is the observation that the GIN is a dual representation of convolutional neural network (CNN) in the graph space where the shift operation is defined using the adjacency matrix. This understanding enables us to exploit CNN-based saliency map techniques for the GNN, which we tailor to the proposed GIN with one-hot encoding, to visualize the important regions of the brain. We validate our proposed framework using large-scale resting-state fMRI (rs-fMRI) data for classifying the sex of the subject based on the graph structure of the brain. The experiment was consistent with our expectation such that the obtained saliency map show high correspondence with previous neuroimaging evidences related to sex differences., Comment: This paper is accepted for Frontiers in Neuroscience

Optimal transport (OT) is a mathematical theory that can provide a tool how to transfer one measure to another measure at minimal cost, thus serve another framework for computer vision tasks of image processing without reference. Cycleconsistent generative adversarial network (cycleGAN) is a recent extension of GAN to learn target distributions with less mode collapsing behavior, and also does not need matched data during training. In this article, we explain the link between these two framework by mathematical formula and experimental results. We prove that cycleGAN architecture can be derived from optimal transport problem, and this implies that cycleGAN is a plausible way to learn target distribution when it comes to handling data far from the training set. Using accelerated MR imaging experiments, we confirmed the flexibility and efficacy of our theoretical framework.

Recently, deep learning based approaches for accelerated MRI have been extensively studied due to its high performance and reduced run time complexity. The existing deep learning methods for accelerated MRI are mostly supervised methods, where matched subsampled k-space data and fully sampled k-space data are necessary. However, it is hard to acquire fully sampled k-space data because of long scan time of MRI. Therefore, unsupervised method without matched label data has become a very important research topic. In this paper, we propose an unsupervised method using a novel cycle-consistent generative adversarial network (cycleGAN) with a single deep generator. We show that the proposed cy-cleGAN architecture can be derived from a dual formulation of the optimal transport with the penalized least squares cost. The results of experiments show that our method can remove aliasing patterns in downsampled MR images without the matched reference data.

Cone-beam computed tomography (CBCT) used in radiotherapy (RT) has the advantage of being taken daily, but is difficult to use for purposes other than patient setup because of the poor image quality compared to fan-beam computed tomography (CT). Even though several methods have been proposed including the deformable image registration method to improve the quality of CBCT, the outcomes have not yet been satisfactory. Recently, deep learning has shown to produce high-quality results for various image-to-image translation tasks, suggesting the possibility of being an effective tool for converting CBCT into CT. In the field of RT, however, it may not always be possible to obtain paired datasets which consist of exactly matching CBCT and CT images. This study aimed to develop a novel, unsupervised deep-learning algorithm, which requires only unpaired CBCT and fan-beam CT images to remove the cone-beam artifact and thereby improve the quality of CBCT. Specifically, two cycle consistency generative adversarial networks (CycleGAN) were trained in the sagittal and coronal directions, and the generated results along those directions were then combined using spectral blending technique. To evaluate our methods, we applied it to American Association of Physicists in Medicine dataset. The experimental results show that our method outperforms the existing CycleGAN-based method both qualitatively and quantitatively.

Magnetic resonance imaging (MRI) acquisition is an inherently slow process whose acceleration has been the subject of much investigation. In recent years, the explosive advance of deep learning techniques for computer vision and image reconstruction has led to the investigation of deep neural networks for the reconstruction of MRI with under-sampled k-space. In this work, we propose a new image domain architecture that directly produces a sum-of-squares image from under-sampled multi-coil MRI acquisition. This model, called BarbellNet, is a fully convolutional neural network architecture that utilizes the channel attention mechanism using the residual channel attention block (RCAB). Through extensive experiments with the fastMRI data set, we confirm the efficacy of BarbellNet.

For circular trajectory conebeam CT, Feldkamp, Davis, and Kress (FDK) algorithm is widely used for its reconstruction. However, the existence of cone-angle artifacts is fatal for the quality when using this algorithm. There are several model-based iterative reconstruction methods for the cone-angle artifacts removal, but these algorithms usually require repeated applications of computational expensive forward and backward. In this paper, we propose a novel deep learning approach for cone-angle artifact removal on differentiated backprojection domain, which performs a data-driven inversion of an ill-posed deconvolution problem related to the Hilbert transform. The reconstruction results along the coronal and sagittal directions are then combined by a spectral blending technique to minimize the spectral leakage. Experimental results show that our method provides superior performance to the existing methods.

Diffuse optical tomography (DOT) non-invasively measures the functional characteristics of breast lesions using near infrared light to probe tissue optical properties. This study aimed to evaluate a new digital breast tomosynthesis (DBT)/DOT fusion imaging technique and obtain preliminary data for breast cancer detection. Twenty-eight women were prospectively enrolled and underwent both DBT and DOT examinations. DBT/DOT fusion imaging was created after acquisition of both examinations. Two breast radiologists analyzed DBT and DOT images independently, and then finally evaluated the fusion images. The diagnostic performance of each reading session was compared and interobserver agreement was assessed. The technical success rate was 96.4%, with one failure due to an error during DOT data storage. Among the 27 women finally included in the analysis, 13 had breast cancer. The areas under the receiver operating characteristic curve (AUCs) for DBT were 0.783 and 0.854 for readers 1 and 2, respectively. DOT showed comparable diagnostic performance to DBT for both readers. The AUCs were significantly improved (P = 0.004) when the DBT/DOT fusion images were used. Interobserver agreements were highest for the DBT/DOT fusion images. In conclusion, this study suggests that DBT/DOT fusion imaging technique appears to be a promising tool for breast cancer diagnosis.

This paper reviews the NTIRE 2020 challenge on perceptual extreme super-resolution with focus on proposed solutions and results. The challenge task was to super-resolve an input image with a magnification factor 16 based on a set of prior examples of low and corresponding high resolution images. The goal is to obtain a network design capable to produce high resolution results with the best perceptual quality and similar to the ground truth. The track had 280 registered participants, and 19 teams submitted the final results. They gauge the state-of-the-art in single image super-resolution., Comment: CVPRW 2020

This paper reviews the second AIM learned ISP challenge and provides the description of the proposed solutions and results. The participating teams were solving a real-world RAW-to-RGB mapping problem, where to goal was to map the original low-quality RAW images captured by the Huawei P20 device to the same photos obtained with the Canon 5D DSLR camera. The considered task embraced a number of complex computer vision subtasks, such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The target metric used in this challenge combined fidelity scores (PSNR and SSIM) with solutions’ perceptual results measured in a user study. The proposed solutions significantly improved the baseline results, defining the state-of-the-art for practical image signal processing pipeline modeling.

Recently, deep learning approaches have been extensively investigated to reconstruct images from accelerated magnetic resonance image (MRI) acquisition. Although these approaches provide significant performance gain compared to compressed sensing MRI (CS-MRI), it is not clear how to choose a suitable network architecture to balance the trade-off between network complexity and performance. Recently, it was shown that an encoder-decoder convolutional neural network (CNN) can be interpreted as a piecewise linear basis-like representation, whose specific representation is determined by the ReLU activation patterns for a given input image. Thus, the expressivity or the representation power is determined by the number of piecewise linear regions. As an extension of this geometric understanding, this paper proposes a systematic geometric approach using bootstrapping and subnetwork aggregation using an attention module to increase the expressivity of the underlying neural network. Our method can be implemented in both k-space domain and image domain that can be trained in an end-to-end manner. Experimental results show that the proposed schemes significantly improve reconstruction performance with negligible complexity increases., Comment: Accepted for IEEE JSTSP Special Issue on Domain Enriched Learning for Medical Imaging

Plane-wave compounding is to sum up several successive plane waves incident at different angles to form an image. By applying time-reversal of the received signals, transmit focusing can be synthesized. Unfortunately, to improve the temporal resolution, the number of plane waves should be reduced, which often degrades the image quality. To address this problem, an image domain learning method using neural networks has been proposed, but the network needs to be retrained when the number of plane waves changes. Herein, we propose, for the first time, a universal plane-wave compounding scheme using deep learning to directly process plane waves and RF data acquired at different view angles and sub-sampling rate to generate high quality US images.

Objective Parkinson's disease (PD) is a neurodegenerative disorder that mainly leads to the impairment of patients' motor function, as well as of cognition, as it progresses. This study tried to investigate the impact of PD on the resting state functional connectivity of the default mode network (DMN), as well as of the entire brain. Methods Sixty patients with PD were included and compared to 60 matched normal control (NC) subjects. For the local connectivity analysis, the resting state fMRI data were analyzed by seed-based correlation analyses, and then a novel persistent homology analysis was implemented to examine the connectivity from a global perspective. Results The functional connectivity of the DMN was decreased in the PD group compared to the NC, with a stronger difference in the medial prefrontal cortex. Moreover, the results of the persistent homology analysis indicated that the PD group had a more locally connected and less globally connected network compared to the NC. Conclusion Our findings suggest that the DMN is altered in PD, and persistent homology analysis, as a useful measure of the topological characteristics of the networks from a broader perspective, was able to identify changes in the large-scale functional organization of the patients' brain.