Search

Recent proposals of deep learning-based beamformers for ultrasound imaging (US) have attracted significant attention as computational efficient alternatives to adaptive and compressive beamformers. Moreover, deep beamformers are versatile in that image post-processing algorithms can be readily combined. Unfortunately, with the existing technology, a large number of beamformers need to be trained and stored for different probes, organs, depth ranges, operating frequency, and desired target 'styles', demanding significant resources such as training data, etc. To address this problem, here we propose a switchable and tunable deep beamformer that can switch between various types of outputs such as DAS, MVBF, DMAS, GCF, etc., and also adjust noise removal levels at the inference phase, by using a simple switch or tunable nozzle. This novel mechanism is implemented through Adaptive Instance Normalization (AdaIN) layers, so that distinct outputs can be generated using a single generator by merely changing the AdaIN codes. Experimental results using B-mode focused ultrasound confirm the flexibility and efficacy of the proposed method for various applications.

Metal artifact reduction (MAR) is one of the most important research topics in computed tomography (CT). With the advance of deep learning approaches for image reconstruction, various deep learning methods have been suggested for metal artifact reduction, among which supervised learning methods are most popular. However, matched metal-artifact-free and metal artifact corrupted image pairs are difficult to obtain in real CT acquisition. Recently, a promising unsupervised learning for MAR was proposed using feature disentanglement, but the resulting network architecture is so complicated that it is difficult to handle large size clinical images. To address this, here we propose a simple and effective unsupervised learning method for MAR. The proposed method is based on a novel β -cycleGAN architecture derived from the optimal transport theory for appropriate feature space disentanglement. Moreover, by adding the convolutional block attention module (CBAM) layers in the generator, we show that the metal artifacts can be more focused so that it can be effectively removed. Experimental results confirm that we can achieve improved metal artifact reduction that preserves the detailed texture of the original image.

Multispectral satellite imaging sensors acquire various spectral band images and have a unique spectroscopic property in each band. Unfortunately, image artifacts from imaging sensor noise often affect the quality of scenes and have a negative impact on applications for satellite imagery. Recently, deep learning approaches have been extensively explored to remove noise in satellite imagery. Most deep learning denoising methods, however, follow a supervised learning scheme, which requires matched noisy image and clean image pairs that are difficult to collect in real situations. In this article, we propose a novel unsupervised multispectral denoising method for satellite imagery using a wavelet directional cycle-consistent adversarial network (WavCycleGAN). The proposed method is based on an unsupervised learning scheme using adversarial loss and cycle-consistency loss to overcome the lack of paired data. Moreover, in contrast to the standard image-domain cycleGAN, we introduce a wavelet directional learning scheme for effective denoising without sacrificing high-frequency components such as edges and detailed information. Experimental results for the removal of vertical stripes and wave noise in satellite imaging sensors demonstrate that the proposed method effectively removes noise and preserves important high-frequency features of satellite images.

Optical diffraction tomography measures the three-dimensional refractive index map of a specimen and visualizes biochemical phenomena at the nanoscale in a non-destructive manner. One major drawback of optical diffraction tomography is poor axial resolution due to limited access to the three-dimensional optical transfer function. This missing cone problem has been addressed through regularization algorithms that use a priori information, such as non-negativity and sample smoothness. However, the iterative nature of these algorithms and their parameter dependency make real-time visualization impossible. In this article, we propose and experimentally demonstrate a deep neural network, which we term DeepRegularizer, that rapidly improves the resolution of a three-dimensional refractive index map. Trained with pairs of datasets (a raw refractive index tomogram and a resolution-enhanced refractive index tomogram via the iterative total variation algorithm), the three-dimensional U-net-based convolutional neural network learns a transformation between the two tomogram domains. The feasibility and generalizability of our network are demonstrated using bacterial cells and a human leukaemic cell line, and by validating the model across different samples. DeepRegularizer offers more than an order of magnitude faster regularization performance compared to the conventional iterative method. We envision that the proposed data-driven approach can bypass the high time complexity of various image reconstructions in other imaging modalities.

Time-resolved MR angiography (tMRA) has been widely used for dynamic contrast enhanced MRI (DCE-MRI) due to its highly accelerated acquisition. In tMRA, the periphery of the ${\textit k}$ -space data are sparsely sampled so that neighbouring frames can be merged to construct one temporal frame. However, this view-sharing scheme fundamentally limits the temporal resolution, and it is not possible to change the view-sharing number to achieve different spatio-temporal resolution trade-offs. Although many deep learning approaches have been recently proposed for MR reconstruction from sparse samples, the existing approaches usually require matched fully sampled ${\textit k}$ -space reference data for supervised training, which is not suitable for tMRA due to the lack of high spatio-temporal resolution ground-truth images. To address this problem, here we propose a novel unpaired training scheme for deep learning using optimal transport driven cycle-consistent generative adversarial network (cycleGAN). In contrast to the conventional cycleGAN with two pairs of generator and discriminator, the new architecture requires just a single pair of generator and discriminator, which makes the training much simpler but still improves the performance. Reconstruction results using in vivo tMRA and simulation data set confirm that the proposed method can immediately generate high quality reconstruction results at various choices of view-sharing numbers, allowing us to exploit better trade-off between spatial and temporal resolution in time-resolved MR angiography.

Optical diffraction tomography (ODT) produces a three-dimensional distribution of the refractive index (RI) by measuring scattering fields at various angles. Although the distribution of the RI is highly informative, due to the missing cone problem stemming from the limited-angle acquisition of holograms, reconstructions have very poor resolution along the axial direction compared to the horizontal imaging plane. To solve this issue, we present a novel unsupervised deep learning framework that learns the probability distribution of missing projection views through an optimal transport-driven CycleGAN. The experimental results show that missing cone artifacts in ODT data can be significantly resolved by the proposed method.

Deep-learning-based tomographic imaging is an important application of artificial intelligence and a new frontier of machine learning. Deep learning has been widely used in computer vision and image analysis, which deal with existing images, improve these images, and produce features from them. Since 2016, deep learning techniques have been actively researched for tomographic imaging, especially in the context of biomedicine, with impressive results and great potential. Tomographic reconstruction produces images of multi-dimensional structures from externally measured ‘encoded’ data in the form of various tomographic transforms (integrals, harmonics, echoes and so on). In this Review, we provide a general background, highlight representative results with an emphasis on medical imaging, and discuss key issues that need to be addressed in this emerging field. In particular, tomographic imaging is an integral part of modern medicine, and will play a key role in personalized, preventive and precision medicine and make it intelligent, inexpensive and indiscriminate. The popularity of deep learning is leading to new areas in biomedical applications. Wang and colleagues summarize in this Review the recent development and future directions of deep neural networks for superior image quality in the tomographic imaging field.

Conebeam CT using a circular trajectory is quite often used for various applications due to its relative simple geometry. For conebeam geometry, Feldkamp, Davis and Kress algorithm is regarded as the standard reconstruction method, but this algorithm suffers from so-called conebeam artifacts as the cone angle increases. Various model-based iterative reconstruction methods have been developed to reduce the cone-beam artifacts, but these algorithms usually require multiple applications of computational expensive forward and backprojections. In this paper, we develop a novel deep learning approach for accurate conebeam artifact removal. In particular, our deep network, designed on the differentiated backprojection domain, performs a data-driven inversion of an ill-posed deconvolution problem associated with the Hilbert transform. The reconstruction results along the coronal and sagittal directions are then combined using a spectral blending technique to minimize the spectral leakage. Experimental results show that our method outperforms the existing iterative methods despite significantly reduced runtime complexity., Comment: This paper is accepted for IEEE Trans. Medical Imaging

The nineteen papers in this special section focus on domain enriched learning for medical imaging. In recent years, learning based methods have emerged to complement traditional model and feature based methods for a variety of medical imaging problems such as image formation, classification and segmentation, quality enhancement etc. In the case of deep neural networks, many solutions have achieved unprecedented performance gains and have defined a new state of the art. Despite the progress, compelling open challenges remain. One such key challenge is that many learning frameworks (notably deep learning) are purely data-driven approaches and their performance depends strongly on the quantity and quality of training image data available. When training is limited or noisy, the performance drops sharply. Deep neural networks based approaches additionally face the challenge of often not being straightforward to interpret. Fortunately, exciting recent progress has emerged in enriching learning frameworks with domain knowledge and signal structure. As a couple of representative examples: in image reconstruction problems, this may involve using statistical/structural image priors; for image segmentation, shape and anatomical knowledge (conveyed by an expert) may be leveraged, etc. This special issue brings together contributions that combine signal, image priors and other flavors of domain knowledge with machine learning methods for solving many diverse medical imaging problems.

In ultrasound (US) imaging, various types of adaptive beamforming techniques have been investigated to improve the resolution and contrast-to-noise ratio of the delay and sum (DAS) beamformers. Unfortunately, the performance of these adaptive beamforming approaches degrade when the underlying model is not sufficiently accurate and the number of channels decreases. To address this problem, here we propose a deep learning-based beamformer to generate significantly improved images over widely varying measurement conditions and channel subsampling patterns. In particular, our deep neural network is designed to directly process full or sub-sampled radio-frequency (RF) data acquired at various subsampling rates and detector configurations so that it can generate high quality ultrasound images using a single beamformer. The origin of such input-dependent adaptivity is also theoretically analyzed. Experimental results using B-mode focused ultrasound confirm the efficacy of the proposed methods., This is a significantly extended version of the original paper in arXiv:1901.01706. This paper is accepted for IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control

Under the global pandemic of COVID-19, the use of artificial intelligence to analyze chest X-ray (CXR) image for COVID-19 diagnosis and patient triage is becoming important. Unfortunately, due to the emergent nature of the COVID-19 pandemic, a systematic collection of the CXR data set for deep neural network training is difficult. To address this problem, here we propose a patch-based convolutional neural network approach with a relatively small number of trainable parameters for COVID-19 diagnosis. The proposed method is inspired by our statistical analysis of the potential imaging biomarkers of the CXR radiographs. Experimental results show that our method achieves state-of-the-art performance and provides clinically interpretable saliency maps, which are useful for COVID-19 diagnosis and patient triage., Accepted for IEEE Trans. on Medical Imaging Special Issue on Imaging-based Diagnosis of COVID-19

A unique advantage of magnetic resonance imaging (MRI) is its mechanism for generating various image contrasts depending on tissue-specific parameters, which provides useful clinical information. Unfortunately, a complete set of MR contrasts is often difficult to obtain in a real clinical environment. Recently, there have been claims that generative models such as generative adversarial networks (GANs) can synthesize MR contrasts that are not acquired. However, the poor scalability of existing GAN-based image synthesis poses a fundamental challenge to understanding the nature of MR contrasts: which contrasts matter, and which cannot be synthesized by generative models? Here, we show that these questions can be addressed systematically by learning the joint manifold of multiple MR contrasts using collaborative generative adversarial networks. Our experimental results show that the exogenous contrast provided by contrast agents is not replaceable, but endogenous contrasts such as T1 and T2 can be synthesized from other contrasts. These findings provide important guidance for the acquisition-protocol design of MR in clinical environments. Magnetic resonance scans use different contrast agents to generate different images, each giving specific clinical information. Lee et al. use a collaborative generative model to synthesize some magnetic resonance contrasts from others, providing guidance for how clinical imaging times can be reduced.

Deconvolution microscopy has been extensively used to improve the resolution of the wide-field fluorescent microscopy, but the performance of classical approaches critically depends on the accuracy of a model and optimization algorithms. Recently, the convolutional neural network (CNN) approaches have been studied as a fast and high performance alternative. Unfortunately, the CNN approaches usually require matched high resolution images for supervised training. In this article, we present a novel unsupervised cycle-consistent generative adversarial network (cycleGAN) with a linear blur kernel, which can be used for both blind- and non-blind image deconvolution. In contrast to the conventional cycleGAN approaches that require two deep generators, the proposed cycleGAN approach needs only a single deep generator and a linear blur kernel, which significantly improves the robustness and efficiency of network training. We show that the proposed architecture is indeed a dual formulation of an optimal transport problem that uses a special form of the penalized least squares cost as a transport cost. Experimental results using simulated and real experimental data confirm the efficacy of the algorithm.

Recent state-of-the-art image segmentation algorithms are mostly based on deep neural networks, thanks to their high performance and fast computation time. However, these methods are usually trained in a supervised manner, which requires large number of high quality ground-truth segmentation masks. On the other hand, classical image segmentation approaches such as level-set methods are formulated in a self-supervised manner by minimizing energy functions such as Mumford-Shah functional, so they are still useful to help generation of segmentation masks without labels. Unfortunately, these algorithms are usually computationally expensive and often have limitation in semantic segmentation. In this paper, we propose a novel loss function based on Mumford-Shah functional that can be used in deep-learning based image segmentation without or with small labeled data. This loss function is based on the observation that the softmax layer of deep neural networks has striking similarity to the characteristic function in the Mumford-Shah functional. We show that the new loss function enables semi-supervised and unsupervised segmentation. In addition, our loss function can be also used as a regularized function to enhance supervised semantic segmentation algorithms. Experimental results on multiple datasets demonstrate the effectiveness of the proposed method., Comment: Accepted for IEEE Transactions on Image Processing

To improve the performance of classical generative adversarial network (GAN), Wasserstein generative adversarial networks (W-GAN) was developed as a Kantorovich dual formulation of the optimal transport (OT) problem using Wasserstein-1 distance. However, it was not clear how cycleGAN-type generative models can be derived from the optimal transport theory. Here we show that a novel cycleGAN architecture can be derived as a Kantorovich dual OT formulation if a penalized least square (PLS) cost with deep learning-based inverse path penalty is used as a transportation cost. One of the most important advantages of this formulation is that depending on the knowledge of the forward problem, distinct variations of cycleGAN architecture can be derived: for example, one with two pairs of generators and discriminators, and the other with only a single pair of generator and discriminator. Even for the two generator cases, we show that the structural knowledge of the forward operator can lead to a simpler generator architecture which significantly simplifies the neural network training. The new cycleGAN formulation, what we call the OT-cycleGAN, have been applied for various biomedical imaging problems, such as accelerated magnetic resonance imaging (MRI), super-resolution microscopy, and low-dose x-ray computed tomography (CT). Experimental results confirm the efficacy and flexibility of the theory., Comment: accepted for publication in the SIAM Journal on Imaging Sciences

Recently, deep learning approaches for MR motion artifact correction have been extensively studied. Although these approaches have shown high performance and lower computational complexity compared to classical methods, most of them require supervised training using paired artifact-free and artifact-corrupted images, which may prohibit its use in many important clinical applications. For example, transient severe motion (TSM) due to acute transient dyspnea in Gd-EOB-DTPA-enhanced MR is difficult to control and model for paired data generation. To address this issue, here we propose a novel unpaired deep learning scheme that does not require matched motion-free and motion artifact images. Specifically, the first step of our method is ${k}$ -space random subsampling along the phase encoding direction that can remove some outliers probabilistically. In the second step, the neural network reconstructs fully sampled resolution image from a downsampled ${k}$ -space data, and motion artifacts can be reduced in this step. Last, the aggregation step through averaging can further improve the results from the reconstruction network. We verify that our method can be applied for artifact correction from simulated motion as well as real motion from TSM successfully from both single and multi-coil data with and without ${k}$ -space raw data, outperforming existing state-of-the-art deep learning methods.

Developing a robust algorithm to diagnose and quantify the severity of the novel coronavirus disease 2019 (COVID-19) using Chest X-ray (CXR) requires a large number of well-curated COVID-19 datasets, which is difficult to collect under the global COVID-19 pandemic. On the other hand, CXR data with other findings are abundant. This situation is ideally suited for the Vision Transformer (ViT) architecture, where a lot of unlabeled data can be used through structural modeling by the self-attention mechanism. However, the use of existing ViT may not be optimal, as the feature embedding by direct patch flattening or ResNet backbone in the standard ViT is not intended for CXR. To address this problem, here we propose a novel Multi-task ViT that leverages low-level CXR feature corpus obtained from a backbone network that extracts common CXR findings. Specifically, the backbone network is first trained with large public datasets to detect common abnormal findings such as consolidation, opacity, edema, etc. Then, the embedded features from the backbone network are used as corpora for a versatile Transformer model for both the diagnosis and the severity quantification of COVID-19. We evaluate our model on various external test datasets from totally different institutions to evaluate the generalization capability. The experimental results confirm that our model can achieve state-of-the-art performance in both diagnosis and severity quantification tasks with outstanding generalization capability, which are sine qua non of widespread deployment., Graphical abstract Image, graphical abstract

In ultrasound (US) imaging, various adaptive beamforming methods have been proposed to improve the resolution and contrast-to-noise ratio of the delay and sum (DAS) beamformers. Unfortunately, they often require computationally expensive calculations and their performance degrades when the underlying model is not sufficiently accurate. Moreover, ultrasound images usually require various type of post filtration such as deblurring and despeckling, etc., which further increase the complexity of the system. Deep learning-based solutions provides a quick remedy to these issue; however, in the current technology, a separate beamformer should be trained and stored for each application, demanding significant scanner resources. To address this problem, here we propose a switchable deep beamformer that can produce various types of output such as DAS, speckle removal, deconvolution, etc., using a single network with a simple switch. In particular, the switch is implemented through Adaptive Instance Normalization (AdaIN) layers, so that distinct outputs can be generated by merely changing the AdaIN code. Experimental results using B-mode focused ultrasound confirm the efficacy of the proposed methods.

Echocardiography is a pivotal imaging tool for emergency medicine. Unfortunately, it suffers from poor image quality due to the intrinsic limitations of sonography systems. Towards this end, a better quality can be achieved at the cost of reduced frame rate by increasing the number of transmit/receive events and utilizing computationally expensive noise suppression algorithms. However, this visual quality and temporal resolution trade-off is a bottleneck for many echocardiography applications. Conventional acceleration methods, such as multi-line acquisition (MLA), work only for limited acceleration factors and produce blocking artifacts at a high frame rate. Accordingly, various machine learning algorithms have been designed to reduce blocking artifacts in MLA. These algorithms require access to either high-quality raw RF data or time-delayed baseband IQ data. Unfortunately, in many lower-end commercial systems, such data are not accessible. On the other hand, ultrasound images are badly affected by speckle noises which significantly reduces the image quality. We propose an image domain unsupervised deep learning framework using cycleGAN architecture for high quality accelerated echocardiography that simultaneously reduces the blocking artifacts and the speckle noise. The method is evaluated on real in-vivo and phantom data and achieves notable performance gain.

The articles in this special section focus on computational magnetic resonance imaging (MRI) using compressed sensing applications. Presents recent developments in computational MRI. These developments are pushing the frontier of computational imaging beyond CS. Similar to CS, most of these algorithms rely on image representation in one form or another. However, the common recent thread is the departure from handcrafted image representations to learning-based image representations. These learned representations are seamlessly combined with clever measurement strategies to significantly advance the state of the art in a number of areas. Several exciting applications including significantly improved spatial and temporal resolution, a considerable reduction in scan time, measurement of biophysical parameters directly from highly undersampled data, and direct measurement of very high-dimensional data are reviewed in this special issue of SPM. This issue describes key ideas underlying the computational approaches used in MRI. These approaches range from CS algorithms that rely on fixed transforms or dictionaries, to adaptive or shallow-learning algorithms that adapt the image representation to the data to recent deep-learning methods that learn a highly nonlinear representation from exemplar data. The articles provide insight into the capabilities of the current algorithms, their limitations, and their utility in challenging MRI problems.

Purpose Nyquist ghost artifacts in echo planar imaging (EPI) are originated from phase mismatch between the even and odd echoes. However, conventional correction methods using reference scans often produce erroneous results especially in high-field MRI due to the nonlinear and time-varying local magnetic field changes. Recently, it was shown that the problem of ghost correction can be reformulated as k-space interpolation problem that can be solved using structured low-rank Hankel matrix approaches. Another recent work showed that data driven Hankel matrix decomposition can be reformulated to exhibit similar structures as deep convolutional neural network. By synergistically combining these findings, we propose a k-space deep learning approach that immediately corrects the phase mismatch without a reference scan in both accelerated and non-accelerated EPI acquisitions. Theory and methods To take advantage of the even and odd-phase directional redundancy, the k-space data are divided into 2 channels configured with even and odd phase encodings. The redundancies between coils are also exploited by stacking the multi-coil k-space data into additional input channels. Then, our k-space ghost correction network is trained to learn the interpolation kernel to estimate the missing virtual k-space data. For the accelerated EPI data, the same neural network is trained to directly estimate the interpolation kernels for missing k-space data from both ghost and subsampling. Results Reconstruction results using 3T and 7T in vivo data showed that the proposed method outperformed the image quality compared to the existing methods, and the computing time is much faster. Conclusions The proposed k-space deep learning for EPI ghost correction is highly robust and fast, and can be combined with acceleration, so that it can be used as a promising correction tool for high-field MRI without changing the current acquisition protocol.

In portable, three dimensional, and ultra-fast ultrasound imaging systems, there is an increasing demand for the reconstruction of high quality images from a limited number of radio-frequency (RF) measurements due to receiver (Rx) or transmit (Xmit) event sub-sampling. However, due to the presence of side lobe artifacts from RF sub-sampling, the standard beamformer often produces blurry images with less contrast, which are unsuitable for diagnostic purposes. Existing compressed sensing approaches often require either hardware changes or computationally expensive algorithms, but their quality improvements are limited. To address this problem, here we propose a novel deep learning approach that directly interpolates the missing RF data by utilizing redundancy in the Rx-Xmit plane. Our extensive experimental results using sub-sampled RF data from a multi-line acquisition B-mode system confirm that the proposed method can effectively reduce the data rate without sacrificing image quality., The title has been changed. This version will appear in IEEE Trans. on Medical Imaging

Localization microscopy, such as STORM / PALM, can reconstruct super-resolution images with a nanometer resolution through the iterative localization of fluorescence molecules. Recent studies in this area have focused mainly on the localization of densely activated molecules to improve temporal resolutions. However, higher density imaging requires an advanced algorithm that can resolve closely spaced molecules. Accordingly, sparsitydriven methods have been studied extensively. One of the major limitations of existing sparsity-driven approaches is the need for a fine sampling grid or for Taylor series approximation which may result in some degree of localization bias toward the grid. In addition, prior knowledge of the point-spread function (PSF) is required. To address these drawbacks, here we propose a true grid-free localization algorithm with adaptive PSF estimation. Specifically, based on the observation that sparsity in the spatial domain implies a low rank in the Fourier domain, the proposed method converts source localization problems into Fourier-domain signal processing problems so that a truly gridfree localization is possible. We verify the performance of the newly proposed method with several numerical simulations and a live-cell imaging experiment.

Recently, CycleGAN was shown to provide high-performance, ultra-fast denoising for low-dose X-ray computed tomography (CT) without the need for a paired training dataset. Although this was possible thanks to cycle consistency, CycleGAN requires two generators and two discriminators to enforce cycle consistency, demanding significant GPU resources and technical skills for training. A recent proposal of tunable CycleGAN with Adaptive Instance Normalization (AdaIN) alleviates the problem in part by using a single generator. However, two discriminators and an additional AdaIN code generator are still required for training. To solve this problem, here we present a novel cycle-free Cycle-GAN architecture, which consists of a single generator and a discriminator but still guarantees cycle consistency. The main innovation comes from the observation that the use of an invertible generator automatically fulfills the cycle consistency condition and eliminates the additional discriminator in the CycleGAN formulation. To make the invertible generator more effective, our network is implemented in the wavelet residual domain. Extensive experiments using various levels of low-dose CT images confirm that our method can significantly improve denoising performance using only 10% of learnable parameters and faster training time compared to the conventional CycleGAN., Comment: 12 pages, 12 figures

Point-of-Care Ultrasound (POCUS) imaging can help efficient resource utilization by reducing the secondary care referrals, and work as an extension in physical examination. Recently, many methods were proposed to reduce the size and power consumption of the system while improving the visual quality, but hand-held POCUS devices still have inferior image contrast and spatial resolution compared to the high-end ultrasound systems. To address this, here we propose an efficient solution for contrast and resolution enhancement of hand-held POCUS images using unsupervised deep learning. In contrast to the existing CycleGAN approaches that have difficulty in improving both contrast and image resolutions, the proposed method mitigate the problem by decomposing the contrast transfer and resolution improvement through CycleGAN and self-supervised learning. Experimental results confirmed that our method is superior than the conventional approaches.

In electrocardiography (ECG) gated cardiac CT angiography (CCTA), multiple images covering the entire cardiac cycle are taken continuously, so reduction of the accumulated radiation dose could be an important issue for patient safety. Although ECG-gated dose modulation (so-called ECG pulsing) is used to acquire many phases of CT images at a low dose, the reduction of the radiation dose introduces noise into the image reconstruction. To address this, we developed a high performance unsupervised deep learning method using noise disentanglement that can effectively learn the noise patterns even from extreme low dose CT images. For noise disentanglement, we use a wavelet transform to extract the high-frequency signals that contain the most noise. Since matched low-dose and high-dose cardiac CT data are impossible to obtain in practice, our neural network was trained in an unsupervised manner using cycleGAN for the extracted high frequency signals from the low-dose and unpaired high-dose CT images. Once the network is trained, denoised images are obtained by subtracting the estimated noise components from the input images. Image quality evaluation of the denoised images from only 4% dose CT images was performed by experienced radiologists for several anatomical structures. Visual grading analysis was conducted according to the sharpness level, noise level, and structural visibility. Also, the signal-to-noise ratio was calculated. The evaluation results showed that the quality of the images produced by the proposed method is much improved compared to low-dose CT images and to the baseline cycleGAN results. The proposed noise-disentangled cycleGAN with wavelet transform effectively removed noise from extreme low-dose CT images compared to the existing baseline algorithms. It can be an important denoising platform for low-dose CT.

In recent years, artificial intelligence (AI) technologies have greatly advanced and become a reality in many areas of our daily lives. In the health care field, numerous efforts are being made to implement the AI technology for practical medical treatments. With the rapid developments in machine learning algorithms and improvements in hardware performances, the AI technology is expected to play an important role in effectively analyzing and utilizing extensive amounts of health and medical data. However, the AI technology has various unique characteristics that are different from the existing health care technologies. Subsequently, there are a number of areas that need to be supplemented within the current health care system for the AI to be utilized more effectively and frequently in health care. In addition, the number of medical practitioners and public that accept AI in the health care is still low; moreover, there are various concerns regarding the safety and reliability of AI technology implementations. Therefore, this paper aims to introduce the current research and application status of AI technology in health care and discuss the issues that need to be resolved., Graphical Abstract

High quality US imaging demand large number of measurements that can increase the cost, size and power requirements. Therefore, low-powered, portable and 3D ultrasound imaging system require reconstruction algorithms that can produce high quality images using fewer receive measurements. Number of model specific methods has been proposed which doesn't work under perturbation. For instance, compressive deconvolution ultrasound which provide a reasonable quality with limited measurements however, it has its own down-sides such as high computation cost and accurate estimation of point spread function (PSF). An other major limitation of conventional methods is that they require RF or base-band signal which is difficult to obtain from portable US systems. To deal with the aforementioned issues, in this study we designed a novel deep deconvolution model for image domain-based deconvolution. The proposed deep deconvolution (DeepDeconv) model can be trained in an unsupervised fashion, alleviate the need of paired high and low quality images. The model was evaluated on both the phantom and in-vivo scans for various sampling configurations. The proposed DeepDeconv significantly enhance the details of anatomical structures and using unsupervised learning on average it achieved 2.14dB, 4.96dB and 0.01 units gain in CR, PSNR and SSIM values respectively, which are comparable to the supervised method.

Recently, deep learning approaches for accelerated MRI have been extensively studied thanks to their high performance reconstruction in spite of significantly reduced runtime complexity. These neural networks are usually trained in a supervised manner, so matched pairs of subsampled and fully sampled k-space data are required. Unfortunately, it is often difficult to acquire matched fully sampled k-space data, since the acquisition of fully sampled k-space data requires long scan time and often leads to the change of the acquisition protocol. Therefore, unpaired deep learning without matched label data has become a very important research topic. In this paper, we propose an unpaired deep learning approach using a optimal transport driven cycle-consistent generative adversarial network (OT-cycleGAN) that employs a single pair of generator and discriminator. The proposed OT-cycleGAN architecture is rigorously derived from a dual formulation of the optimal transport formulation using a specially designed penalized least squares cost. The experimental results show that our method can reconstruct high resolution MR images from accelerated k- space data from both single and multiple coil acquisition, without requiring matched reference data., Accepted for IEEE Transactions on Computational Imaging

Image registration is a fundamental task in medical image analysis. Recently, many deep learning based image registration methods have been extensively investigated due to their comparable performance with the state-of-the-art classical approaches despite the ultra-fast computational time. However, the existing deep learning methods still have limitations in the preservation of original topology during the deformation with registration vector fields. To address this issues, here we present a cycle-consistent deformable image registration, dubbed CycleMorph. The cycle consistency enhances image registration performance by providing an implicit regularization to preserve topology during the deformation. The proposed method is so flexible that it can be applied for both 2D and 3D registration problems for various applications, and can be easily extended to multi-scale implementation to deal with the memory issues in large volume registration. Experimental results on various datasets from medical and non-medical applications demonstrate that the proposed method provides effective and accurate registration on diverse image pairs within a few seconds. Qualitative and quantitative evaluations on deformation fields also verify the effectiveness of the cycle consistency of the proposed method.

Time-of-flight magnetic resonance angiography (TOF-MRA) is one of the most widely used non-contrast MR imaging methods to visualize blood vessels, but due to the 3-D volume acquisition highly accelerated acquisition is necessary. Accordingly, high quality reconstruction from undersampled TOF-MRA is an important research topic for deep learning. However, most existing deep learning works require matched reference data for supervised training, which are often difficult to obtain. By extending the recent theoretical understanding of cycleGAN from the optimal transport theory, here we propose a novel two-stage unsupervised deep learning approach, which is composed of the multi-coil reconstruction network along the coronal plane followed by a multi-planar refinement network along the axial plane. Specifically, the first network is trained in the square-root of sum of squares (SSoS) domain to achieve high quality parallel image reconstruction, whereas the second refinement network is designed to efficiently learn the characteristics of highly-activated blood flow using double-headed projection discriminator. Extensive experiments demonstrate that the proposed learning process without matched reference exceeds performance of state-of-the-art compressed sensing (CS)-based method and provides comparable or even better results than supervised learning approaches.

Graph neural networks (GNN) rely on graph operations that include neural network training for various graph related tasks. Recently, several attempts have been made to apply the GNNs to functional magnetic resonance image (fMRI) data. Despite recent progresses, a common limitation is its difficulty to explain the classification results in a neuroscientifically explainable way. Here, we develop a framework for analyzing the fMRI data using the Graph Isomorphism Network (GIN), which was recently proposed as a powerful GNN for graph classification. One of the important contributions of this paper is the observation that the GIN is a dual representation of convolutional neural network (CNN) in the graph space where the shift operation is defined using the adjacency matrix. This understanding enables us to exploit CNN-based saliency map techniques for the GNN, which we tailor to the proposed GIN with one-hot encoding, to visualize the important regions of the brain. We validate our proposed framework using large-scale resting-state fMRI (rs-fMRI) data for classifying the sex of the subject based on the graph structure of the brain. The experiment was consistent with our expectation such that the obtained saliency map show high correspondence with previous neuroimaging evidences related to sex differences., Comment: This paper is accepted for Frontiers in Neuroscience

Optimal transport (OT) is a mathematical theory that can provide a tool how to transfer one measure to another measure at minimal cost, thus serve another framework for computer vision tasks of image processing without reference. Cycleconsistent generative adversarial network (cycleGAN) is a recent extension of GAN to learn target distributions with less mode collapsing behavior, and also does not need matched data during training. In this article, we explain the link between these two framework by mathematical formula and experimental results. We prove that cycleGAN architecture can be derived from optimal transport problem, and this implies that cycleGAN is a plausible way to learn target distribution when it comes to handling data far from the training set. Using accelerated MR imaging experiments, we confirmed the flexibility and efficacy of our theoretical framework.

Recently, deep learning based approaches for accelerated MRI have been extensively studied due to its high performance and reduced run time complexity. The existing deep learning methods for accelerated MRI are mostly supervised methods, where matched subsampled k-space data and fully sampled k-space data are necessary. However, it is hard to acquire fully sampled k-space data because of long scan time of MRI. Therefore, unsupervised method without matched label data has become a very important research topic. In this paper, we propose an unsupervised method using a novel cycle-consistent generative adversarial network (cycleGAN) with a single deep generator. We show that the proposed cy-cleGAN architecture can be derived from a dual formulation of the optimal transport with the penalized least squares cost. The results of experiments show that our method can remove aliasing patterns in downsampled MR images without the matched reference data.

Cone-beam computed tomography (CBCT) used in radiotherapy (RT) has the advantage of being taken daily, but is difficult to use for purposes other than patient setup because of the poor image quality compared to fan-beam computed tomography (CT). Even though several methods have been proposed including the deformable image registration method to improve the quality of CBCT, the outcomes have not yet been satisfactory. Recently, deep learning has shown to produce high-quality results for various image-to-image translation tasks, suggesting the possibility of being an effective tool for converting CBCT into CT. In the field of RT, however, it may not always be possible to obtain paired datasets which consist of exactly matching CBCT and CT images. This study aimed to develop a novel, unsupervised deep-learning algorithm, which requires only unpaired CBCT and fan-beam CT images to remove the cone-beam artifact and thereby improve the quality of CBCT. Specifically, two cycle consistency generative adversarial networks (CycleGAN) were trained in the sagittal and coronal directions, and the generated results along those directions were then combined using spectral blending technique. To evaluate our methods, we applied it to American Association of Physicists in Medicine dataset. The experimental results show that our method outperforms the existing CycleGAN-based method both qualitatively and quantitatively.

Magnetic resonance imaging (MRI) acquisition is an inherently slow process whose acceleration has been the subject of much investigation. In recent years, the explosive advance of deep learning techniques for computer vision and image reconstruction has led to the investigation of deep neural networks for the reconstruction of MRI with under-sampled k-space. In this work, we propose a new image domain architecture that directly produces a sum-of-squares image from under-sampled multi-coil MRI acquisition. This model, called BarbellNet, is a fully convolutional neural network architecture that utilizes the channel attention mechanism using the residual channel attention block (RCAB). Through extensive experiments with the fastMRI data set, we confirm the efficacy of BarbellNet.

For circular trajectory conebeam CT, Feldkamp, Davis, and Kress (FDK) algorithm is widely used for its reconstruction. However, the existence of cone-angle artifacts is fatal for the quality when using this algorithm. There are several model-based iterative reconstruction methods for the cone-angle artifacts removal, but these algorithms usually require repeated applications of computational expensive forward and backward. In this paper, we propose a novel deep learning approach for cone-angle artifact removal on differentiated backprojection domain, which performs a data-driven inversion of an ill-posed deconvolution problem related to the Hilbert transform. The reconstruction results along the coronal and sagittal directions are then combined by a spectral blending technique to minimize the spectral leakage. Experimental results show that our method provides superior performance to the existing methods.

This paper reviews the NTIRE 2020 challenge on perceptual extreme super-resolution with focus on proposed solutions and results. The challenge task was to super-resolve an input image with a magnification factor 16 based on a set of prior examples of low and corresponding high resolution images. The goal is to obtain a network design capable to produce high resolution results with the best perceptual quality and similar to the ground truth. The track had 280 registered participants, and 19 teams submitted the final results. They gauge the state-of-the-art in single image super-resolution., Comment: CVPRW 2020

This paper reviews the second AIM learned ISP challenge and provides the description of the proposed solutions and results. The participating teams were solving a real-world RAW-to-RGB mapping problem, where to goal was to map the original low-quality RAW images captured by the Huawei P20 device to the same photos obtained with the Canon 5D DSLR camera. The considered task embraced a number of complex computer vision subtasks, such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The target metric used in this challenge combined fidelity scores (PSNR and SSIM) with solutions’ perceptual results measured in a user study. The proposed solutions significantly improved the baseline results, defining the state-of-the-art for practical image signal processing pipeline modeling.

Recently, deep learning approaches have been extensively investigated to reconstruct images from accelerated magnetic resonance image (MRI) acquisition. Although these approaches provide significant performance gain compared to compressed sensing MRI (CS-MRI), it is not clear how to choose a suitable network architecture to balance the trade-off between network complexity and performance. Recently, it was shown that an encoder-decoder convolutional neural network (CNN) can be interpreted as a piecewise linear basis-like representation, whose specific representation is determined by the ReLU activation patterns for a given input image. Thus, the expressivity or the representation power is determined by the number of piecewise linear regions. As an extension of this geometric understanding, this paper proposes a systematic geometric approach using bootstrapping and subnetwork aggregation using an attention module to increase the expressivity of the underlying neural network. Our method can be implemented in both k-space domain and image domain that can be trained in an end-to-end manner. Experimental results show that the proposed schemes significantly improve reconstruction performance with negligible complexity increases., Comment: Accepted for IEEE JSTSP Special Issue on Domain Enriched Learning for Medical Imaging

Plane-wave compounding is to sum up several successive plane waves incident at different angles to form an image. By applying time-reversal of the received signals, transmit focusing can be synthesized. Unfortunately, to improve the temporal resolution, the number of plane waves should be reduced, which often degrades the image quality. To address this problem, an image domain learning method using neural networks has been proposed, but the network needs to be retrained when the number of plane waves changes. Herein, we propose, for the first time, a universal plane-wave compounding scheme using deep learning to directly process plane waves and RF data acquired at different view angles and sub-sampling rate to generate high quality US images.

X-ray computed tomography (CT) using sparse projection views is a recent approach to reduce the radiation dose. However, due to the insufficient projection views, an analytic reconstruction approach using the filtered back projection (FBP) produces severe streaking artifacts. Recently, deep learning approaches using large receptive field neural networks such as U-Net have demonstrated impressive performance for sparse- view CT reconstruction. However, theoretical justification is still lacking. Inspired by the recent theory of deep convolutional framelets, the main goal of this paper is, therefore, to reveal the limitation of U-Net and propose new multi-resolution deep learning schemes. In particular, we show that the alternative U- Net variants such as dual frame and the tight frame U-Nets satisfy the so-called frame condition which make them better for effective recovery of high frequency edges in sparse view- CT. Using extensive experiments with real patient data set, we demonstrate that the new network architectures provide better reconstruction performance., This will appear in IEEE Transaction on Medical Imaging, a special issue of Machine Learning for Image Reconstruction

BACKGROUND Cone-beam computed tomography (CBCT) is widely used in various medical imaging applications, including dental examinations. Dental CBCT images often suffer from motion artifacts caused by involuntary rigid motion of patients. However, earlier motion compensation studies are not applicable for dental CBCT systems using truncated detectors. OBJECTIVE This study proposes a novel motion correction algorithm that can be applied for truncated dental CBCT images. METHODS We propose a two-step method for motion correction. First, we estimate the relative displacement of each pair of opposite projections by finding the motion vector that maximizes the two-dimensional correlation coefficients of the opposite projections. Second, we convert the relative displacement into the absolute coordinate motion that yields the highest image sharpness of the reconstruction image. Using the motion vectors in the absolute coordinate system, motion artifacts are then compensated by modifying the trajectory of the source and detector during the back-projection step of the image reconstruction process. RESULTS In simulation, the proposed method successfully estimated the true relative displacement. After converting to the absolute coordinate motions, the motion-compensated image was close to the ground-truth image and exhibited a lower mean-square-error than that of the uncompensated image. The results from the real data experiment also confirmed that the proposed method successfully compensated for the motion artifacts. CONCLUSIONS The experimental results confirmed that the proposed method was applicable to most dental CBCT systems using a truncated detector without any use of an additional motion tracking system nor prior knowledge.

Parallel MRI (pMRI) and compressed sensing MRI (CS-MRI) have been considered as two distinct reconstruction problems. Inspired by recent k-space interpolation methods, an annihilating filter based low-rank Hankel matrix approach (ALOHA) is proposed as a general framework for sparsity-driven k-space interpolation method which unifies pMRI and CS-MRI. Specifically, our framework is based on the fundamental duality between the transform domain sparsity in the primary space and the low-rankness of weighted Hankel matrix in the reciprocal space, which converts pMRI and CS-MRI to a k-space interpolation problem using structured matrix completion. Using theoretical results from the latest compressed sensing literatures, we showed that the required sampling rates for ALOHA may achieve the optimal rate. Experimental results with in vivo data for single/multi-coil imaging as well as dynamic imaging confirmed that the proposed method outperforms the state-of-the-art pMRI and CS-MRI.

Diffuse optical tomography (DOT) has been investigated as an alternative imaging modality for breast cancer detection thanks to its excellent contrast to hemoglobin oxidization level. However, due to the complicated non-linear photon scattering physics and ill-posedness, the conventional reconstruction algorithms are sensitive to imaging parameters such as boundary conditions. To address this, here we propose a novel deep learning approach that learns non-linear photon scattering physics and obtains an accurate three dimensional (3D) distribution of optical anomalies. In contrast to the traditional black-box deep learning approaches, our deep network is designed to invert the Lippman-Schwinger integral equation using the recent mathematical theory of deep convolutional framelets. As an example of clinical relevance, we applied the method to our prototype DOT system. We show that our deep neural network, trained with only simulation data, can accurately recover the location of anomalies within biomimetic phantoms and live animals without the use of an exogenous contrast agent., Accepted for IEEE Trans. on Medical Imaging

Purpose Magnetic resonance (MR) imaging with a long scan time can lead to degraded images due to patient motion, patient discomfort, and increased costs. For these reasons, the role of rapid MR imaging is important. In this study, we propose the joint reconstruction of multicontrast brain MR images from down-sampled data to accelerate the data acquisition process using a novel deep-learning network. Methods Twenty-one healthy volunteers (female/male = 7/14, age = 26 ± 4 yr, range 22-35 yr) and 16 postoperative patients (female/male = 7/9, age = 49 ± 9 yr, range 37-62 yr) were scanned on a 3T whole-body scanner for prospective and retrospective studies, respectively, using both T1-weighted spin-echo (SE) and T2-weighted fast spin-echo (FSE) sequences. We proposed a network which we term "X-net" to reconstruct both T1- and T2-weighted images from down-sampled images as well as a network termed "Y-net" which reconstructs T2-weighted images from highly down-sampled T2-weighted images and fully sampled T1-weighted images. Both X-net and Y-net are composed of two concatenated subnetworks. We investigate optimal sampling patterns, the optimal patch size for augmentation, and the optimal acceleration factors for network training. An additional Y-net combined with a generative adversarial network (GAN) was also implemented and tested to investigate the effects of the GAN on the Y-net performance. Single- and joint-reconstruction parallel-imaging and compressed-sensing algorithms along with a conventional U-net were also tested and compared with the proposed networks. For this comparison, the structural similarity (SSIM), normalized mean square error (NMSE), and Frechet inception distance (FID) were calculated between the outputs of the networks and fully sampled images. The statistical significance of the performance was evaluated by assessing the interclass correlation and in paired t-tests. Results The outputs from the two concatenated subnetworks were closer to the fully sampled images compared to those from one subnetwork, with this result showing statistical significance. Uniform down-sampling led to a statically significant improvement in the image quality compared to random or central down-sampling patterns. In addition, the proposed networks provided higher SSIM and NMSE values than U-net, compressed-sensing, and parallel-imaging algorithms, all at statistically significant levels. The GAN-based Y-net showed a better FID and more realistic images compared to a non-GAN-based Y-net. The performance capabilities of the networks were similar between normal subjects and patients. Conclusions The proposed X-net and Y-net effectively reconstructed full images from down-sampled images, outperforming the conventional parallel-imaging, compressed-sensing and U-net methods and providing more realistic images in combination with a GAN. The developed networks potentially enable us to accelerate multicontrast anatomical MR imaging in routine clinical studies including T1-and T2-weighted imaging.

Quantitative susceptibility mapping (QSM) inevitably suffers from streaking artifacts caused by zeros on the conical surface of the dipole kernel in k-space. This work proposes a novel and accurate QSM reconstruction method based on a direct k-space interpolation approach, avoiding problems of over smoothing and streaking artifacts. Inspired by the recent theory of annihilating filter-based low-rank Hankel matrix approach (ALOHA), QSM reconstruction problem is formulated as deconvolution problem under low-rank Hankel matrix constraint in the k-space. To reduce the computational complexity and the memory requirement, the problem is formulated as successive reconstruction of 2-D planes along three independent axes of the 3-D phase image in Fourier domain. Extensive experiments were performed to verify and compare the proposed method with existing QSM reconstruction methods. The proposed ALOHA-QSM effectively reduced streaking artifacts and accurately estimated susceptibility values in deep gray matter structures, compared to the existing QSM methods. Our suggested ALOHA-QSM algorithm successfully solves the three-dimensional QSM dipole inversion problem without additional anatomical information or prior assumption and provides good image quality and quantitative accuracy., accepted for Magnetic Resonance in Medicine

Magnetic resonance imaging (MRI) is an inherently slow imaging modality, since it acquires multi-dimensional k-space data through 1-D free induction decay or echo signals. This often limits the use of MRI, especially for high resolution or dynamic imaging. Accordingly, many investigators has developed various acceleration techniques to allow fast MR imaging. For the last two decades, one of the most important breakthroughs in this direction is the introduction of compressed sensing (CS) that allows accurate reconstruction from sparsely sampled k-space data. The recent FDA approval of compressed sensing products for clinical scans clearly reflect the maturity of this technology. Therefore, this paper reviews the basic idea of CS and how this technology have been evolved for various MR imaging problems.