Search

Brain functional connectivity network (FCN) based on resting-state functional magnetic resonance imaging (rs-fMRI) has been widely used to identify neuropsychiatric disorders such as autism spectrum disorder (ASD). Most existing FCN-based methods only estimate the correlation between brain regions of in terest (ROIs), without exploring more informative higher-level inter actions among multiple ROIs which could be beneficial to disease diagnosis. To fully explore the discriminative information provided by different brain networks, a cluster-based multi-view high-order FCN (Ho-FCN) framework is proposed in this paper. Specifically, we first group the functional connectivity (FC) time series into different clusters and compute the multi-order central moment series for the FC time series in each cluster. Then we utilize the correlation of central moment series between different clusters to reveal the high-order FC relationships among multiple ROIs. In addition, to address the phase mismatch issue in conventional FCNs, we also adopt the central moments of the correlation time series as the temporal-invariance features to capture the dynamic characteristics of low-order dynamic FCN (Lo-D-FCN). Experimental results on the ABIDE dataset validate that: 1) the proposed multi-view Ho-FCNs is able to explore rich discriminative information for ASD diagnosis; 2) the phase mismatch issue can be well circumvented by using central moments; and 3) the combination of different types of FCNs can significantly improve the diagnostic accuracy of ASD (86.2%).

Resting-state functional magnetic resonance imaging (rs-fMRI) has become a popular non-invasive way of diagnosing neurological disorders or their early stages by probing functional connectivity between different brain regions of interest (ROIs) across subjects. In the past decades, researchers have proposed many methods to estimate brain functional networks (BFNs) based on blood-oxygen-level-dependent (BOLD) signals captured by rs-fMRI. However, most of the existing methods estimate BFNs under the assumption that signals are independently sampled, which ignores the temporal dependency and sequential order of different time points (or volumes). To address this problem, in this paper, we first propose a novel BFN estimation model by introducing a latent variable to control the sequence of volumes for encoding the temporal dependency and sequential information of signals into the estimated BFNs. Then, we develop an efficient learning algorithm to solve the proposed model by the alternating optimization scheme. To verify the effectiveness of the proposed method, the estimated BFNs are used to identify subjects with mild cognitive impairment (MCIs) from normal controls (NCs). Experimental results show that our method outperforms the baseline methods in the sense of classification performance.

Purpose Dynamic contrast-enhanced MRI (DCE-MRI) registration is a challenging task because of the effect of remarkable intensity changes caused by contrast agent injections. Unrealistic deformation usually occurs by using traditional intensity-based algorithms. To alleviate the effect of contrast agent on registration, we proposed a DCE-MRI registration strategy and investigated the registration performance on the clinical DCE-MRI time series of liver. Method We reconstructed the time-intensity curves of the contrast agent through sparse representation with a predefined dictionary whose columns were the time-intensity curves with high correlations with respect to a preselected contrast agent curve. After reshaping 1D-reconstructed contrast agent time-intensity curves into a 4D contrast agent time series, we aligned the original time series to the reconstructed contrast agent time series through traditional free-form deformation (FFD) registration scheme combined with a residual complexity (RC) similarity and an iterative registration strategy. This study included the DCE-MRI time series of 20 patients with liver cancer. Results Qualitatively, the time-cut images and subtraction images of different registration methods did not obviously differ. Quantitatively, the mean (standard deviation) of temporal intensity smoothness of all the patients achieved 54.910 (18.819), 54.609 (18.859), and 53.391 (19.031) in FFD RC, RDDR, Zhou et al.'s method and the proposed method, respectively. The mean (standard deviation) of changes in the lesion volume were 0.985 (0.041), 0.983 (0.041), 0.981 (0.046), and 0.989 (0.036) in FFD RC, RDDR, Zhou et al.'s method and the proposed method. Conclusion Our proposed method would be an effective registration strategy for DCE-MRI time series, and its performance was comparable with that of three advanced registration methods.

Magnetic resonance imaging (MRI) is a major imaging technique for studying neuroanatomy. By applying different pulse sequences and parameters, different modalities can be generated regarding the same anatomical structure, which can provide complementary information for diagnosis. However, limited by the scanning time and related cost, multiple different modalities are often not available for the same patient in clinic. Recently, many methods have been proposed for cross-modality MRI synthesis, but most of them only consider pixel-level differences between the synthetic and ground-truth images, ignoring the edge information, which is critical to provide clinical information. In this paper, we propose a novel edge-preserving MRI image synthesis method with iterative multi-scale feature fusion based generative adversarial network (EP_IMF-GAN). Particularly, the generator consists of a shared encoder and two specific decoders to carry out different tasks: 1) a primary task aiming to generate the target modality and 2) an auxiliary task aiming to generate the corresponding edge image of target modality. We assume that infusing the auxiliary edge image generation task can help preserve edge information and learn better latent representation features through the shared encoder. Meanwhile, an iterative multi-scale fusion module is embedded in the primary decoder to fuse supplementary information of feature maps at different scales, thereby further improving quality of the synthesized target modality. Experiments on the BRATS dataset indicate that our proposed method is superior to the state-of-the-art image synthesis approaches in both qualitative and quantitative measures. Ablation study further validates the effectiveness of the proposed components.

Cortical surface registration is an essential step and prerequisite for surface-based neuroimaging analysis. It aligns cortical surfaces across individuals and time points to establish cross-sectional and longitudinal cortical correspondences to facilitate neuroimaging studies. Though achieving good performance, available methods are either time consuming or not flexible to extend to multiple or high dimensional features. Considering the explosive availability of large-scale and multimodal brain MRI data, fast surface registration methods that can flexibly handle multimodal features are desired. In this study, we develop a Superfast Spherical Surface Registration (S3Reg) framework for the cerebral cortex. Leveraging an end-to-end unsupervised learning strategy, S3Reg offers great flexibility in the choice of input feature sets and output similarity measures for registration, and meanwhile reduces the registration time significantly. Specifically, we exploit the powerful learning capability of spherical Convolutional Neural Network (CNN) to directly learn the deformation fields in spherical space and implement diffeomorphic design with “scaling and squaring” layers to guarantee topology-preserving deformations. To handle the polar-distortion issue, we construct a novel spherical CNN model using three orthogonal Spherical U-Nets. Experiments are performed on two different datasets to align both adult and infant multimodal cortical features. Results demonstrate that our S3Reg shows superior or comparable performance with state-of-the-art methods, while improving the registration time from 1 min to 10 sec.

Accurate segmentation of the prostate is a key step in external beam radiation therapy treatments. In this paper, we tackle the challenging task of prostate segmentation in CT images by a two-stage network with 1) the first stage to fast localize, and 2) the second stage to accurately segment the prostate. To precisely segment the prostate in the second stage, we formulate prostate segmentation into a multi-task learning framework, which includes a main task to segment the prostate, and an auxiliary task to delineate the prostate boundary. Here, the second task is applied to provide additional guidance of unclear prostate boundary in CT images. Besides, the conventional multi-task deep networks typically share most of the parameters (i.e., feature representations) across all tasks, which may limit their data fitting ability, as the specificity of different tasks are inevitably ignored. By contrast, we solve them by a hierarchically-fused U-Net structure, namely HF-UNet. The HF-UNet has two complementary branches for two tasks, with the novel proposed attention-based task consistency learning block to communicate at each level between the two decoding branches. Therefore, HF-UNet endows the ability to learn hierarchically the shared representations for different tasks, and preserve the specificity of learned representations for different tasks simultaneously. We did extensive evaluations of the proposed method on a large planning CT image dataset and a benchmark prostate zonal dataset. The experimental results show HF-UNet outperforms the conventional multi-task network architectures and the state-of-the-art methods.

Automatic pancreas segmentation is crucial to the diagnostic assessment of diabetes or pancreatic cancer. However, the relatively small size of the pancreas in the upper body, as well as large variations of its location and shape in retroperitoneum, make the segmentation task challenging. To alleviate these challenges, in this article, we propose a cascaded multitask 3-D fully convolution network (FCN) to automatically segment the pancreas. Our cascaded network is composed of two parts. The first part focuses on fast locating the region of the pancreas, and the second part uses a multitask FCN with dense connections to refine the segmentation map for fine voxel-wise segmentation. In particular, our multitask FCN with dense connections is implemented to simultaneously complete tasks of the voxel-wise segmentation and skeleton extraction from the pancreas. These two tasks are complementary, that is, the extracted skeleton provides rich information about the shape and size of the pancreas in retroperitoneum, which can boost the segmentation of pancreas. The multitask FCN is also designed to share the low- and mid-level features across the tasks. A feature consistency module is further introduced to enhance the connection and fusion of different levels of feature maps. Evaluations on two pancreas datasets demonstrate the robustness of our proposed method in correctly segmenting the pancreas in various settings. Our experimental results outperform both baseline and state-of-the-art methods. Moreover, the ablation study shows that our proposed parts/modules are critical for effective multitask learning.

Brain connectivity alterations associated with mental disorders have been widely reported in both functional MRI (fMRI) and diffusion MRI (dMRI). However, extracting useful information from the vast amount of information afforded by brain networks remains a great challenge. Capturing network topology, graph convolutional networks (GCNs) have demonstrated to be superior in learning network representations tailored for identifying specific brain disorders. Existing graph construction techniques generally rely on a specific brain parcellation to define regions-of-interest (ROIs) to construct networks, often limiting the analysis into a single spatial scale. In addition, most methods focus on the pairwise relationships between the ROIs and ignore high-order associations between subjects. In this letter, we propose a mutual multi-scale triplet graph convolutional network (MMTGCN) to analyze functional and structural connectivity for brain disorder diagnosis. We first employ several templates with different scales of ROI parcellation to construct coarse-to-fine brain connectivity networks for each subject. Then, a triplet GCN (TGCN) module is developed to learn functional/structural representations of brain connectivity networks at each scale, with the triplet relationship among subjects explicitly incorporated into the learning process. Finally, we propose a template mutual learning strategy to train different scale TGCNs collaboratively for disease classification. Experimental results on 1, 160 subjects from three datasets with fMRI or dMRI data demonstrate that our MMTGCN outperforms several state-of-the-art methods in identifying three types of brain disorders.

Deep-learning methods (especially convolutional neural networks) using structural magnetic resonance imaging (sMRI) data have been successfully applied to computer-aided diagnosis (CAD) of Alzheimer's disease (AD) and its prodromal stage [i.e., mild cognitive impairment (MCI)]. As it is practically challenging to capture local and subtle disease-associated abnormalities directly from the whole-brain sMRI, most of those deep-learning approaches empirically preselect disease-associated sMRI brain regions for model construction. Considering that such isolated selection of potentially informative brain locations might be suboptimal, very few methods have been proposed to perform disease-associated discriminative region localization and disease diagnosis in a unified deep-learning framework. However, those methods based on task-oriented discriminative localization still suffer from two common limitations, that is: 1) identified brain locations are strictly consistent across all subjects, which ignores the unique anatomical characteristics of each brain and 2) only limited local regions/patches are used for model training, which does not fully utilize the global structural information provided by the whole-brain sMRI. In this article, we propose an attention-guided deep-learning framework to extract multilevel discriminative sMRI features for dementia diagnosis. Specifically, we first design a backbone fully convolutional network to automatically localize the discriminative brain regions in a weakly supervised manner. Using the identified disease-related regions as spatial attention guidance, we further develop a hybrid network to jointly learn and fuse multilevel sMRI features for CAD model construction. Our proposed method was evaluated on three public datasets (i.e., ADNI-1, ADNI-2, and AIBL), showing superior performance compared with several state-of-the-art methods in both tasks of AD diagnosis and MCI conversion prediction.

Fast and automated image quality assessment (IQA) for diffusion MR images is a crucial step for swiftly making a rescan decision during or after the scanning session. However, learning a model for this task is challenging as the number of annotated data is limited and the annotation labels might not always be correct. As a remedy, we will introduce in this paper an automatic image quality assessment (IQA) method based on hierarchical non-local residual networks for pediatric diffusion MR images. Our IQA is performed in three sequential stages, i.e., 1) slice-wise IQA, where a nonlocal residual network is first pre-trained to annotate each slice with an initial quality rating (i.e., pass/questionable/fail), which is subsequently refined via iterative semi-supervised learning and slice self-training; 2) volume-wise IQA, which agglomerates the features extracted from slices of a volume, and uses a nonlocal network to annotate the quality rating for each volume via iterative volume self-training; and 3) subject-wise IQA, which ensembles the volumetric IQA results to determine the overall image quality pertaining to a subject. Experimental results demonstrate that our method, trained using only samples of modest size, exhibits great generalizability, and is capable of conducting rapid hierarchical IQA with near-perfect accuracy.

The resting-state functional magnetic resonance imaging (rs-fMRI) reflects functional activity of brain regions by blood-oxygen-level dependent (BOLD) signals. Up to now, many computer-aided diagnosis methods based on rs-fMRI have been developed for Autism Spectrum Disorder (ASD). These methods are mostly the binary classification approaches to determine whether a subject is an ASD patient or not. However, the disease often consists of several sub-categories, which are complex and thus still confusing to many automatic classification methods. Besides, existing methods usually focus on the functional connectivity (FC) features in grey matter regions, which only account for a small portion of the rs-fMRI data. Recently, the possibility to reveal the connectivity information in the white matter regions of rs-fMRI has drawn high attention. To this end, we propose to use the patch-based functional correlation tensor (PBFCT) features extracted from rs-fMRI in white matter, in addition to the traditional FC features from gray matter, to develop a novel multi-class ASD diagnosis method in this work. Our method has two stages. Specifically, in the first stage of multi-source domain adaptation (MSDA), the source subjects belonging to multiple clinical centers (thus called as source domains) are all transformed into the same target feature space. Thus each subject in the target domain can be linearly reconstructed by the transformed subjects. In the second stage of multi-view sparse representation (MVSR), a multi-view classifier for multi-class ASD diagnosis is developed by jointly using both views of the FC and PBFCT features. The experimental results using the ABIDE dataset verify the effectiveness of our method, which is capable of accurately classifying each subject into a respective ASD sub-category.

Obtaining accurate segmentation of the prostate and nearby organs at risk (e.g., bladder and rectum) in CT images is critical for radiotherapy of prostate cancer. Currently, the leading automatic segmentation algorithms are based on Fully Convolutional Networks (FCNs), which achieve remarkable performance but usually need large-scale datasets with high-quality voxel-wise annotations for full supervision of the training. Unfortunately, such annotations are difficult to acquire, which becomes a bottleneck to build accurate segmentation models in real clinical applications. In this paper, we propose a novel weakly supervised segmentation approach that only needs 3D bounding box annotations covering the organs of interest to start the training. Obviously, the bounding box includes many non-organ voxels that carry noisy labels to mislead the segmentation model. To this end, we propose the label denoising module and embed it into the iterative training scheme of the label denoising network (LDnet) for segmentation. The labels of the training voxels are predicted by the tentative LDnet, while the label denoising module identifies the voxels with unreliable labels. As only the good training voxels are preserved, the iteratively re-trained LDnet can refine its segmentation capability gradually. Our results are remarkable, i.e., reaching $\sim$ 94% (prostate), $\sim$ 91% (bladder), and $\sim$ 86% (rectum) of the Dice Similarity Coefficients (DSCs), compared to the case of fully supervised learning upon high-quality voxel-wise annotations and also superior to several state-of-the-art approaches. To our best knowledge, this is the first work to achieve voxel-wise segmentation in CT images from simple 3D bounding box annotations, which can greatly reduce many labeling efforts and meet the demands of the practical clinical applications.

Accurate segmentation of organs at risk (OARs) from head and neck (H&N) CT images is crucial for effective H&N cancer radiotherapy. However, the existing deep learning methods are often not trained in an end-to-end fashion, i.e., they independently predetermine the regions of target organs before organ segmentation, causing limited information sharing between related tasks and thus leading to suboptimal segmentation results. Furthermore, when conventional segmentation network is used to segment all the OARs simultaneously, the results often favor big OARs over small OARs. Thus, the existing methods often train a specific model for each OAR, ignoring the correlation between different segmentation tasks. To address these issues, we propose a new multi-view spatial aggregation framework for joint localization and segmentation of multiple OARs using H&N CT images. The core of our framework is a proposed region-of-interest (ROI)-based fine-grained representation convolutional neural network (CNN), which is used to generate multi-OAR probability maps from each 2D view (i.e., axial, coronal, and sagittal view) of CT images. Specifically, our ROI-based fine-grained representation CNN (1) unifies the OARs localization and segmentation tasks and trains them in an end-to-end fashion, and (2) improves the segmentation results of various-sized OARs via a novel ROI-based fine-grained representation. Our multi-view spatial aggregation framework then spatially aggregates and assembles the generated multi-view multi-OAR probability maps to segment all the OARs simultaneously. We evaluate our framework using two sets of H&N CT images and achieve competitive and highly robust segmentation performance for OARs of various sizes.

Multi-modal neuroimages, such as magnetic resonance imaging (MRI) and positron emission tomography (PET), can provide complementary structural and functional information of the brain, thus facilitating automated brain disease identification. Incomplete data problem is unavoidable in multi-modal neuroimage studies due to patient dropouts and/or poor data quality. Conventional methods usually discard data-missing subjects, thus significantly reducing the number of training samples. Even though several deep learning methods have been proposed, they usually rely on pre-defined regions-of-interest in neuroimages, requiring disease-specific expert knowledge. To this end, we propose a spatially-constrained Fisher representation framework for brain disease diagnosis with incomplete multi-modal neuroimages. We first impute missing PET images based on their corresponding MRI scans using a hybrid generative adversarial network. With the complete (after imputation) MRI and PET data, we then develop a spatially-constrained Fisher representation network to extract statistical descriptors of neuroimages for disease diagnosis, assuming that these descriptors follow a Gaussian mixture model with a strong spatial constraint (i.e., images from different subjects have similar anatomical structures). Experimental results on three databases suggest that our method can synthesize reasonable neuroimages and achieve promising results in brain disease identification, compared with several state-of-the-art methods.

Early identification of dementia at the stage of mild cognitive impairment (MCI) is crucial for timely diagnosis and intervention of Alzheimer's disease (AD). Although several pioneering studies have been devoted to automated AD diagnosis based on resting-state functional magnetic resonance imaging (rs-fMRI), their performance is somewhat limited due to non-effective mining of spatial-temporal dependency. Besides, few of these existing approaches consider the explicit detection and modeling of discriminative brain regions (i.e., network hubs) that are sensitive to AD progression. In this paper, we propose a unique Spatial-Temporal convolutional-recurrent neural Network (STNet) for automated prediction of AD progression and network hub detection from rs-fMRI time series. Our STNet incorporates the spatial-temporal information mining and AD-related hub detection into an end-to-end deep learning model. Specifically, we first partition rs-fMRI time series into a sequence of overlapping sliding windows. A sequence of convolutional components are then designed to capture the local-to-global spatially-dependent patterns within each sliding window, based on which we are able to identify discriminative hubs and characterize their unique contributions to disease diagnosis. A recurrent component with long short-term memory (LSTM) units is further employed to model the whole-brain temporal dependency from the spatially-dependent pattern sequences, thus capturing the temporal dynamics along time. We evaluate the proposed method on 174 subjects with 563 rs-fMRI scans from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, with results suggesting the effectiveness of our method in both tasks of disease progression prediction and AD-related hub detection.

Hippocampus plays an important role in the memory and spatial navigation function of human brain. Study on its growth and change during first year of life would assist the investigation of early brain development as well as the biomarker for neurological disorders. With the help of Magnetic Resonance (MR) imaging techniques, infant brain at different development stage can be acquired with multiple imaging modalities. In this situation, the longitudinal segmentation of infant hippocampus is highly demanded and feasible for the clinical studies regarding to the hippocampal volume changes. However, since the brain structures undergo dynamic appearance, structural changes and various tissue contrast during the first year of life, substantial challenges will be imposed for ensuring the robustness and accuracy of automatic hippocampus segmentation algorithms. In addition, most of the existing hippocampus segmentation methods generally handle each brain development stage independently without considering the potential longitudinal consistency among different stages. In view of the above factors, we propose a longitudinal classification-regression model for segmenting hippocampus in infant brain MRIs. Generally, our model proceeds on a per-timepoint basis, guided by the output of latter timepoint towards the infant hippocampus in the previous timepoint. The key ingredient of our method is a combination of longitudinal context, static context and appearance learning strategies under the classification-regression forest architecture. Specifically, the longitudinal context is borrowed from the mask of prior-timepoint estimation and the static context is from the current-timepoint estimation. Furthermore, we implement the proposed model in a multi-scale and iterative manner to improve the efficiency and effectiveness. The proposed method is evaluated on segmenting infant hippocampi from T1-weighted brain MR images acquired at the age of 2 weeks, 3 months, 6 months, 9 months, and 12 months. Experimental results demonstrate that our method achieves better performance in segmentation accuracy over the state-of-the-art classification and regression random forest model.

Structural magnetic resonance imaging (sMRI) has been widely used for computer-aided diagnosis of neurodegenerative disorders, e.g., Alzheimer’s disease (AD), due to its sensitivity to morphological changes caused by brain atrophy. Recently, a few deep learning methods (e.g., convolutional neural networks, CNNs) have been proposed to learn task-oriented features from sMRI for AD diagnosis, and achieved superior performance than the conventional learning-based methods using hand-crafted features. However, these existing CNN-based methods still require the pre-determination of informative locations in sMRI. That is, the stage of discriminative atrophy localization is isolated to the latter stages of feature extraction and classifier construction. In this paper, we propose a hierarchical fully convolutional network (H-FCN) to automatically identify discriminative local patches and regions in the whole brain sMRI, upon which multi-scale feature representations are then jointly learned and fused to construct hierarchical classification models for AD diagnosis. Our proposed H-FCN method was evaluated on a large cohort of subjects from two independent datasets (i.e., ADNI-1 and ADNI-2), demonstrating good performance on joint discriminative atrophy localization and brain disease diagnosis.

Compared to computed tomography (CT), magnetic resonance imaging (MRI) delineation of craniomaxillofacial (CMF) bony structures can avoid harmful radiation exposure. However, bony boundaries are blurry in MRI, and structural information needs to be borrowed from CT during the training. This is challenging since paired MRI-CT data are typically scarce. In this paper, we propose to make full use of unpaired data, which are typically abundant, along with a single paired MRI-CT data to construct a one-shot generative adversarial model for automated MRI segmentation of CMF bony structures. Our model consists of a cross-modality image synthesis sub-network , which learns the mapping between CT and MRI, and an MRI segmentation sub-network . These two sub-networks are trained jointly in an end-to-end manner. Moreover, in the training phase, a neighbor-based anchoring method is proposed to reduce the ambiguity problem inherent in cross-modality synthesis, and a feature-matching-based semantic consistency constraint is proposed to encourage segmentation-oriented MRI synthesis. Experimental results demonstrate the superiority of our method both qualitatively and quantitatively in comparison with the state-of-the-art MRI segmentation methods.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder that is characterized by a wide range of symptoms. Identifying biomarkers for accurate diagnosis is crucial for early intervention of ASD. While multi-site data increase sample size and statistical power, they suffer from inter-site heterogeneity. To address this issue, we propose a multi-site adaption framework via low-rank representation decomposition (maLRR) for ASD identification based on functional MRI (fMRI). The main idea is to determine a common low-rank representation for data from the multiple sites, aiming to reduce differences in data distributions. Treating one site as a target domain and the remaining sites as source domains, data from these domains are transformed ( i.e. , adapted) to a common space using low-rank representation. To reduce data heterogeneity between the target and source domains, data from the source domains are linearly represented in the common space by those from the target domain. We evaluated the proposed method on both synthetic and real multi-site fMRI data for ASD identification. The results suggest that our method yields superior performance over several state-of-the-art domain adaptation methods.

Multi-atlas parcellation (MAP) is carried out on a brain image by propagating and fusing labelled regions from brain atlases. Typical nonlinear registration-based label propagation is time-consuming and sensitive to inter-subject differences. Recently, deep learning parcellation (DLP) has been proposed to avoid nonlinear registration for better efficiency and robustness than MAP. However, most existing DLP methods neglect using brain atlases, which contain high-level information (e.g., manually labelled brain regions), to provide auxiliary features for improving the parcellation accuracy. In this paper, we propose a novel multi-atlas DLP method for brain parcellation. Our method is based on fully convolutional networks (FCN) and squeeze-and-excitation (SE) modules. It can automatically and adaptively select features from the most relevant brain atlases to guide parcellation. Moreover, our method is trained via a generative adversarial network (GAN), where a convolutional neural network (CNN) with multi-scale $l_{1}$ loss is used as the discriminator. Benefiting from brain atlases, our method outperforms MAP and state-of-the-art DLP methods on two public image datasets (LPBA40 and NIREP-NA0).

As the population becomes older worldwide, accurate computer-aided diagnosis for Alzheimer's disease (AD) in the early stage has been regarded as a crucial step for neurodegeneration care in recent years. Since it extracts the low-level features from the neuroimaging data, previous methods regarded this computer-aided diagnosis as a classification problem that ignored latent featurewise relation. However, it is known that multiple brain regions in the human brain are anatomically and functionally interlinked according to the current neuroscience perspective. Thus, it is reasonable to assume that the extracted features from different brain regions are related to each other to some extent. Also, the complementary information between different neuroimaging modalities could benefit multimodal fusion. To this end, we consider leveraging the coupled interactions in the feature level and modality level for diagnosis in this paper. First, we propose capturing the feature-level coupled interaction using a coupled feature representation. Then, to model the modality-level coupled interaction, we present two novel methods: 1) the coupled boosting (CB) that models the correlation of pairwise coupled-diversity on both inconsistently and incorrectly classified samples between different modalities and 2) the coupled metric ensemble (CME) that learns an informative feature projection from different modalities by integrating the intrarelation and interrelation of training samples. We systematically evaluated our methods with the AD neuroimaging initiative data set. By comparison with the baseline learning-based methods and the state-of-the-art methods that are specially developed for AD/MCI (mild cognitive impairment) diagnosis, our methods achieved the best performance with accuracy of 95.0% and 80.7% (CB), 94.9% and 79.9% (CME) for AD/NC (normal control), and MCI/NC identification, respectively.

Precise segmentation of teeth from intra-oral scanner images is an essential task in computer-aided orthodontic surgical planning. The state-of-the-art deep learning-based methods often simply concatenate the raw geometric attributes (i.e., coordinates and normal vectors) of mesh cells to train a single-stream network for automatic intra-oral scanner image segmentation. However, since different raw attributes reveal completely different geometric information, the naive concatenation of different raw attributes at the (low-level) input stage may bring unnecessary confusion in describing and differentiating between mesh cells, thus hampering the learning of high-level geometric representations for the segmentation task. To address this issue, we design a two-stream graph convolutional network (i.e., TSGCN), which can effectively handle inter-view confusion between different raw attributes to more effectively fuse their complementary information and learn discriminative multi-view geometric representations. Specifically, our TSGCN adopts two input-specific graph-learning streams to extract complementary high-level geometric representations from coordinates and normal vectors, respectively. Then, these single-view representations are further fused by a self-attention module to adaptively balance the contributions of different views in learning more discriminative multi-view representations for accurate and fully automatic tooth segmentation. We have evaluated our TSGCN on a real-patient dataset of dental (mesh) models acquired by 3D intraoral scanners. Experimental results show that our TSGCN significantly outperforms state-of-the-art methods in 3D tooth (surface) segmentation. Github: https://github.com/ZhangLingMing1/TSGCNet., Comment: 11 pages, 6 figures. arXiv admin note: text overlap with arXiv:2012.13697

Missing data is a common problem in longitudinal studies due to subject dropouts and failed scans. We present a graph-based convolutional neural network to predict missing diffusion MRI data. In particular, we consider the relationships between sampling points in the spatial domain and the diffusion wave-vector domain to construct a graph. We then use a graph convolutional network to learn the non-linear mapping from available data to missing data. Our method harnesses a multi-scale residual architecture with adversarial learning for prediction with greater accuracy and perceptual quality. Experimental results show that our method is accurate and robust in the longitudinal prediction of infant brain diffusion MRI data.

Retrospective artifact correction (RAC) improves image quality post acquisition and enhances image usability. Recent machine learning driven techniques for RAC are predominantly based on supervised learning and therefore practical utility can be limited as data with paired artifact-free and artifact-corrupted images are typically insufficient or even non-existent. Here we show that unwanted image artifacts can be disentangled and removed from an image via an RAC neural network learned with unpaired data. This implies that our method does not require matching artifact-corrupted data to be either collected via acquisition or generated via simulation. Experimental results demonstrate that our method is remarkably effective in removing artifacts and retaining anatomical details in images with different contrasts.

Segmentation of infections from CT scans is important for accurate diagnosis and follow-up in tackling the COVID-19. Although the convolutional neural network has great potential to automate the segmentation task, most existing deep learning-based infection segmentation methods require fully annotated ground-truth labels for training, which is time-consuming and labor-intensive. This paper proposed a novel weakly supervised segmentation method for COVID-19 infections in CT slices, which only requires scribble supervision and is enhanced with the uncertainty-aware self-ensembling and transformation-consistent techniques. Specifically, to deal with the difficulty caused by the shortage of supervision, an uncertainty-aware mean teacher is incorporated into the scribble-based segmentation method, encouraging the segmentation predictions to be consistent under different perturbations for an input image. This mean teacher model can guide the student model to be trained using information in images without requiring manual annotations. On the other hand, considering the output of the mean teacher contains both correct and unreliable predictions, equally treating each prediction in the teacher model may degrade the performance of the student network. To alleviate this problem, the pixel level uncertainty measure on the predictions of the teacher model is calculated, and then the student model is only guided by reliable predictions from the teacher model. To further regularize the network, a transformation-consistent strategy is also incorporated, which requires the prediction to follow the same transformation if a transform is performed on an input image of the network. The proposed method has been evaluated on two public datasets and one local dataset. The experimental results demonstrate that the proposed method is more effective than other weakly supervised methods and achieves similar performance as those fully supervised.

The detection of amyloid-β (Aβ) deposition in the brain provides crucial evidence in the clinical diagnosis of Alzheimer’s disease (AD). However, the efficiency of the current PET-based brain Aβ examination suffers from both coarse, visual inspection-based bi-class stratification and high scanning cost and risks. In this work, we explored the feasibility of using non-invasive functional magnetic resonance imaging (fMRI) to predict Aβ-PET phenotypes in the AD continuum with graph learning on brain networks. First, three whole-brain Aβ-PET phenotypes were identified through clustering and their association with clinical phenotypes were investigated. Second, both conventional and high-order functional connectivity (FC) networks were constructed using resting-state fMRI and the network topological architectures were learned with graph convolutional networks (GCNs) to predict such Aβ-PET phenotypes. The experiment of Aβ-PET phenotype prediction on 258 samples from the AD continuum showed that our algorithm achieved a high fMRI-to-PET prediction accuracy (78.8%). The results demonstrated the existence of distinguishable brain Aβ deposition phenotypes in the AD continuum and the feasibility of using artificial intelligence and non-invasive brain imaging technique to approximate PET-based evaluations. It can be a promising technique for high-throughput screening of AD with less costs and restrictions.

With the advance of medical imaging technologies, multimodal images such as magnetic resonance images (MRI) and positron emission tomography (PET) can capture subtle structural and functional changes of brain, facilating the diagnosis of brain diseases such as Alzheimers disease (AD). In practice, multimodal images may be incomplete since PET is often missing due to high financial cost or availability. Most of existing methods simply excluded subjects with missing data, which unfortunately reduced sample size. In addition, how to extract and combine multimodal features is still challenging. To address these problems, we propose a deep learning framework to integrate a task-induced pyramid and attention generative adversarial network (TPA-GAN) with a pathwise transfer dense convolution network (PT-DCN) for imputation and also classification of multimodal brain images. First, we propose a TPA-GAN to integrate pyramid convolution and attention module as well as disease classification task into GAN for generating the missing PET data with their MRI. Then, with the imputed multimodal brain images, we build a dense convolution network with pathwise transfer blocks to gradually learn and combine multimodal features for final disease classification. Experiments are performed on ADNI-1 and ADNI-2 datasets to evaluate our proposed method, achiving superior performance in image imputation and brain disease diagnosis compared to state-of-the-art methods.

PURPOSE: The purpose of this study was to reduce the experience dependence during the orthognathic surgical planning that involves virtually simulating the corrective procedure for jaw deformities. METHODS: We introduce a geometric deep learning framework for generating reference facial bone shape models for objective guidance in surgical planning. First, we propose a surface deformation network to warp a patient's deformed bone to a set of normal bones for generating a dictionary of patient-specific normal bony shapes. Subsequently, sparse representation learning is employed to estimate a reference shape model based on the dictionary. RESULTS: We evaluated our method on a clinical dataset containing 24 patients, and compared it with a state-of-the-art method that relies on landmark-based sparse representation. Our method yields significantly higher accuracy than the competing method for estimating normal jaws and maintains the midfaces of patients’ facial bones as well as the conventional way. CONCLUSIONS: Experimental results indicate that our method generates accurate shape models that meet clinical standards.

The sliding-window-based dynamic functional connectivity network (D-FCN) has been becoming an increasingly useful tool for understanding the changes of brain connectivity patterns and the association of neurological diseases with these dynamic variations. However, conventional D-FCN is essentially low-order network, which only reflects the pairwise interaction pattern between brain regions and thus overlooking the high-order interactions among multiple brain regions. In addition, D-FCN is innate with temporal sensitivity issue, i.e., D-FCN is sensitive to the chronological order of its subnetworks. To deal with the above issues, we propose a novel high-order functional connectivity network framework based on the central moment feature of D-FCN. Specifically, we firstly adopt a central moment approach to extract multiple central moment feature matrices from D-FCN. Furthermore, we regard the matrices as the profiles to build multiple high-order functional connectivity networks which further capture the higher level and more complex interaction relationships among multiple brain regions. Finally, we use the voting strategy to combine the high-order networks with D-FCN for autism spectrum disorder diagnosis. Experimental results show that the combination of multiple functional connectivity networks achieves accuracy of 88.06%, and the best single network achieves accuracy of 79.5%.

Performance degradation due to domain shift remains a major challenge in medical image analysis. Unsupervised domain adaptation that transfers knowledge learned from the source domain with ground truth labels to the target domain without any annotation is the mainstream solution to resolve this issue. In this paper, we present a novel unsupervised domain adaptation framework for cross-modality cardiac segmentation, by explicitly capturing a common cardiac structure embedded across different modalities to guide cardiac segmentation. In particular, we first extract a set of 3D landmarks, in a self-supervised manner, to represent the cardiac structure of different modalities. The high-level structure information is then combined with another complementary feature, the Canny edges, to produce accurate cardiac segmentation results both in the source and target domains. We extensively evaluate our method on the MICCAI 2017 MM-WHS dataset for cardiac segmentation. The evaluation, comparison and comprehensive ablation studies demonstrate that our approach achieves satisfactory segmentation results and outperforms state-of-the-art unsupervised domain adaptation methods by a significant margin.

Ultrasound is a widely used technology for diagnosing developmental dysplasia of the hip (DDH) because it does not use radiation. Due to its low cost and convenience, 2-D ultrasound is still the most common examination in DDH diagnosis. In clinical usage, the complexity of both ultrasound image standardization and measurement leads to a high error rate for sonographers. The automatic segmentation results of key structures in the hip joint can be used to develop a standard plane detection method that helps sonographers decrease the error rate. However, current automatic segmentation methods still face challenges in robustness and accuracy. Thus, we propose a neonatal hip bone segmentation network (NHBS-Net) for the first time for the segmentation of seven key structures. We design three improvements, an enhanced dual attention module, a two-class feature fusion module, and a coordinate convolution output head, to help segment different structures. Compared with current state-of-the-art networks, NHBS-Net gains outstanding performance accuracy and generalizability, as shown in the experiments. Additionally, image standardization is a common need in ultrasonography. The ability of segmentation-based standard plane detection is tested on a 50-image standard dataset. The experiments show that our method can help healthcare workers decrease their error rate from 6%-10% to 2%. In addition, the segmentation performance in another ultrasound dataset (fetal heart) demonstrates the ability of our network.

Functional brain network (FBN), estimated with functional magnetic resonance imaging (fMRI), has become a potentially useful way of diagnosing neurological disorders in their early stages by comparing the connectivity patterns between different brain regions across subjects. However, this depends, to a great extent, on the quality of the estimated FBNs, indicating that FBN estimation is a key step for the subsequent task of disorder identification. In the past decades, researchers have developed many methods to estimate FBNs, including Pearson's correlation and (regularized) partial correlation, etc. Despite their widespread applications in current studies, most of the existing methods estimate FBNs only based on the dependency between the measured blood oxygen level dependent (BOLD) signals, which ignores spatial relationship of signals associated with different brain regions. Due to the space and material parsimony principle of our brain, we believe that the spatial distance between brain regions has an important influence on FBN topology. Therefore, in this paper, we assume that spatially neighboring brain regions tend to have stronger connections and/or share similar connections with others; based on this assumption, we propose two novel methods to estimate FBNs by incorporating the information of brain region distance into the estimation model. To validate the effectiveness of the proposed methods, we use the estimated FBNs to identify subjects with mild cognitive impairment (MCI) from normal controls (NCs). Experimental results show that the proposed methods are better than the baseline methods in the sense of MCI identification accuracy.

The ability to segment teeth precisely from digitized 3D dental models is an essential task in computer-aided orthodontic surgical planning. To date, deep learning based methods have been popularly used to handle this task. State-of-the-art methods directly concatenate the raw attributes of 3D inputs, namely coordinates and normal vectors of mesh cells, to train a single-stream network for fully-automated tooth segmentation. This, however, has the drawback of ignoring the different geometric meanings provided by those raw attributes. This issue might possibly confuse the network in learning discriminative geometric features and result in many isolated false predictions on the dental model. Against this issue, we propose a two-stream graph convolutional network (TSGCNet) to learn multi-view geometric information from different geometric attributes. Our TSGCNet adopts two graph-learning streams, designed in an input-aware fashion, to extract more discriminative high-level geometric representations from coordinates and normal vectors, respectively. These feature representations learned from the designed two different streams are further fused to integrate the multi-view complementary information for the cell-wise dense prediction task. We evaluate our proposed TSGCNet on a real-patient dataset of dental models acquired by 3D intraoral scanners, and experimental results demonstrate that our method significantly outperforms state-of-the-art methods for 3D shape segmentation.

BackgroundThe outbreak of coronavirus disease 2019 (COVID-19) causes tens of million infection world-wide. Many machine learning methods have been proposed for the computer-aided diagnosis between COVID-19 and community-acquired pneumonia (CAP) from chest computed tomography (CT) images. Most of these methods utilized the location-specific handcrafted features based on the segmentation results to improve the diagnose performance. However, the prerequisite segmentation step is time-consuming and needs the intervention by lots of expert radiologists, which cannot be achieved in the areas with limited medical resources.MethodsWe propose a generative adversarial feature completion and diagnosis network (GACDN) that simultaneously generates handcrafted features by radiomic counterparts and makes accurate diagnoses based on both original and generated features. Specifically, we first calculate the radiomic features from the CT images. Then, in order to fast obtain the location-specific handcrafted features, we use the proposed GACDN to generate them by its corresponding radiomic features. Finally, we use both radiomic features and location-specific handcrafted features for COVID-19 diagnosis.ResultsFor the performance of our generated location-specific handcrafted features, the results of four basic classifiers show that it has an average of 3.21% increase in diagnoses accuracy. Besides, the experimental results on COVID-19 dataset show that our proposed method achieved superior performance in COVID-19 vs. community acquired pneumonia (CAP) classification compared with the state-of-the-art methods.ConclusionsThe proposed method significantly improves the diagnoses accuracy of COVID-19 vs. CAP in the condition of incomplete location-specific handcrafted features. Besides, it is also applicable in some regions lacking of expert radiologists and high-performance computing resources.

Precise segmentation of infant brain MR images into gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF) are essential for studying neuroanatomical hallmarks of early brain development. However, for 6-month-old infants, the extremely low-intensity contrast caused by inherent myelination hinders accurate tissue segmentation. Existing convolutional neural networks (CNNs) based segmentation models for this task generally employ single-scale symmetric convolutions, which are inefficient for encoding the isointense tissue boundaries in baby brain images. Here, we propose a 3D mixed-scale asymmetric convolutional segmentation network (3D-MASNet) framework for brain MR images of 6-month-old infants. We replaced the traditional convolutional layer of an existing to-be-trained network with a 3D mixed-scale convolution block consisting of asymmetric kernels (MixACB) during the training phase and then equivalently converted it into the original network. Five canonical CNN segmentation models were evaluated using both T1- and T2-weighted images of 23 6-month-old infants from iSeg-2019 datasets, which contained manual labels as ground truth. MixACB significantly enhanced the average accuracy of all five models and obtained the most considerable improvement in the fully convolutional network model (CC-3D-FCN) and the highest performance in the Dense U-Net model. This approach further obtained Dice coefficient accuracies of 0.931, 0.912, and 0.961 in GM, WM, and CSF, respectively, ranking first among 30 teams on the validation dataset of the iSeg-2019 Grand Challenge. Thus, the proposed 3D-MASNet can improve the accuracy of existing CNNs-based segmentation models as a plug-and-play solution that offers a promising technique for future infant brain MRI studies.

Tumor segmentation in 3D automated breast ultrasound (ABUS) plays an important role in breast disease diagnosis and surgical planning. However, automatic segmentation of tumors in 3D ABUS images is still challenging, due to the large tumor shape and size variations, and uncertain tumor locations among patients. In this paper, we develop a novel cross-model attention-guided tumor segmentation network with a hybrid loss for 3D ABUS images. Specifically, we incorporate the tumor location into a segmentation network by combining an improved 3D Mask R-CNN head into V-Net as an end-to-end architecture. Furthermore, we introduce a cross-model attention mechanism that is able to aggregate the segmentation probability map from the improved 3D Mask R-CNN to each feature extraction level in the V-Net. Then, we design a hybrid loss to balance the contribution of each part in the proposed cross-model segmentation network. We conduct extensive experiments on 170 3D ABUS from 107 patients. Experimental results show that our method outperforms other state-of-the-art methods, by achieving the Dice similarity coefficient (DSC) of 64.57%, Jaccard coefficient (JC) of 53.39%, recall (REC) of 64.43%, precision (PRE) of 74.51%, 95th Hausdorff distance (95HD) of 11.91 mm, and average surface distance (ASD) of 4.63 mm. Our code will be available online (https://github.com/zhouyuegithub/CMVNet).

Highlights • A multi-task multi-instance learning framework is proposed to jointly assess the severity of the COVID-19 patients and segment lung lobe. • A unique hierarchical multi-instance learning strategy is developed to predict the severity of patients in a weakly supervised manner for 3D CT images. • An embedding-level multi-instance learning method is proposed to predict labels beyond the instance-level. • The proposed method achieving promising results in severity assessment in a real-world COVID-19 dataset compared to several state-of-the-art methods., Understanding chest CT imaging of the coronavirus disease 2019 (COVID-19) will help detect infections early and assess the disease progression. Especially, automated severity assessment of COVID-19 in CT images plays an essential role in identifying cases that are in great need of intensive clinical care. However, it is often challenging to accurately assess the severity of this disease in CT images, due to variable infection regions in the lungs, similar imaging biomarkers, and large inter-case variations. To this end, we propose a synergistic learning framework for automated severity assessment of COVID-19 in 3D CT images, by jointly performing lung lobe segmentation and multi-instance classification. Considering that only a few infection regions in a CT image are related to the severity assessment, we first represent each input image by a bag that contains a set of 2D image patches (with each cropped from a specific slice). A multi-task multi-instance deep network (called M2UNet) is then developed to assess the severity of COVID-19 patients and also segment the lung lobe simultaneously. Our M2UNet consists of a patch-level encoder, a segmentation sub-network for lung lobe segmentation, and a classification sub-network for severity assessment (with a unique hierarchical multi-instance learning strategy). Here, the context information provided by segmentation can be implicitly employed to improve the performance of severity assessment. Extensive experiments were performed on a real COVID-19 CT image dataset consisting of 666 chest CT images, with results suggesting the effectiveness of our proposed method compared to several state-of-the-art methods.

Image classification is an essential and challenging task in computer vision. Despite its prevalence, the combination of the deep convolutional neural network (DCNN) and the Fisher vector (FV) encoding method has limited performance since the class-irrelevant background used in the traditional FV encoding may result in less discriminative image features. In this paper, we propose the foreground FV (fgFV) encoding algorithm and its fast approximation for image classification. We try to separate implicitly the class-relevant foreground from the class-irrelevant background during the encoding process via tuning the weights of the partial gradients corresponding to each Gaussian component under the supervision of image labels and, then, use only those local descriptors extracted from the class-relevant foreground to estimate FVs. We have evaluated our fgFV against the widely used FV and improved FV (iFV) under the combined DCNN-FV framework and also compared them to several state-of-the-art image classification approaches on ten benchmark image datasets for the recognition of fine-grained natural species and artificial manufactures, categorization of course objects, and classification of scenes. Our results indicate that the proposed fgFV encoding algorithm can construct more discriminative image presentations from local descriptors than FV and iFV, and the combined DCNN-fgFV algorithm can improve the performance of image classification.

The fusion of complementary information contained in multi-modality data [ e.g. , magnetic resonance imaging (MRI), positron emission tomography (PET), and genetic data] has advanced the progress of automated Alzheimer’s disease (AD) diagnosis. However, multi-modality based AD diagnostic models are often hindered by the missing data, i.e. , not all the subjects have complete multi-modality data. One simple solution used by many previous studies is to discard samples with missing modalities. However, this significantly reduces the number of training samples, thus leading to a sub-optimal classification model. Furthermore, when building the classification model, most existing methods simply concatenate features from different modalities into a single feature vector without considering their underlying associations. As features from different modalities are often closely related ( e.g. , MRI and PET features are extracted from the same brain region), utilizing their inter-modality associations may improve the robustness of the diagnostic model. To this end, we propose a novel latent representation learning method for multi-modality based AD diagnosis. Specifically, we use all the available samples (including samples with incomplete modality data) to learn a latent representation space. Within this space, we not only use samples with complete multi-modality data to learn a common latent representation, but also use samples with incomplete multi-modality data to learn independent modality-specific latent representations. We then project the latent representations to the label space for AD diagnosis. We perform experiments using 737 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database, and the experimental results verify the effectiveness of our proposed method.

Diffusion MRI (DMRI) is a powerful tool for studying early brain development and disorders. However, the typically low spatio-angular resolution of DMRI diminishes structural details and limits quantitative analysis to simple diffusion models. This problem is aggravated for infant DMRI since (i) the infant brain is significantly smaller than that of an adult, demanding higher spatial resolution to capture subtle structures; and (ii) the typically limited scan time of unsedated infants poses significant challenges to DMRI acquisition with high spatio-angular resolution. Post-acquisition super-resolution (SR) is an important alternative for increasing the resolution of DMRI data without prolonging acquisition times. However, most existing methods focus on the SR of only either the spatial domain (x-space) or the diffusion wavevector domain (q-space). For more effective resolution enhancement, we propose a framework for joint SR in both spatial and wavevector domains. More specifically, we first establish the signal relationships in x-q space using a robust neighborhood matching technique. We then harness the signal relationships to regularize the ill-posed inverse problem associated with the recovery of high-resolution data from their low-resolution counterpart. Extensive experiments on synthetic, adult, and infant DMRI data demonstrate that our method is able to recover high-resolution DMRI data with remarkably improved quality.

Groupwise registration aligns a set of images to a common space. It can however be inefficient and ineffective when dealing with datasets with significant anatomical variations. To mitigate these problems, we propose a groupwise registration framework based on hierarchical multi-level and multi-resolution shrinkage of a graph set. First, to deal with datasets with complex inhomogeneous image distributions, we divide the images hierarchically into multiple clusters. Since the images in each cluster have similar appearances, they can be registered effectively. Second, we employ a multi-resolution strategy to reduce computational cost. Experimental results on two public datasets show that our proposed method yields state-of-the-art registration accuracy with significantly reduced computational time.

Backgroud and Objective: X-ray angiography, a powerful technique for blood vessel visualization, is widely used for interventional diagnosis of coronary artery disease because of its fast imaging speed and perspective inspection ability. Matching feature points in angiographic images is a considerably challenging task due to repetitive weak-textured regions. Methods: In this paper, we propose an angiographic image matching method based on the hierarchical dense matching framework, where a novel deep feature descriptor is designed to compute multilevel correlation maps. In particular, the deep feature descriptor is computed by a deep learning model specifically designed and trained for angiographic images, thereby making the correlation maps more distinctive for corresponding feature points in different angiographic images. Moreover, point correspondences are further hierarchically extracted from multilevel correlation maps with the highest similarity response(s), which is relatively robust and accurate. To overcome the problem regarding the lack of training samples, the convolutional neural network (designed for deep feature descriptor) is initially trained on samples from natural images and then fine-tuned on manually annotated angiographic images. Finally, a dense matching completion method, based on the distance between deep feature descriptors, is proposed to generate dense matches between images. Results: The proposed method has been evaluated on the number and accuracy of extracted matches and the performance of subtraction images. Experiments on a variety of angiographic images show promising matching accuracy, compared with state-of-the-art methods. Conclusions: The proposed angiographic image matching method is shown to be accurate and effective for feature matching in angiographic images, and further achieves good performance in image subtraction.

Brain functional networks (BFNs) constructed from resting-state functional magnetic resonance imaging (rs-fMRI) have been widely applied to the analysis and diagnosis of brain diseases, such as Alzheimer's disease and its prodrome, namely mild cognitive impairment (MCI). Constructing a meaningful brain network based on, for example, sparse representation (SR) is the most essential step prior to the subsequent analysis or disease identification. However, the independent coding process of SR fails to capture the intrinsic locality and similarity characteristics in the data. To address this problem, we propose a novel weighted graph (Laplacian) regularized SR framework, based on which BFN can be optimized by considering both intrinsic correlation similarity and local manifold structure in the data, as well as sparsity prior of the brain connectivity. Additionally, the non-convergence of the graph Laplacian in the self-representation model has been solved properly. Combined with a pipeline of sparse feature selection and classification, the effectiveness of our proposed method is demonstrated by identifying MCI based on the constructed BFNs.

Positron emission tomography (PET) has been substantially used recently. To minimize the potential health risk caused by the tracer radiation inherent to PET scans, it is of great interest to synthesize the high-quality PET image from the low-dose one to reduce the radiation exposure. In this paper, we propose a 3D auto-context-based locality adaptive multi-modality generative adversarial networks model (LA-GANs) to synthesize the high-quality FDG PET image from the low-dose one with the accompanying MRI images that provide anatomical information. Our work has four contributions. First, different from the traditional methods that treat each image modality as an input channel and apply the same kernel to convolve the whole image, we argue that the contributions of different modalities could vary at different image locations, and therefore a unified kernel for a whole image is not optimal. To address this issue, we propose a locality adaptive strategy for multi-modality fusion. Second, we utilize $1 \times 1 \times 1$ kernel to learn this locality adaptive fusion so that the number of additional parameters incurred by our method is kept minimum. Third, the proposed locality adaptive fusion mechanism is learned jointly with the PET image synthesis in a 3D conditional GANs model, which generates high-quality PET images by employing large-sized image patches and hierarchical features. Fourth, we apply the auto-context strategy to our scheme and propose an auto-context LA-GANs model to further refine the quality of synthesized images. Experimental results show that our method outperforms the traditional multi-modality fusion methods used in deep networks, as well as the state-of-the-art PET estimation approaches.

Using multi-atlas registration (MAR), information carried by atlases can be transferred onto a new input image for the tasks of region of interest (ROI) segmentation, anatomical landmark detection, and so on. Conventional atlases used in MAR methods are monomodal and contain only normal anatomical structures. Therefore, the majority of MAR methods cannot handle input multimodal pathological images, which are often collected in routine image-based diagnosis. This is because registering monomodal atlases with normal appearances to multimodal pathological images involves two major problems: (1) missing imaging modalities in the monomodal atlases, and (2) influence from pathological regions. In this paper, we propose a new MAR framework to tackle these problems. In this framework, a deep learning based image synthesizers are applied for synthesizing multimodal normal atlases from conventional monomodal normal atlases. To reduce the influence from pathological regions, we further propose a multimodal low-rank approach to recover multimodal normal-looking images from multimodal pathological images. Finally, the multimodal normal atlases can be registered to the recovered multimodal images in a multi-channel way. We evaluate our MAR framework via brain ROI segmentation of multimodal tumor brain images. Due to the utilization of multimodal information and the reduced influence from pathological regions, experimental results show that registration based on our method is more accurate and robust, leading to significantly improved brain ROI segmentation compared with state-of-the-art methods.

Accurate segmentation of pelvic organs is important for prostate radiation therapy. Modern radiation therapy starts to use a magnetic resonance image (MRI) as an alternative to computed tomography image because of its superior soft tissue contrast and also free of risk from radiation exposure. However, segmentation of pelvic organs from MRI is a challenging problem due to inconsistent organ appearance across patients and also large intrapatient anatomical variations across treatment days. To address such challenges, we propose a novel deep network architecture, called "Spatially varying sTochastic Residual AdversarIal Network" (STRAINet), to delineate pelvic organs from MRI in an end-to-end fashion. Compared to the traditional fully convolutional networks (FCN), the proposed architecture has two main contributions: 1) inspired by the recent success of residual learning, we propose an evolutionary version of the residual unit, i.e., stochastic residual unit, and use it to the plain convolutional layers in the FCN. We further propose long-range stochastic residual connections to pass features from shallow layers to deep layers; and 2) we propose to integrate three previously proposed network strategies to form a new network for better medical image segmentation: a) we apply dilated convolution in the smallest resolution feature maps, so that we can gain a larger receptive field without overly losing spatial information; b) we propose a spatially varying convolutional layer that adapts convolutional filters to different regions of interest; and c) an adversarial network is proposed to further correct the segmented organ structures. Finally, STRAINet is used to iteratively refine the segmentation probability maps in an autocontext manner. Experimental results show that our STRAINet achieved the state-of-the-art segmentation accuracy. Further analysis also indicates that our proposed network components contribute most to the performance.

Sparse representation-based brain functional network modeling often results in large inter-subject variability in the network structure. This could reduce the statistical power in group comparison, or even deteriorate the generalization capability of the individualized diagnosis of brain diseases. Although group sparse representation (GSR) can alleviate such a limitation by increasing network similarity across subjects, it could, in turn, fail in providing satisfactory separability between the subjects from different groups (e.g., patients vs. controls). In this study, we propose to integrate individual functional connectivity (FC) information into the GSR-based network construction framework to achieve higher between-group separability while maintaining the merit of within-group consistency. Our method was based on an observation that the subjects from the same group have generally more similar FC patterns than those from different groups. To this end, we propose our new method, namely “strength and similarity guided GSR (SSGSR)”, which exploits both BOLD signal temporal correlation-based “low-order” FC (LOFC) and inter-subject LOFC-profile similarity-based “high-order” FC (HOFC) as two priors to jointly guide the GSR-based network modeling. Extensive experimental comparisons are carried out, with the rs-fMRI data from mild cognitive impairment (MCI) subjects and healthy controls, between the proposed algorithm and other state-of-the-art brain network modeling approaches. Individualized MCI identification results show that our method could achieve a balance between the individually consistent brain functional network construction and the adequately maintained inter-group brain functional network distinctions, thus leading to a more accurate classification result. Our method also provides a promising and generalized solution for the future connectome-based individualized diagnosis of brain disease.

Accurate segmentation of infant brain images into different regions of interest is one of the most important fundamental steps in studying early brain development. In the isointense phase (approximately 6–8 months of age), white matter and gray matter exhibit similar levels of intensities in magnetic resonance (MR) images, due to the ongoing myelination and maturation. This results in extremely low tissue contrast and thus makes tissue segmentation very challenging. Existing methods for tissue segmentation in this isointense phase usually employ patch-based sparse labeling on single modality. To address the challenge, we propose a novel 3-D multimodal fully convolutional network (FCN) architecture for segmentation of isointense phase brain MR images. Specifically, we extend the conventional FCN architectures from 2-D to 3-D, and, rather than directly using FCN, we intuitively integrate coarse (naturally high-resolution) and dense (highly semantic) feature maps to better model tiny tissue regions, in addition, we further propose a transformation module to better connect the aggregating layers; we also propose a fusion module to better serve the fusion of feature maps. We compare the performance of our approach with several baseline and state-of-the-art methods on two sets of isointense phase brain images. The comparison results show that our proposed 3-D multimodal FCN model outperforms all previous methods by a large margin in terms of segmentation accuracy. In addition, the proposed framework also achieves faster segmentation results compared to all other methods. Our experiments further demonstrate that: 1) carefully integrating coarse and dense feature maps can considerably improve the segmentation performance; 2) batch normalization can speed up the convergence of the networks, especially when hierarchical feature aggregations occur; and 3) integrating multimodal information can further boost the segmentation performance.

Accurate segmentation of infant brain magnetic resonance (MR) images into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) is an indispensable foundation for early studying of brain growth patterns and morphological changes in neurodevelopmental disorders. Nevertheless, in the isointense phase (approximately 6-9 months of age), due to inherent myelination and maturation process, WM and GM exhibit similar levels of intensity in both T1-weighted (T1w) and T2-weighted (T2w) MR images, making tissue segmentation very challenging. Despite many efforts were devoted to brain segmentation, only few studies have focused on the segmentation of 6-month infant brain images. With the idea of boosting methodological development in the community, iSeg-2017 challenge (http://iseg2017.web.unc.edu) provides a set of 6-month infant subjects with manual labels for training and testing the participating methods. Among the 21 automatic segmentation methods participating in iSeg-2017, we review the 8 top-ranked teams, in terms of Dice ratio, modified Hausdorff distance and average surface distance, and introduce their pipelines, implementations, as well as source codes. We further discuss limitations and possible future directions. We hope the dataset in iSeg-2017 and this review article could provide insights into methodological development for the community.