Search

Background: Deformable registration is required to generate a time-integrated activity (TIA) map which is essential for voxel-based dosimetry. The conventional iterative registration algorithm using anatomical images (e.g., computed tomography (CT)) could result in registration errors in functional images (e.g., single photon emission computed tomography (SPECT) or positron emission tomography (PET)). Various deep learning-based registration tools have been proposed, but studies specifically focused on the registration of serial hybrid images were not found., Purpose: In this study, we introduce CoRX-NET, a novel unsupervised deep learning network designed for deformable registration of hybrid medical images. The CoRX-NET structure is based on the Swin-transformer (ST), allowing for the representation of complex spatial connections in images. Its self-attention mechanism aids in the effective exchange and integration of information across diverse image regions. To augment the amalgamation of SPECT and CT features, cross-stitch layers have been integrated into the network., Methods: Two different 177 Lu DOTATATE SPECT/CT datasets were acquired at different medical centers. 22 sets from Seoul National University and 14 sets from Sunway Medical Centre are used for training/internal validation and external validation respectively. The CoRX-NET architecture builds upon the ST, enabling the modeling of intricate spatial relationships within images. To further enhance the fusion of SPECT and CT features, cross-stitch layers have been incorporated within the network. The network takes a pair of SPECT/CT images (e.g., fixed and moving images) and generates a deformed SPECT/CT image. The performance of the network was compared with Elastix and TransMorph using L1 loss and structural similarity index measure (SSIM) of CT, SSIM of normalized SPECT, and local normalized cross correlation (LNCC) of SPECT as metrics. The voxel-wise root mean square errors (RMSE) of TIA were compared among the different methods., Results: The ablation study revealed that cross-stitch layers improved SPECT/CT registration performance. The cross-stitch layers notably enhance SSIM (internal validation: 0.9614 vs. 0.9653, external validation: 0.9159 vs. 0.9189) and LNCC of normalized SPECT images (internal validation: 0.7512 vs. 0.7670, external validation: 0.8027 vs. 0.8027). CoRX-NET with the cross-stitch layer achieved superior performance metrics compared to Elastix and TransMorph, except for CT SSIM in the external dataset. When qualitatively analyzed for both internal and external validation cases, CoRX-NET consistently demonstrated superior SPECT registration results. In addition, CoRX-NET accomplished SPECT/CT image registration in less than 6 s, whereas Elastix required approximately 50 s using the same PC's CPU. When employing CoRX-NET, it was observed that the voxel-wise RMSE values for TIA were approximately 27% lower for the kidney and 33% lower for the tumor, compared to when Elastix was used., Conclusion: This study represents a major advancement in achieving precise SPECT/CT registration using an unsupervised deep learning network. It outperforms conventional methods like Elastix and TransMorph, reducing uncertainties in TIA maps for more accurate dose assessments., (© 2024 The Author(s). Medical Physics published by Wiley Periodicals LLC on behalf of American Association of Physicists in Medicine.)

Lung image registration can effectively describe the relative motion of lung tissues, thereby helping to solve series problems in clinical applications. Since the lungs are soft and fairly passive organs, they are influenced by respiration and heartbeat, resulting in discontinuity of lung motion and large deformation of anatomic features. This poses great challenges for accurate registration of lung image and its applications. The recent application of deep learning (DL) methods in the field of medical image registration has brought promising results. However, a versatile registration framework has not yet emerged due to diverse challenges of registration for different regions of interest (ROI). DL-based image registration methods used for other ROI cannot achieve satisfactory results in lungs. In addition, there are few review articles available on DL-based lung image registration. In this review, the development of conventional methods for lung image registration is briefly described and a more comprehensive survey of DL-based methods for lung image registration is illustrated. The DL-based methods are classified according to different supervision types, including fully-supervised, weakly-supervised and unsupervised. The contributions of researchers in addressing various challenges are described, as well as the limitations of these approaches. This review also presents a comprehensive statistical analysis of the cited papers in terms of evaluation metrics and loss functions. In addition, publicly available datasets for lung image registration are also summarized. Finally, the remaining challenges and potential trends in DL-based lung image registration are discussed., Competing Interests: Declaration of competing interest The authors declare no conflicts of interest to the manuscript content., (Copyright © 2023 Elsevier Ltd. All rights reserved.)

Purpose: To evaluate deep learning (DL)-based deformable image registration (DIR) for dose accumulation during radiotherapy of prostate cancer patients., Methods and Materials: Data including 341 CBCTs (209 daily, 132 weekly) and 23 planning CTs from 23 patients was retrospectively analyzed. Anatomical deformation during treatment was estimated using free-form deformation (FFD) method from Elastix and DL-based VoxelMorph approaches. The VoxelMorph method was investigated using anatomical scans (VMorph_Sc) or label images (VMorph_Msk), or the combination of both (VMorph_Sc_Msk). Accumulated doses were compared with the planning dose., Results: The DSC ranges, averaged for prostate, rectum and bladder, were 0.60-0.71, 0.67-0.79, 0.93-0.98, and 0.89-0.96 for the FFD, VMorph_Sc, VMorph_Msk, and VMorph_Sc_Msk methods, respectively. When including both anatomical and label images, VoxelMorph estimated more complex deformations resulting in heterogeneous determinant of Jacobian and higher percentage of deformation vector field (DVF) folding (up to a mean value of 1.90% in the prostate). Large differences were observed between DL-based methods regarding estimation of the accumulated dose, showing systematic overdosage and underdosage of the bladder and rectum, respectively. The difference between planned mean dose and accumulated mean dose with VMorph_Sc_Msk reached a median value of +6.3 Gy for the bladder and -5.1 Gy for the rectum., Conclusion: The estimation of the deformations using DL-based approach is feasible for male pelvic anatomy but requires the inclusion of anatomical contours to improve organ correspondence. High variability in the estimation of the accumulated dose depending on the deformable strategy suggests further investigation of DL-based techniques before clinical deployment., (© 2023 The Authors. Journal of Applied Clinical Medical Physics published by Wiley Periodicals, LLC on behalf of The American Association of Physicists in Medicine.)

Objective . In radiotherapy, the internal movement of organs between treatment sessions causes errors in the final radiation dose delivery. To assess the need for adaptation, motion models can be used to simulate dominant motion patterns and assess anatomical robustness before delivery. Traditionally, such models are based on principal component analysis (PCA) and are either patient-specific (requiring several scans per patient) or population-based, applying the same set of deformations to all patients. We present a hybrid approach which, based on population data, allows to predict patient-specific inter-fraction variations for an individual patient. Approach . We propose a deep learning probabilistic framework that generates deformation vector fields warping a patient's planning computed tomography (CT) into possible patient-specific anatomies. This daily anatomy model (DAM) uses few random variables capturing groups of correlated movements. Given a new planning CT, DAM estimates the joint distribution over the variables, with each sample from the distribution corresponding to a different deformation. We train our model using dataset of 312 CT pairs with prostate, bladder, and rectum delineations from 38 prostate cancer patients. For 2 additional patients (22 CTs), we compute the contour overlap between real and generated images, and compare the sampled and 'ground truth' distributions of volume and center of mass changes. Results . With a DICE score of 0.86 ± 0.05 and a distance between prostate contours of 1.09 ± 0.93 mm, DAM matches and improves upon previously published PCA-based models, using as few as 8 latent variables. The overlap between distributions further indicates that DAM's sampled movements match the range and frequency of clinically observed daily changes on repeat CTs. Significance . Conditioned only on planning CT values and organ contours of a new patient without any pre-processing, DAM can accurately deformations seen during following treatment sessions, enabling anatomically robust treatment planning and robustness evaluation against inter-fraction anatomical changes., (Creative Commons Attribution license.)

Purpose: Recently, two-dimensional-to-three-dimensional (2D-3D) deformable registration has been applied to deform liver tumor contours from prior reference images onto estimated cone-beam computed tomography (CBCT) target images to automate on-board tumor localizations. Biomechanical modeling has also been introduced to fine-tune the intra-liver deformation-vector-fields (DVFs) solved by 2D-3D deformable registration, especially at low-contrast regions, using tissue elasticity information and liver boundary DVFs. However, the caudal liver boundary shows low contrast from surrounding tissues in the cone-beam projections, which degrades the accuracy of the intensity-based 2D-3D deformable registration there and results in less accurate boundary conditions for biomechanical modeling. We developed a deep-learning (DL)-based method to optimize the liver boundary DVFs after 2D-3D deformable registration to further improve the accuracy of subsequent biomechanical modeling and liver tumor localization., Methods: The DL-based network was built based on the U-Net architecture. The network was trained in a supervised fashion to learn motion correlation between cranial and caudal liver boundaries to optimize the liver boundary DVFs. Inputs of the network had three channels, and each channel featured the 3D DVFs estimated by the 2D-3D deformable registration along one Cartesian direction (x, y, z). To incorporate patient-specific liver boundary information into the DVFs, the DVFs were masked by a liver boundary ring structure generated from the liver contour of the prior reference image. The network outputs were the optimized DVFs along the liver boundary with higher accuracy. From these optimized DVFs, boundary conditions were extracted for biomechanical modeling to further optimize the solution of intra-liver tumor motion. We evaluated the method using 34 liver cancer patient cases, with 24 for training and 10 for testing. We evaluated and compared the performance of three methods: 2D-3D deformable registration, 2D-3D-Bio (2D-3D deformable registration with biomechanical modeling), and DL-Bio (DL model prediction with biomechanical modeling). The tumor localization errors were quantified through calculating the center-of-mass-errors (COMEs), DICE coefficients, and Hausdorff distance between deformed liver tumor contours and manually segmented "gold-standard" contours., Results: The predicted DVFs by the DL model showed improved accuracy at the liver boundary, which translated into more accurate liver tumor localizations through biomechanical modeling. On a total of 90 evaluated images and tumor contours, the average (± sd) liver tumor COMEs of the 2D-3D, 2D-3D-Bio, and DL-Bio techniques were 4.7 ± 1.9 mm, 2.9 ± 1.0 mm, and 1.7 ± 0.4 mm. The corresponding average (± sd) DICE coefficients were 0.60 ± 0.12, 0.71 ± 0.07, and 0.78 ± 0.03; and the average (± sd) Hausdorff distances were 7.0 ± 2.6 mm, 5.4 ± 1.5 mm, and 4.5 ± 1.3 mm, respectively., Conclusion: DL-Bio solves a general correlation model to improve the accuracy of the DVFs at the liver boundary. With improved boundary conditions, the accuracy of biomechanical modeling can be further increased for accurate intra-liver low-contrast tumor localization., (© 2021 American Association of Physicists in Medicine.)

Background and Objectives: Deep learning is being increasingly used for deformable image registration and unsupervised approaches, in particular, have shown great potential. However, the registration of abdominopelvic Computed Tomography (CT) images remains challenging due to the larger displacements compared to those in brain or prostate Magnetic Resonance Imaging datasets that are typically considered as benchmarks. In this study, we investigate the use of the commonly used unsupervised deep learning framework VoxelMorph for the registration of a longitudinal abdominopelvic CT dataset acquired in patients with bone metastases from breast cancer., Methods: As a pre-processing step, the abdominopelvic CT images were refined by automatically removing the CT table and all other extra-corporeal components. To improve the learning capabilities of the VoxelMorph framework when only a limited amount of training data is available, a novel incremental training strategy is proposed based on simulated deformations of consecutive CT images in the longitudinal dataset. This devised training strategy was compared against training on simulated deformations of a single CT volume. A widely used software toolbox for deformable image registration called NiftyReg was used as a benchmark. The evaluations were performed by calculating the Dice Similarity Coefficient (DSC) between manual vertebrae segmentations and the Structural Similarity Index (SSIM)., Results: The CT table removal procedure allowed both VoxelMorph and NiftyReg to achieve significantly better registration performance. In a 4-fold cross-validation scheme, the incremental training strategy resulted in better registration performance compared to training on a single volume, with a mean DSC of 0.929±0.037 and 0.883±0.033, and a mean SSIM of 0.984±0.009 and 0.969±0.007, respectively. Although our deformable image registration method did not outperform NiftyReg in terms of DSC (0.988±0.003) or SSIM (0.995±0.002), the registrations were approximately 300 times faster., Conclusions: This study showed the feasibility of deep learning based deformable registration of longitudinal abdominopelvic CT images via a novel incremental training strategy based on simulated deformations., Competing Interests: Declaration of Competing Interest All authors in this paper do not have any conflict of interest., (Copyright © 2021 The Author(s). Published by Elsevier B.V. All rights reserved.)

Background: Image registration is a challenging problem in many clinical tasks, but deep learning has made significant progress in this area over the past few years. Real‐time and robust registration has been made possible by supervised transformation estimation. However, the quality of registrations using this framework depends on the quality of ground truth labels such as displacement field. Purpose: To propose a simple and reliable method for registering medical images based on image structure similarity in a completely unsupervised manner. Methods: We proposed a deep cascade unsupervised deformable registration approach to align images without reliable clinical data labels. Our basic network was composed of a displacement estimation module (ResUnet) and a deformation module (spatial transformer layers). We adopted l2$l_2$‐norm to regularize the deformation field instead of the traditional l1$l_1$‐norm regularization. Additionally, we utilized structural similarity (ssim) estimation during the training stage to enhance the structural consistency between the deformed images and the reference images. Results: Experiments results indicated that by incorporating ssim loss, our cascaded methods not only achieved higher dice score of 0.9873, ssim score of 0.9559, normalized cross‐correlation (NCC) score of 0.9950, and lower relative sum of squared difference (SSD) error of 0.0313 on CT images, but also outperformed the comparative methods on ultrasound dataset. The statistical t$t$‐test results also proved that these improvements of our method have statistical significance. Conclusions: In this study, the promising results based on diverse evaluation metrics have demonstrated that our model is simple and effective in deformable image registration (DIR). The generalization ability of the model was also verified through experiments on liver CT images and cardiac ultrasound images. [ABSTRACT FROM AUTHOR]

Image registration is a primary task in various medical image analysis applications. However, cardiac image registration is difficult due to the large non-rigid deformation of the heart and the complex anatomical structure. This paper proposes a structure-aware independently trained multi-scale registration network (SIMReg) to address this challenge. Using image pairs of different resolutions, independently train each registration network to extract image features of large deformation image pairs at different resolutions. In the testing stage, the large deformation registration is decomposed into a multi-scale registration process, and the deformation fields of different resolutions are fused by a step-by-step deformation method, thus solving the difficulty of directly processing large deformation. Meanwhile, the targeted introduction of MIND (modality independent neighborhood descriptor) structural features to guide network training enhances the registration of cardiac structural contours and improves the registration effect of local details. Experiments were carried out on the open cardiac dataset ACDC (automated cardiac diagnosis challenge), and the average Dice value of the experimental results of the proposed method was 0.833. Comparative experiments showed that the proposed SIMReg could better solve the problem of heart image registration and achieve a better registration effect on cardiac images. [ABSTRACT FROM AUTHOR]

Lung image registration is more challenging than other organs. This is because the breath of the human body causes large deformations in the lung parenchyma and small deformations in tissues such as the pulmonary vascular. Many studies have recently used multi-resolution networks to solve the lung registration problem. However, they use the same structure of registration modules on each level, which makes it difficult to handle complex and small deformations. We propose an unsupervised heterogeneous multi-resolution network (UHMR-Net) to overcome the above problem. The image detail registration module (IDRM) is designed on the highest resolution level. Within this module, the cascaded network is used on the same resolution image to continuously learn the "remaining" detail deformation fields. The shallow shrinkage loss (SS-Loss) is designed to supervise the cascaded network, thus further improving the ability of the network to handle small deformations. Moreover, with the lightweight feature local correlation layer we proposed, the image boundary registration module (IBRM), on multiple low-resolution levels, can better solve the large deformation registration problem. The target registration error on the public DIR-Lab 4DCT dataset was 1.56 ± 1.39 mm, which was significantly better than the classic conventional methods and advanced deep-based methods. [ABSTRACT FROM AUTHOR]

PurposeMissing or discrepant imaging volume is a common challenge in deformable image registration (DIR). To minimize the adverse impact, we train a neural network to synthesize cropped portions of head and neck CT's and then test its use in DIR.MethodsUsing a training dataset of 409 head and neck CT's, we trained a generative adversarial network to take in a cropped 3D image and output an image with synthesized anatomy in the cropped region. The network used a 3D U-Net generator along with Visual Geometry Group (VGG) deep feature losses. To test our technique, for each of the 53 test volumes, we used Elastix to deformably register combinations of a randomly cropped, full, and synthetically full volume to a single cropped, full, and synthetically full target volume. We additionally tested our method's robustness to crop extent by progressively increasing the amount of cropping, synthesizing the missing anatomy using our network, and then performing the same registration combinations. Registration performance was measured using 95% Hausdorff distance across 16 contours.ResultsWe successfully trained a network to synthesize missing anatomy in superiorly and inferiorly cropped images. The network can estimate large regions in an incomplete image, far from the cropping boundary. Registration using our estimated full images was not significantly different from registration using the original full images. The average contour matching error for full image registration was 9.9 mm, whereas our method was 11.6, 12.1, and 13.6 mm for synthesized-to-full, full-to-synthesized, and synthesized-to-synthesized registrations, respectively. In comparison, registration using the cropped images had errors of 31.7 mm and higher. Plotting the registered image contour error as a function of initial preregistered error shows that our method is robust to registration difficulty. Synthesized-to-full registration was statistically independent of cropping extent up to 18.7 cm superiorly cropped. Synthesized-to-synthesized registration was nearly independent, with a -0.04 mm of change in average contour error for every additional millimeter of cropping.ConclusionsDifferent or inadequate in scan extent is a major cause of DIR inaccuracies. We address this challenge by training a neural network to complete cropped 3D images. We show that with image completion, the source of DIR inaccuracy is eliminated, and the method is robust to varying crop extent.

Accurate registration of lung X-rays is an important task in medical image analysis. However, the conventional methods usually cost a lot in running time, and the existing deep learning methods are hard to deal with the large deformation caused by respiratory and cardiac motion. In this paper, we attempt to use deep learning methods to deal with large deformation and enable it to achieve the accuracy of conventional methods. We proposed the cascading affine and B-spline network (CABN), which consists of convolutional cross-stitch affine block (CCAB) and B-splines U-net-like block (BUB) for large lung motion. CCAB makes use of the convolutional cross-stitch model to learn global features among images. And BUB adopts the idea of cubic B-splines which is suitable for large deformation. We separately demonstrated CCAB, BUB, and CABN on two chest X-ray datasets. The experimental results indicate that our methods are highly competitive both in accuracy and runtime when compared to both other deep learning methods and iterative conventional approaches. Moreover, CCAB also can be used for the preprocessing of non-rigid registration methods, replacing affine in conventional methods. [ABSTRACT FROM AUTHOR]

To address the low precision of pairwise registration method based on the deep learning and the time-consuming nature of traditional registration algorithm, this paper presents a method of unsupervised end-to-end groupwise registration based on variational inference, as well as a registration framework based on normalized cross correlation (NCC) and prior knowledge. The framework can warp all images in the group into a common space and effectively control the deformation field of the regularization, and it doesn't need a real deformation field or a reference image. The estimation of deformation field by this method can be modeled as a probability generation model and solved by variational inference. Then unsupervised training is implemented with the help of spatial transformer network and loss function. The registration results of 3D brain magnetic resonance image from the public data set LPBA40 show that: compared with the baseline method, the proposed method has better Dice score, less running time, better diffeomorphisms domain, and is robust to noise.

Organ shape reconstruction based on a single-projection image during treatment has wide clinical scope, e.g., in image-guided radiotherapy and surgical guidance. We propose an image-to-graph convolutional network that achieves deformable registration of a three-dimensional (3D) organ mesh for a low-contrast two-dimensional (2D) projection image. This framework enables simultaneous training of two types of transformation: from the 2D projection image to a displacement map, and from the sampled per-vertex feature to a 3D displacement that satisfies the geometrical constraint of the mesh structure. Assuming application to radiation therapy, the 2D/3D deformable registration performance is verified for multiple abdominal organs that have not been targeted to date, i.e., the liver, stomach, duodenum, and kidney, and for pancreatic cancer. The experimental results show shape prediction considering relationships among multiple organs can be used to predict respiratory motion and deformation from digitally reconstructed radiographs with clinically acceptable accuracy. [ABSTRACT FROM AUTHOR]

Deformable medical image registration is an important component of medical image analysis. Transformer networks have recently received more attention due to their ability to improve image registration by utilizing long-range spatial relationships. However, these approaches often struggle to accurately align images compromised by artifacts, typically caused by patients' organ movement during scans across various diagnostic devices. To solve this challenge, this study introduces a triple-stream architecture that enhances organ alignment accuracy while maintaining the anatomical structure. Our proposed approach incorporates a cascade-based cross-attention technique within a Tri-UNet framework, allowing for the continuous integration of moving, fixed, and new moving images. The technique addresses the optimal alignment and extensive motion artifacts often encountered in clinical imaging. To ensure that the anatomical structure remains maintained during deformations, we optimized the framework by integrating motion correction techniques that generated the final deformation ϕ (DVF) by the maintaining anatomical structure from the multi-scale deformation of two primary sources: moving-fixed and new moving-fixed. This technique significantly aligns image pairs and reduces motion artifacts from deformation fields, resulting in stable and accurate deformation. The results show that the proposed methods achieve state-of-the-art performance compared with other learning-based methods. Our proposed method achieved an average Dice similarity coefficient of 0.787 and 0.726 on Open Access Series of Imaging Studies (OASIS) and LONI Probabilistic Brain Atlas (LPBA40), respectively. [ABSTRACT FROM AUTHOR]

Diabetic retinopathy is one of the most dangerous complications of diabetes. It affects the eyes causing damage to the blood vessels of the retina. Eventually, as the disease develops, it is possible to lose sight. The main cure for this pathology is based on the early detection which plays a crucial role in slowing the progress of the underlying disease and protecting many patients from losing their sight. However, the detection of diabetic retinopathy at its early stages remains an arduous task that requires human expert interpretation of fundus images in order to vigilantly follow-up the patient. In this paper, we shall propose a new automatic diabetic retinopathy detection method that based on deep-learning. The aforementioned approach is composed of two main steps: an initial pre-processing step where the deformable registration is applied on the retina to occupy the entire image and eliminate the effect of the background on the classification process. The second step is the classification phase in which we train four convolutional neural networks (CNN) models (Densenet-121, Xception, Inception-v3, Resnet-50) to detect the stage of diabetic retinopathy. The performance of our proposed architecture has been tested on the APTOS 2019 dataset. As the latter is relatively small, a transfer learning is adopted by pre-training the mentioned CNNs on the ImageNet dataset and fine-tuning them on the APTOS dataset. In the testing phase, the final prediction is obtained by a system of voting based on the output of the four convolutional neural networks. Our model has performed an accuracy of 85.28% in the testing phase. [ABSTRACT FROM AUTHOR]

Medical image segmentation and registration are two tasks to analyze the anatomical structures in clinical research. Still, deep-learning solutions utilizing the connections between segmentation and registration remain underdiscovered. This article designs a joint learning framework named RSegNet that can realize concurrent deformable registration and segmentation by minimizing an integrated loss function, including three parts: diffeomorphic registration loss, segmentation similarity loss, and dual-consistency supervision loss. The probabilistic diffeomorphic registration branch could benefit from the auxiliary segmentations available from the segmentation branch to achieve anatomical consistency and better deformation regularity by dual-consistency supervision. Simultaneously, the segmentation performance could also be improved by data augmentation based on the registration with well-behaved diffeomorphic guarantees. Experiments on the human brain 3-D magnetic resonance images have been implemented to demonstrate the effectiveness of our approach. We trained and validated RSegNet with 1000 images and tested its performances on four public datasets, which shows that our method successfully yields concurrent improvements of both segmentation and registration compared with separately trained networks. Specifically, our method can increase the accuracy of segmentation and registration by 7.0% and 1.4%, respectively, in terms of Dice scores. Note to Practitioners—Registration and segmentation of medical images are two significant tasks in medical research and clinical application. However, most existing approaches consider these two tasks independently while neglecting the potential association between them. Therefore, we suggest a new approach that combines these two tasks into one joint deep learning framework, boosting registration, and segmentation performance by introducing dual-consistency supervision. Besides, our framework could generate outputs within 1 s by taking an affinely aligned medical image pair as input, which is suitable for time-critical requirements in a clinic. We tested it on four public datasets and achieved state-of-the-art performance to demonstrate the proposed method’s feasibility and robustness. Furthermore, our proposed RSegNet is a general learning framework suitable for various image modalities and anatomical structures. Hence, we expect our framework to serve as a practical clinical tool to speed up medical image analysis procedures and improve diagnostic accuracy. [ABSTRACT FROM AUTHOR]

Deformable registration is fundamental to longitudinal and population-based image analyses. However, it is challenging to precisely align longitudinal infant brain MR images of the same subject, as well as cross-sectional infant brain MR images of different subjects, due to fast brain development during infancy. In this paper, we propose a recurrently usable deep neural network for the registration of infant brain MR images. There are three main highlights of our proposed method. (i) We use brain tissue segmentation maps for registration, instead of intensity images, to tackle the issue of rapid contrast changes of brain tissues during the first year of life. (ii) A single registration network is trained in a one-shot manner, and then recurrently applied in inference for multiple times, such that the complex deformation field can be recovered incrementally. (iii) We also propose both the adaptive smoothing layer and the tissue-aware anti-folding constraint into the registration network to ensure the physiological plausibility of estimated deformations without degrading the registration accuracy. Experimental results, in comparison to the state-of-the-art registration methods, indicate that our proposed method achieves the highest registration accuracy while still preserving the smoothness of the deformation field. The implementation of our proposed registration network is available online https://github.com/Barnonewdm/ACTA-Reg-Net. [ABSTRACT FROM AUTHOR]

Image-guided ablation therapy has been widely used as a minimally invasive treatment for hepatic malignant tumors. Image registration is very useful during such operations, especially for multimode tumor ablation therapy. This research proposes a novel idea for performing MR-CT registration in multimode tumor ablation therapy based on liver segmentation. First, the liver is segmented from the preoperative 3D MR image and the intraoperative 3D CT image using a deep learning method based on a modified UNet++ architecture. Then, the preoperative MR image and the intraoperative CT image are coregistered using rigid and nonrigid registration methods with the segmented liver as the region of interest. The segmented binary images, rather than gray-level images, are aligned in the rigid registration step, which proves to be faster and more accurate than the registration method based on gray information. For the nonrigid registration, a multilevel free-form deformation method is applied to correct tiny misalignments. Finally, our method was validated using clinical data from 15 patients. The proposed method achieved an average Dice coefficient and target registration error of 93.36±1.21% and 4.42±2.35 mm, respectively, and it can help interventional radiologists adjust the probe position in clinical work.

Purpose: The deformable registration of 3D chest computed tomography (CT) images is one of the most important tasks in the field of medical image registration. However, the nonlinear deformation and large‐scale displacement of lung tissues caused by respiratory motion cause great challenges in the deformable registration of 3D lung CT images. Materials and methods: We proposed an end‐to‐end fast registration method based on unsupervised learning, optimized the classic U‐Net, and added inception modules between skip connections. The inception module attempts to capture and merge information at different spatial scales to generate a high‐precision dense displacement vector field. To solve the problem of voxel folding in flexible registration, we put the Jacobian regularization term into the loss function to directly penalize the singularity of the displacement field during training to ensure a smooth displacement vector field. In the stage of data preprocessing, we segmented the lung fields to eliminate the interference of irrelevant information in the network during training. The existing publicly available datasets cannot implement model training. To alleviate the problem of overfitting caused by limited data resources being available, we proposed a data augmentation method based on the 3D‐TPS (3D thin plate spline) transform to expand the training data. Results: Compared with the experimental results obtained by using the VoxelMorph deep learning method and registration packages, such as ANTs and Elastix, we achieved a competitive target registration error of 2.09 mm, an optimal Dice score of 0.987, and almost no folding voxels. Additionally, the proposed method was much faster than the traditional methods. Conclusions: In this study, we have shown that the proposed method was efficient in 3D chest image registration. The promising results demonstrated that our method showed strong robustness in the deformable registration of 3D chest CT images. [ABSTRACT FROM AUTHOR]

In the last two years learning-based methods have started to show encouraging results in different supervised and unsupervised medical image registration tasks. Deep neural networks enable (near) real time applications through fast inference times and have tremendous potential for increased registration accuracies by task-specific learning. However, estimation of large 3D deformations, for example present in inhale to exhale lung CT or interpatient abdominal MRI registration, is still a major challenge for the widely adopted U-Net-like network architectures. Even when using multi-level strategies, current state-of-the-art DL registration results do not yet reach the high accuracy of conventional frameworks. To overcome the problem of large deformations for deep learning approaches, in this work, we present GraphRegNet, a sparse keypoint-based geometric network for dense deformable medical image registration. Similar to the successful 2D optical flow estimation of FlowNet or PWC-Net we leverage discrete dense displacement maps to facilitate the registration process. In order to cope with enormously increasing memory requirements when working with displacement maps in 3D medical volumes and to obtain a well-regularised and accurate deformation field we 1) formulate the registration task as the prediction of displacement vectors on a sparse irregular grid of distinctive keypoints and 2) introduce our efficient GraphRegNet for displacement regularisation, a combination of convolutional and graph neural network layers in a unified architecture. In our experiments on exhale to inhale lung CT registration we demonstrate substantial improvements (TRE below 1.4 mm) over other deep learning methods. Our code is publicly available at https://github.com/multimodallearning/graphregnet. [ABSTRACT FROM AUTHOR]

Medical image segmentation and registration have vital roles in computer-assisted diagnosis procedures, challenging tasks suffering from various limitations and artifacts inside images. Recently, deep learning techniques accomplish these two tasks and achieve outstanding performances. However, most deep learning-based methods overlook the potential correlation between each other. In this paper, an unsupervised probabilistic model named U-RSNet is proposed to realize concurrent medical image registration and segmentation in one framework. Specifically, the unsupervised segmentation branch is derived from Bayesian inference. The prior warped atlas for segmentation can be obtained by deforming a known probabilistic atlas by the corresponding invertible deformation field with a well-behaved diffeomorphic guarantee, which can perfectly integrate these two tasks to form a complete intelligent prediction system. In this case, the segmentation performance could be largely improved based on the warped probabilistic atlas obtained from the registration branch. Experiments on human brain 3D magnetic resonance images have demonstrated the effectiveness of our approach. We trained and validated U-RSNet with 1000 images and tested its performances on four public datasets. We showed our method successfully realized concurrent segmentation and registration and yielded better segmentation results than a separately trained network. [ABSTRACT FROM AUTHOR]

Objective: In image-guided neurosurgery, co-registered preoperative anatomical, functional, and diffusion tensor imaging can be used to facilitate a safe resection of brain tumors in eloquent areas of the brain. However, the brain deforms during surgery, particularly in the presence of tumor resection. Non-Rigid Registration (NRR) of the preoperative image data can be used to create a registered image that captures the deformation in the intraoperative image while maintaining the quality of the preoperative image. Using clinical data, this paper reports the results of a comparison of the accuracy and performance among several non-rigid registration methods for handling brain deformation. A new adaptive method that automatically removes mesh elements in the area of the resected tumor, thereby handling deformation in the presence of resection is presented. To improve the user experience, we also present a new way of using mixed reality with ultrasound, MRI, and CT.Materials and methods: This study focuses on 30 glioma surgeries performed at two different hospitals, many of which involved the resection of significant tumor volumes. An Adaptive Physics-Based Non-Rigid Registration method (A-PBNRR) registers preoperative and intraoperative MRI for each patient. The results are compared with three other readily available registration methods: a rigid registration implemented in 3D Slicer v4.4.0; a B-Spline non-rigid registration implemented in 3D Slicer v4.4.0; and PBNRR implemented in ITKv4.7.0, upon which A-PBNRR was based. Three measures were employed to facilitate a comprehensive evaluation of the registration accuracy: (i) visual assessment, (ii) a Hausdorff Distance-based metric, and (iii) a landmark-based approach using anatomical points identified by a neurosurgeon.Results: The A-PBNRR using multi-tissue mesh adaptation improved the accuracy of deformable registration by more than five times compared to rigid and traditional physics based non-rigid registration, and four times compared to B-Spline interpolation methods which are part of ITK and 3D Slicer. Performance analysis showed that A-PBNRR could be applied, on average, in

Image registration and segmentation are the two most studied problems in medical image analysis. Deep learning algorithms have recently gained a lot of attention due to their success and state-of-the-art results in variety of problems and communities. In this paper, we propose a novel, efficient, and multi-task algorithm that addresses the problems of image registration and brain tumor segmentation jointly. Our method exploits the dependencies between these tasks through a natural coupling of their interdependencies during inference. In particular, the similarity constraints are relaxed within the tumor regions using an efficient and relatively simple formulation. We evaluated the performance of our formulation both quantitatively and qualitatively for registration and segmentation problems on two publicly available datasets (BraTS 2018 and OASIS 3), reporting competitive results with other recent state-of-the-art methods. Moreover, our proposed framework reports significant amelioration (p < 0.005) for the registration performance inside the tumor locations, providing a generic method that does not need any predefined conditions (e.g., absence of abnormalities) about the volumes to be registered. Our implementation is publicly available online at https://github.com/TheoEst/joint_registration_tumor_segmentation.

Objective: To improve the efficiency of computed tomography (CT)‐magnetic resonance (MR) deformable image registration while ensuring the registration accuracy. Methods: Two fully convolutional networks (FCNs) for generating spatial deformable grids were proposed using the Cycle‐Consistent method to ensure the deformed image consistency with the reference image data. In all, 74 pelvic cases consisting of both MR and CT images were studied, among which 64 cases were used as training data and 10 cases as the testing data. All training data were standardized and normalized, following simple image preparation to remove the redundant air. Dice coefficients and average surface distance (ASD) were calculated for regions of interest (ROI) of CT‐MR image pairs, before and after the registration. The performance of the proposed method (FCN with Cycle‐Consistent) was compared with that of Elastix software, MIM software, and FCN without cycle‐consistent. Results: The results show that the proposed method achieved the best performance among the four registration methods tested in terms of registration accuracy and the method was more stable than others in general. In terms of average registration time, Elastix took 64 s, MIM software took 28 s, and the proposed method was found to be significantly faster, taking <0.1 s. Conclusion: The proposed method not only ensures the accuracy of deformable image registration but also greatly reduces the time required for image registration and improves the efficiency of the registration process. In addition, compared with other deep learning methods, the proposed method is completely unsupervised and end‐to‐end. [ABSTRACT FROM AUTHOR]

Image registration and segmentation are the two most studied problems in medical image analysis. Deep learning algorithms have recently gained a lot of attention due to their success and state-of-the-art results in variety of problems and communities. In this paper, we propose a novel, efficient, and multi-task algorithm that addresses the problems of image registration and brain tumor segmentation jointly. Our method exploits the dependencies between these tasks through a natural coupling of their interdependencies during inference. In particular, the similarity constraints are relaxed within the tumor regions using an efficient and relatively simple formulation. We evaluated the performance of our formulation both quantitatively and qualitatively for registration and segmentation problems on two publicly available datasets (BraTS 2018 and OASIS 3), reporting competitive results with other recent state-of-the-art methods. Moreover, our proposed framework reports significant amelioration (p < 0.005) for the registration performance inside the tumor locations, providing a generic method that does not need any predefined conditions (e.g., absence of abnormalities) about the volumes to be registered. Our implementation is publicly available online at https://github.com/TheoEst/joint_registration_tumor_segmentation. [ABSTRACT FROM AUTHOR]

We introduce CartiMorph, a framework for automated knee articular cartilage morphometrics. It takes an image as input and generates quantitative metrics for cartilage subregions, including the percentage of full-thickness cartilage loss (FCL), mean thickness, surface area, and volume. CartiMorph leverages the power of deep learning models for hierarchical image feature representation. Deep learning models were trained and validated for tissue segmentation, template construction, and template-to-image registration. We established methods for surface-normal-based cartilage thickness mapping, FCL estimation, and rule-based cartilage parcellation. Our cartilage thickness map showed less error in thin and peripheral regions. We evaluated the effectiveness of the adopted segmentation model by comparing the quantitative metrics obtained from model segmentation and those from manual segmentation. The root-mean-squared deviation of the FCL measurements was less than 8%, and strong correlations were observed for the mean thickness (Pearson's correlation coefficient ρ ∈ [ 0. 82 , 0. 97 ]), surface area (ρ ∈ [ 0. 82 , 0. 98 ]) and volume (ρ ∈ [ 0. 89 , 0. 98 ]) measurements. We compared our FCL measurements with those from a previous study and found that our measurements deviated less from the ground truths. We observed superior performance of the proposed rule-based cartilage parcellation method compared with the atlas-based approach. CartiMorph has the potential to promote imaging biomarkers discovery for knee osteoarthritis. [Display omitted] • We proposed an automatic and quantitative deep-learning-based image processing pipeline for knee articular cartilage morphometrics. • We established a method for automated cartilage thickness mapping. • We established a method for automated full-thickness cartilage loss estimation. • We established a rule-based cartilage parcellation method. [ABSTRACT FROM AUTHOR]

We propose to learn a low-dimensional probabilistic deformation model from data which can be used for the registration and the analysis of deformations. The latent variable model maps similar deformations close to each other in an encoding space. It enables to compare deformations, to generate normal or pathological deformations for any new image, or to transport deformations from one image pair to any other image. Our unsupervised method is based on the variational inference. In particular, we use a conditional variational autoencoder network and constrain transformations to be symmetric and diffeomorphic by applying a differentiable exponentiation layer with a symmetric loss function. We also present a formulation that includes spatial regularization such as the diffusion-based filters. In addition, our framework provides multi-scale velocity field estimations. We evaluated our method on 3-D intra-subject registration using 334 cardiac cine-MRIs. On this dataset, our method showed the state-of-the-art performance with a mean DICE score of 81.2% and a mean Hausdorff distance of 7.3 mm using 32 latent dimensions compared to three state-of-the-art methods while also demonstrating more regular deformation fields. The average time per registration was 0.32 s. Besides, we visualized the learned latent space and showed that the encoded deformations can be used to transport deformations and to cluster diseases with a classification accuracy of 83% after applying a linear projection. [ABSTRACT FROM AUTHOR]

Significance: Analysis of modern large-scale, multicenter or diseased data requires deformable registration algorithms that can cope with data of diverse nature. Objective: We propose a novel deformable registration method, which is based on a cue-aware deep regression network, to deal with multiple databases with minimal parameter tuning. Methods: Our method learns and predicts the deformation field between a reference image and a subject image. Specifically, given a set of training images, our method learns the displacement vector associated with a pair of reference–subject patches. To achieve this, we first introduce a key-point truncated-balanced sampling strategy to facilitate accurate learning from the image database of limited size. Then, we design a cue-aware deep regression network, where we propose to employ the contextual cue, i.e., the scale-adaptive local similarity, to more apparently guide the learning process. The deep regression network is aware of the contextual cue for accurate prediction of local deformation. Results and Conclusion: Our experiments show that the proposed method can tackle various registration tasks on different databases, giving consistent good performance without the need of manual parameter tuning, which could be applicable to various clinical applications. [ABSTRACT FROM AUTHOR]

• A comprehensive review on the literature of medical image registration using deep neural networks is presented. • The review is systematic and encompasses all the related works previously published in the field. • Key concepts, statistical analysis, confining challenges, novelties, techniques, and prospective trends are surveyed. • This review allows a deep insight for the readers active in the field who are investigating the state-of-the-art. Image-guided interventions are saving the lives of a large number of patients where the image registration should indeed be considered as the most complex and complicated issue to be tackled. On the other hand, a huge progress in the field of machine learning has recently made by the possibility of implementing deep neural networks on the contemporary many-core GPUs. It has opened up a promising window to challenge with many medical applications in more efficient and effective ways, where the registration is not an exception. In this paper, a comprehensive review on the state-of-the-art literature known as medical image registration using deep neural networks is presented. The review is systematic and encompasses all the related works previously published in the field. Key concepts, statistical analysis from different points of view, confining challenges, novelties and main contributions, key-enabling techniques, future directions, and prospective trends all are discussed and surveyed in details in this comprehensive review. This review allows a deep understanding and insight for the readers active in the field who are investigating the state-of-the-art and seeking to contribute the future literature. Image, graphical abstract [ABSTRACT FROM AUTHOR]