Search

Objective. In computer-assisted minimally invasive surgery, the intraoperative x-ray image is enhanced by overlapping it with a preoperative CT volume to improve visualization of vital anatomical structures. Therefore, accurate and robust 3D/2D registration of CT volume and x-ray image is highly desired in clinical practices. However, previous registration methods were prone to initial misalignments and struggled with local minima, leading to issues of low accuracy and vulnerability. Approach. To improve registration performance, we propose a novel CT/x-ray image registration agent (CT2X-IRA) within a task-driven deep reinforcement learning framework, which contains three key strategies: (1) a multi-scale-stride learning mechanism provides multi-scale feature representation and flexible action step size, establishing fast and globally optimal convergence of the registration task. (2) A domain adaptation module reduces the domain gap between the x-ray image and digitally reconstructed radiograph projected from the CT volume, decreasing the sensitivity and uncertainty of the similarity measurement. (3) A weighted reward function facilitates CT2X-IRA in searching for the optimal transformation parameters, improving the estimation accuracy of out-of-plane transformation parameters under large initial misalignments. Main results. We evaluate the proposed CT2X-IRA on both the public and private clinical datasets, achieving target registration errors of 2.13 mm and 2.33 mm with the computation time of 1.5 s and 1.1 s, respectively, showing an accurate and fast workflow for CT/x-ray image rigid registration. Significance. The proposed CT2X-IRA obtains the accurate and robust 3D/2D registration of CT and x-ray images, suggesting its potential significance in clinical applications., (© 2023 Institute of Physics and Engineering in Medicine.)

Objective. Smoke, uneven lighting, and color deviation are common issues in endoscopic surgery, which have increased the risk of surgery and even lead to failure. Approach. In this study, we present a new physics model driven semi-supervised learning framework for high-quality pixel-wise endoscopic image enhancement, which is generalizable for smoke removal, light adjustment, and color correction. To improve the authenticity of the generated images, and thereby improve the network performance, we integrated specific physical imaging defect models with the CycleGAN framework. No ground-truth data in pairs are required. In addition, we propose a transfer learning framework to address the data scarcity in several endoscope enhancement tasks and improve the network performance. Main results. Qualitative and quantitative studies reveal that the proposed network outperforms the state-of-the-art image enhancement methods. In particular, the proposed method performs much better than the original CycleGAN, for example, the structural similarity improved from 0.7925 to 0.8648, feature similarity for color images from 0.8917 to 0.9283, and quaternion structural similarity from 0.8097 to 0.8800 in the smoke removal task. Experimental results of the proposed transfer learning method also reveal its superior performance when trained with small datasets of target tasks. Significance. Experimental results on endoscopic images prove the effectiveness of the proposed network in smoke removal, light adjustment, and color correction, showing excellent clinical usefulness., (© 2022 Institute of Physics and Engineering in Medicine.)

Motion tracking techniques can revise the bias arising from respiration-caused motion in radiation therapy. Tracking key structures accurately and at a real-time speed is necessary for effective motion tracking. In this work, we propose a fusion Siamese network with drift correction for target tracking in ultrasound sequences. Specifically, the network fuses four response maps generated by the cross-correlation between convolution layers at different resolutions to reduce up-sampling error. A correction strategy combining local structural similarity and target trajectory is proposed to revise the target drift predicted by the network. Moreover, a coarse-to-fine strategy is proposed to train the network with a limited number of annotated images, in which an augmented dataset is generated by corner points to learn network features with high generalizability. The proposed method is evaluated on the basis of the public dataset of the MICCAI 2015 Challenge on Liver UltraSound Tracking (CLUST) and our ultrasound image dataset, which is provided by the Chinese People's Liberation Army General Hospital (CPLAGH). A tracking error of 0.80 ± 1.16 mm is observed for 85 targets across 39 ultrasound sequences in the CLUST dataset. A tracking error of 0.61 ± 0.36 mm is observed for 20 targets across 10 ultrasound sequences in the CPLAGH dataset. The effectiveness of the proposed fusion and correction strategies is verified via two ablation experiments. Overall, the experimental results demonstrate the effectiveness of the proposed fusion Siamese network with drift correction and reveal its potential in clinical practice., (© 2022 Institute of Physics and Engineering in Medicine.)

Vessel centerline extraction from x-ray angiography images is essential for vessel structure analysis in the diagnosis of coronary artery disease. However, complete and continuous centerline extraction remains a challenging task due to image noise, poor contrast, and complexity of vessel structure. Thus, an iterative multi-path search framework for automatic vessel centerline extraction is proposed. First, the seed points of the vessel structure are detected and sorted by confidence. With the ordered seed points, multi-bifurcation centerline is searched through multi-path propagation of wavefront and accumulated voting. Finally, the centerline is further extended piecewise by wavefront propagation on the basis of keypoint detection. The latter two steps are performed alternately to obtain the final centerline result. The proposed method is qualitatively and quantitatively evaluated on 1260 synthetic images and 50 clinical angiography images. The results demonstrate that our method has a highF1score of 87.8% ± 2.7% for the angiography images and achieves accurate and continuous results of vessel centerline extraction., (© 2021 Institute of Physics and Engineering in Medicine.)

Three-dimensional (3D) vessel segmentation can provide full spatial information about an anatomic structure to help physicians gain increased understanding of vascular structures, which plays an utmost role in many medical image-processing and analysis applications. The purpose of this paper aims to develop a 3D vessel-segmentation method that can improve segmentation accuracy in 3D ultrasound (US) images. We propose a 3D tensor-based active contour model method for accurate 3D vessel segmentation. With our method, the contrast-independent multiscale bottom-hat tensor representation and local-global information are captured. This strategy ensures the effective extraction of the boundaries of vessels from inhomogeneous and homogeneous regions without being affected by the noise and low-contrast of the 3D US images. Experimental results in clinical 3D US and public 3D Multiphoton Microscopy datasets are used for quantitative and qualitative comparison with several state-of-the-art vessel segmentation methods. Clinical experiments demonstrate that our method can achieve a smoother and more accurate boundary of the vessel object than competing methods. The mean SE, SP and ACC of the proposed method are: 0.7768 ± 0.0597, 0.9978 ± 0.0013 and 0.9971 ± 0.0015 respectively. Experiments on the public dataset show that our method can segment complex vessels in different medical images with noise and low- contrast., (© 2021 Institute of Physics and Engineering in Medicine.)

Periacetabular osteotomy (PAO) is a joint preservation procedure for developmental dysplasia of the hip. Such a procedure requires osteotomy of the medial wall of the acetabulum, which may cause the narrow of the bony birth canal and increase the risk of complications during the childbirth process in the future. Using quantitative analysis of the bony birth canal to determine the risk of complications for the childbirth process remains a challenging task. The purpose of this paper is to explore a new 3D CT measurement method to quantify the narrowest parameters of the bony birth canal of the female patients with hip dysplasia before and after unilateral PAO surgery. By analyzing the impact of PAO surgery on the bony birth canal, the patient's risk of complications during the childbirth process in the future can be estimated, and it can be utilized for doctors to determine the impact of unilateral PAO for choosing appropriate delivery method. In this paper, a mean shape of the preoperative pelvises is obtained by using the statistical shape model algorithm, and the mean shape includes pelvic shape features of all the preoperative pelvises, and it can be utilized as the standard pelvic template. A bidirectional iterative algorithm is used to generate a standard bony birth canal path template. Then, the pelvic registration and principal plane deformation constraint are utilized to calculate the optimal bony birth canal path. The proposed method is verified in 31 cases of CT data with the approval of the institutional review board. The test data contain preoperative and postoperative CT images. Compared with the benchmark method, the measurement accuracy of the narrowest position and diameter of the bony birth canal is improved by 65% and 78%, respectively. In addition, the processing speed is increased by 32%. Experimental results demonstrate that the proposed method has high accuracy and validity for quantifying the bony birth canal. The proposed method can measure the anatomical parameters of the bony birth canal accurately. In addition, the doctor can make optimal planning for childbirth with the help of the quantitative analysis of the bony birth canal.

Motion compensation can eliminate inconsistencies of respiratory movement during image acquisitions for precise vascular reconstruction in the clinical diagnosis of vascular disease from x-ray angiographic image sequences. In x-ray-based vascular interventional therapy, motion modeling can simulate the process of organ deformation driven by motion signals to display a dynamic organ on angiograms without contrast agent injection. Automatic respiratory signal estimation from x-ray angiographic image sequences is essential for motion compensation and modeling. The effects of respiratory motion, cardiac impulses, and tremors on structures in the chest and abdomen bring difficulty in extracting accurate respiratory signals individually. In this study, an end-to-end deep learning framework based on a motion-flow-guided recurrent network is proposed to address the aforementioned problem. The proposed method utilizes a convolutional neural network to learn the spatial features of every single frame, and a recurrent neural network to learn the temporal features of the entire sequence. The combination of the two networks can effectively analyze the image sequence to realize respiratory signal estimation. In addition, the motion-flow between consecutive frames is introduced to provide a dynamic constraint of spatial features, which enables the recurrent network to learn better temporal features from dynamic spatial features than from static spatial features. We demonstrate the advantages of our approach on designed datasets which contain coronary and hepatic angiographic sequences with diaphragm structures, and coronary angiographic sequences without diaphragm structures. Our method improves over state-of-the-art manifold-learning-based methods by 85.7%, 81.5% and 75.3% in respiratory signal accuracy metric on these datasets. The results demonstrate that the proposed method can effectively estimate respiratory signals from multiple motion patterns.

Identification of surgical instruments is crucial in understanding surgical scenarios and providing an assistive process in endoscopic image-guided surgery. This study proposes a novel multilevel feature-aggregated deep convolutional neural network (MLFA-Net) for identifying surgical instruments in endoscopic images. First, a global feature augmentation layer is created on the top layer of the backbone to improve the localization ability of object identification by boosting the high-level semantic information to the feature flow network. Second, a modified interaction path of cross-channel features is proposed to increase the nonlinear combination of features in the same level and improve the efficiency of information propagation. Third, a multiview fusion branch of features is built to aggregate the location-sensitive information of the same level in different views, increase the information diversity of features, and enhance the localization ability of objects. By utilizing the latent information, the proposed network of multilevel feature aggregation can accomplish multitask instrument identification with a single network. Three tasks are handled by the proposed network, including object detection, which classifies the type of instrument and locates its border; mask segmentation, which detects the instrument shape; and pose estimation, which detects the keypoint of instrument parts. The experiments are performed on laparoscopic images from MICCAI 2017 Endoscopic Vision Challenge, and the mean average precision (AP) and average recall (AR) are utilized to quantify the segmentation and pose estimation results. For the bounding box regression, the AP and AR are 79.1% and 63.2%, respectively, while the AP and AR of mask segmentation are 78.1% and 62.1%, and the AP and AR of the pose estimation achieve 67.1% and 55.7%, respectively. The experiments demonstrate that our method efficiently improves the recognition accuracy of the instrument in endoscopic images, and outperforms the other state-of-the-art methods.

The 3D/2D registration of pre-operative computed tomography angiography (CTA) and intra-operative x-ray angiography (XRA) images in vascular intervention is imperative for guiding surgical instruments and reducing the dosage of toxic contrast agents. In this study, 3D/2D vascular registration is formulated as a search tree problem on the basis of the topological continuity of vessels and the fact that matching can be decomposed into continuous states. In each node of the tree, a closed-solution of 3D/2D transformation is used to obtain the registration results based on the dense correspondences of vessel points, and the results of matching and registration are calculated and recorded. Then, a hand-crafted score that quantifies the qualities of matching and registration of vessels is used, and the remaining problem focuses on finding the highest score in the search tree. An improved heuristic tree search strategy is also proposed to find the best registration. The proposed method is evaluated and compared with four state-of-the-art methods. Experiments on simulated data demonstrate that our method is insensitive to initial pose and robust to noise and deformation. It outperforms other methods in terms of registering real model data and clinical coronary data. In the 3D/2D registration of uninitialized and initialized coronary arteries, the average registration errors are 1.85 and 1.79 mm, respectively. Given that the proposed method is independent of the initial pose, it can be used to navigate vascular intervention for clinical practice.

The structural information of coronary arteries has important clinical value for quantitative diagnosis and treatment of coronary artery disease. In this study, a deep feature regression (DFR) method based on a convolutional regression network (CRN) and a stable point clustering mechanism for 3D vessel segmentation is proposed. First, the vessel model is constructed by a vessel section generator and a series of deviation parameter estimators. The generator provides 2D images for the training and prediction processes, while the estimators calculate pose parameters of an input vessel section. Second, estimators are trained by a series of CRNs, in which deep vessel features are automatically learned from 600 000 sample images. Third, we propose a stable point clustering mechanism that evaluates the reliability of the CRN estimation through iterative regression of vessel parameters. This mechanism eliminates the outliers, thereby increasing tracking robustness. Finally, we present a vessel segmentation algorithm using trained deviation parameter estimators. And, the termination criteria are designed based on both the number of stable points and an intensity constraint. The proposed method is evaluated on a public coronary artery data set. The average overlapping ratio and error are 97.5% and 0.27 mm, respectively. A quantitative test on a public cerebral artery data set demonstrates that the proposed DFR method tracks the vessel centerline with high accuracy, for which the average error is less than 0.33 mm.

In the diagnosis and interventional treatment of coronary artery disease, the 3D[Formula: see text]time reconstruction of the coronary artery on the basis of x-ray angiographic image sequences can provide dynamic structural information. The synchronization of cardiac phases in the sequences is essential for minimizing the influence of cardiorespiratory motion and realizing precise 3D[Formula: see text]time reconstruction. Key points are initially extracted from the first image of a sequence. Matching grid points between consecutive images in the sequence are extracted by a multi-layer matching strategy. Then deep motion tracking (DMT) of key points is achieved by local deformation based on the neighboring grid points of key points. The local deformation is optimized by the Random sample consensus (RANSAC) algorithm. Then, a simple harmonic motion (SHM) model is utilized to distinguish cardiac motion from other motion sources (e.g. respiratory, patient movement, etc). Next, the signal which is composed of cardiac motions is filtered by a band-pass filter to reconstruct the cardiac phases. Finally, the synchronization of cardiac phases from different imaging angles is realized by a piece-wise linear transformation. The proposed method was evaluated using clinical x-ray angiographic image sequences from 13 patients. [Formula: see text] matching points can be accurately computed by the DMT method. The mean peak temporal distance (MPTD) between the reconstructed cardiac phases and the electrocardiograph signal is [Formula: see text] s. The correlation between the cardiac phases of the same patient is over [Formula: see text]. Compared with three other state-of-the-art methods, the proposed method accurately reconstructs and synchronizes the cardiac phases from different sequences of the same patient. The proposed DMT method is robust and highly effective in synchronizing cardiac phases of angiographic image sequences captured from different imaging angles.

Statistical shape models are capable of extracting shape prior information, and are usually utilized to assist the task of segmentation of medical images. However, such models require large training datasets in the case of multi-object structures, and it also is difficult to achieve satisfactory results for complex shapes. This study proposed a novel statistical model for cervical vertebra segmentation, called sparse intervertebral fence composition (SiFC), which can reconstruct the boundary between adjacent vertebrae by modeling intervertebral fences. The complex shape of the cervical spine is replaced by a simple intervertebral fence, which considerably reduces the difficulty of cervical segmentation. The final segmentation results are obtained by using a 3D active contour deformation model without shape constraint, which substantially enhances the recognition capability of the proposed method for objects with complex shapes. The proposed segmentation framework is tested on a dataset with CT images from 20 patients. A quantitative comparison against corresponding reference vertebral segmentation yields an overall mean absolute surface distance of 0.70 mm and a dice similarity index of 95.47% for cervical vertebral segmentation. The experimental results show that the SiFC method achieves competitive cervical vertebral segmentation performances, and completely eliminates inter-process overlap.