Search

Objective. Smoke, uneven lighting, and color deviation are common issues in endoscopic surgery, which have increased the risk of surgery and even lead to failure. Approach. In this study, we present a new physics model driven semi-supervised learning framework for high-quality pixel-wise endoscopic image enhancement, which is generalizable for smoke removal, light adjustment, and color correction. To improve the authenticity of the generated images, and thereby improve the network performance, we integrated specific physical imaging defect models with the CycleGAN framework. No ground-truth data in pairs are required. In addition, we propose a transfer learning framework to address the data scarcity in several endoscope enhancement tasks and improve the network performance. Main results. Qualitative and quantitative studies reveal that the proposed network outperforms the state-of-the-art image enhancement methods. In particular, the proposed method performs much better than the original CycleGAN, for example, the structural similarity improved from 0.7925 to 0.8648, feature similarity for color images from 0.8917 to 0.9283, and quaternion structural similarity from 0.8097 to 0.8800 in the smoke removal task. Experimental results of the proposed transfer learning method also reveal its superior performance when trained with small datasets of target tasks. Significance. Experimental results on endoscopic images prove the effectiveness of the proposed network in smoke removal, light adjustment, and color correction, showing excellent clinical usefulness., (© 2022 Institute of Physics and Engineering in Medicine.)

Three-dimensional (3D) vessel segmentation can provide full spatial information about an anatomic structure to help physicians gain increased understanding of vascular structures, which plays an utmost role in many medical image-processing and analysis applications. The purpose of this paper aims to develop a 3D vessel-segmentation method that can improve segmentation accuracy in 3D ultrasound (US) images. We propose a 3D tensor-based active contour model method for accurate 3D vessel segmentation. With our method, the contrast-independent multiscale bottom-hat tensor representation and local-global information are captured. This strategy ensures the effective extraction of the boundaries of vessels from inhomogeneous and homogeneous regions without being affected by the noise and low-contrast of the 3D US images. Experimental results in clinical 3D US and public 3D Multiphoton Microscopy datasets are used for quantitative and qualitative comparison with several state-of-the-art vessel segmentation methods. Clinical experiments demonstrate that our method can achieve a smoother and more accurate boundary of the vessel object than competing methods. The mean SE, SP and ACC of the proposed method are: 0.7768 ± 0.0597, 0.9978 ± 0.0013 and 0.9971 ± 0.0015 respectively. Experiments on the public dataset show that our method can segment complex vessels in different medical images with noise and low- contrast., (© 2021 Institute of Physics and Engineering in Medicine.)

Motion compensation can eliminate inconsistencies of respiratory movement during image acquisitions for precise vascular reconstruction in the clinical diagnosis of vascular disease from x-ray angiographic image sequences. In x-ray-based vascular interventional therapy, motion modeling can simulate the process of organ deformation driven by motion signals to display a dynamic organ on angiograms without contrast agent injection. Automatic respiratory signal estimation from x-ray angiographic image sequences is essential for motion compensation and modeling. The effects of respiratory motion, cardiac impulses, and tremors on structures in the chest and abdomen bring difficulty in extracting accurate respiratory signals individually. In this study, an end-to-end deep learning framework based on a motion-flow-guided recurrent network is proposed to address the aforementioned problem. The proposed method utilizes a convolutional neural network to learn the spatial features of every single frame, and a recurrent neural network to learn the temporal features of the entire sequence. The combination of the two networks can effectively analyze the image sequence to realize respiratory signal estimation. In addition, the motion-flow between consecutive frames is introduced to provide a dynamic constraint of spatial features, which enables the recurrent network to learn better temporal features from dynamic spatial features than from static spatial features. We demonstrate the advantages of our approach on designed datasets which contain coronary and hepatic angiographic sequences with diaphragm structures, and coronary angiographic sequences without diaphragm structures. Our method improves over state-of-the-art manifold-learning-based methods by 85.7%, 81.5% and 75.3% in respiratory signal accuracy metric on these datasets. The results demonstrate that the proposed method can effectively estimate respiratory signals from multiple motion patterns.

Identification of surgical instruments is crucial in understanding surgical scenarios and providing an assistive process in endoscopic image-guided surgery. This study proposes a novel multilevel feature-aggregated deep convolutional neural network (MLFA-Net) for identifying surgical instruments in endoscopic images. First, a global feature augmentation layer is created on the top layer of the backbone to improve the localization ability of object identification by boosting the high-level semantic information to the feature flow network. Second, a modified interaction path of cross-channel features is proposed to increase the nonlinear combination of features in the same level and improve the efficiency of information propagation. Third, a multiview fusion branch of features is built to aggregate the location-sensitive information of the same level in different views, increase the information diversity of features, and enhance the localization ability of objects. By utilizing the latent information, the proposed network of multilevel feature aggregation can accomplish multitask instrument identification with a single network. Three tasks are handled by the proposed network, including object detection, which classifies the type of instrument and locates its border; mask segmentation, which detects the instrument shape; and pose estimation, which detects the keypoint of instrument parts. The experiments are performed on laparoscopic images from MICCAI 2017 Endoscopic Vision Challenge, and the mean average precision (AP) and average recall (AR) are utilized to quantify the segmentation and pose estimation results. For the bounding box regression, the AP and AR are 79.1% and 63.2%, respectively, while the AP and AR of mask segmentation are 78.1% and 62.1%, and the AP and AR of the pose estimation achieve 67.1% and 55.7%, respectively. The experiments demonstrate that our method efficiently improves the recognition accuracy of the instrument in endoscopic images, and outperforms the other state-of-the-art methods.

Background: Automatic vascular segmentation in X-ray angiographic image sequence is of crucial interest, for instance, for better quantifying coronary arteries in diagnostic and interventional procedures., Methods: A novel inter/intra-frame constrained vascular segmentation method is proposed to automatically segment vessels in coronary X-ray angiographic image sequence. First, a morphological filter operator is applied to remove structures undergoing the respiratory motion from the original image sequence. Second, an inter-frame constrained robust principal component analysis (RPCA) is utilized to remove the quasi-static structures from the image sequence. Third, an intra-frame constrained RPCA is employed to smooth the final extracted vascular sequence. Fourth, a multi-feature fusion is designed to improve the vascular contrast and the final vascular segmentation is realized by thresholding-based method., Results: Experiments are conducted on 22 clinical X-ray angiographic image sequences. The global and local contrast-to-noise ratio of the proposed method are 6.6344 and 4.2882, respectively. And the precision, sensitivity and F1 value are 0.7378, 0.7960 and 0.7658, respectively. It demonstrates that our method is effective and robust for vascular segmentation from image sequence., Conclusions: The proposed method is effective to remove non-vascular structures, reduce motion artefacts and other non-uniform illumination caused noises. Also, the proposed method is online which can just process one image per time without re-optimizing the model.

Objective: Automatic vascular enhancement in X-ray cineangiography is of crucial interest, for instance, for better visualizing and quantifying coronary arteries in diagnostic and interventional procedures., Methods: A novel patch-based adaptive background subtraction method (PABSM) is proposed automatically enhancing vessels in coronary X-ray cineangiography. First, pixels in the cineangiogram are described by the vesselness and Gabor features. Second, a classifier is utilized to separate the cineangiogram into the rough vascular and non-vascular region. Dilation is applied to the classified binary image to include more vascular region. Third, a patch-based background synthesis is utilized to fill the removed vascular region., Results: A database containing 320 cineangiograms of 175 patients was collected, and then an interventional cardiologist annotated all vascular structures. The performance of PABSM is compared with six state-of-the-art vascular enhancement methods regarding the precision-recall curve and C-value. The area under the precision-recall curve is 0.7133, and the C-value is 0.9659., Conclusion: PABSM can automatically enhance the coronary artery in the cineangiograms. It preserves the integrity of vascular topological structures, particularly in complex vascular regions, and removes noise caused by the non-uniform gray-level distribution in the cineangiogram., Significance: PABSM can avoid the motion artifacts and it eases the subsequent vascular segmentation, which is crucial for the diagnosis and interventional procedures of coronary artery diseases.

Purpose: Respiration causes the deformation and sliding motion of the soft tissues, and affects the accuracy of the assessment of minimally invasive abdominal surgery. Nonrigid registration is used to eliminate the effects of respiration for the assessment. Because the soft tissues with high water content are volume preserving during deformation, incompressibility has to be considered when tracking soft tissues for nonrigid registration. The purpose of the study was to develop an incompressible nonrigid registration for tracking deformable soft tissues with sliding motion., Methods: The nonrigid registration framework proposed in the present study includes two main steps: (a) The solution in the subspace of diffeomorphisms is searched and encoded to stationary velocity field in the log domain. (b) The divergence-free component and harmonic remainder are extracted by Fourier-based Helmholtz-Hodge decomposition (FHHD) and further integrated by an adaptive weight to simultaneously retain the incompressibility of deformation and compensate the sliding motion. The method was evaluated on 11 groups of synthetic datasets and five groups of clinical images. Registration accuracy is evaluated by using four quantitative measures, including mean surface distance (MSD), Hausdorff distance (HD), mean corresponding distance (MCD), and Dice similarity coefficient (DSC). Incompressibility is evaluated by using two quantitative measures, including relative volume change (RVC) and Jacobian determinant (J)., Results: Compared with three state-of-the-art nonrigid registration methods, the proposed method shows an advantage in handling the incompressible deformation of images with large sliding motion. The lowest (MSD, 0.631 mm), (HD, 6.000 mm), and (MCD, 3.555 mm) and the highest (DSC, 0.970) are obtained proving the high registration accuracy with sliding motion compensation of the proposed method. The (RVC, 0.006) and Jacobian determinant (J, 1.008 ± 0.070) are nearly close to 0 and 1, respectively, showing the strong incompressibility of the proposed method. The proposed method improves registration accuracy in nearly all cases, which maintains the incompressibility of tissue transformation while simultaneously compensating the sliding motion on clinical datasets., Conclusions: The proposed method improves the registration accuracy of incompressible tissues when dealing with large sliding motion, and thus has the potential to improve current minimally invasive abdominal surgery., (© 2019 American Association of Physicists in Medicine.)

A graph-based groupwise shape registration algorithm for building statistical shape model (SSM) is proposed, which has been successfully applied to shape prediction of foot scans. Establishing unbiased and effective shape correspondences of large-scale data sets is extremely challenging, for the inappropriate selection of initial mean shape and non-rigid registration of shape with large-scale deformation. To address these issues, first, we use a simplified graph to model the shape distribution in metric space and an edge-guided graph shrinkage to deform the shapes. Then, the groupwise registration is performed by iteratively performing the graph shrinkage until the shape converges. And, the correspondences of training shapes are obtained by propagating the converged shape to the original data along each shrinkage path. Compared with traditional forward and backward models of groupwise registration, the proposed method is data-driven without initial mean shape as input. Moreover, under the constraint of the established graph, the non-rigid registration can perform more accurately by restricting shape register to its neighbors. Based on the shape correspondence, the SSM of foot shapes is constructed and applied to shape prediction by taking the collected anthropometric information as predictor. Experiments demonstrate that the proposed method can obtain robust shape correspondences and SSM capability with respect to model generalization, specificity, and compactness. The application of shape prediction model shows an average prediction error lower than 1% for general foot size. Graphical abstract The graphical abstract of unbiased groupwise registration for foot prediction.

Backgroud and Objective: X-ray angiography, a powerful technique for blood vessel visualization, is widely used for interventional diagnosis of coronary artery disease because of its fast imaging speed and perspective inspection ability. Matching feature points in angiographic images is a considerably challenging task due to repetitive weak-textured regions., Methods: In this paper, we propose an angiographic image matching method based on the hierarchical dense matching framework, where a novel deep feature descriptor is designed to compute multilevel correlation maps. In particular, the deep feature descriptor is computed by a deep learning model specifically designed and trained for angiographic images, thereby making the correlation maps more distinctive for corresponding feature points in different angiographic images. Moreover, point correspondences are further hierarchically extracted from multilevel correlation maps with the highest similarity response(s), which is relatively robust and accurate. To overcome the problem regarding the lack of training samples, the convolutional neural network (designed for deep feature descriptor) is initially trained on samples from natural images and then fine-tuned on manually annotated angiographic images. Finally, a dense matching completion method, based on the distance between deep feature descriptors, is proposed to generate dense matches between images., Results: The proposed method has been evaluated on the number and accuracy of extracted matches and the performance of subtraction images. Experiments on a variety of angiographic images show promising matching accuracy, compared with state-of-the-art methods., Conclusions: The proposed angiographic image matching method is shown to be accurate and effective for feature matching in angiographic images, and further achieves good performance in image subtraction., (Copyright © 2019 Elsevier B.V. All rights reserved.)

The structural information of coronary arteries has important clinical value for quantitative diagnosis and treatment of coronary artery disease. In this study, a deep feature regression (DFR) method based on a convolutional regression network (CRN) and a stable point clustering mechanism for 3D vessel segmentation is proposed. First, the vessel model is constructed by a vessel section generator and a series of deviation parameter estimators. The generator provides 2D images for the training and prediction processes, while the estimators calculate pose parameters of an input vessel section. Second, estimators are trained by a series of CRNs, in which deep vessel features are automatically learned from 600 000 sample images. Third, we propose a stable point clustering mechanism that evaluates the reliability of the CRN estimation through iterative regression of vessel parameters. This mechanism eliminates the outliers, thereby increasing tracking robustness. Finally, we present a vessel segmentation algorithm using trained deviation parameter estimators. And, the termination criteria are designed based on both the number of stable points and an intensity constraint. The proposed method is evaluated on a public coronary artery data set. The average overlapping ratio and error are 97.5% and 0.27 mm, respectively. A quantitative test on a public cerebral artery data set demonstrates that the proposed DFR method tracks the vessel centerline with high accuracy, for which the average error is less than 0.33 mm.

Background and Objective: The development of biometric identification technology and intelligent medication has enabled researchers to analyze subcutaneous veins from near-infrared images. However, the stereo reconstruction of subcutaneous veins has not been well studied, and the results are difficult to utilize in clinical practice., Methods: We present a hybrid constraint optimization (HCO) matching algorithm for vein reconstruction to solve the matching failure problems caused by the incomplete segmentation of vein structures captured from different views. This algorithm initially introduces the existence of the epipolar and homography constraints in the subcutaneous vein matching. Then, the HCO matching algorithm of the vascular centerline is established by homography point-to-point matching, homography matrix optimization, and vascular section matching. Finally, the 3D subcutaneous vein is reconstructed on the basis of the principle of triangulation and system calibration parameters., Results: To validate the performance of the proposed matching method, we designed a series of experiments to evaluate the effectiveness of the hybrid constraint optimization method. The experiments were performed on simulated and real datasets. 42 real vascular images were analyzed on the basis of different matching strategies. Experimental result shows that the matching accuracy increased significantly with the proposed optimization matching method. To calculate the reconstruction accuracy, we reconstructed seven simulated cardboards and measured 10 vascular distances in each simulated cardboard. The average vascular distance error of each simulated image was within 1.0 mm, and the distance errors of 75% feature points were less than 1.5 mm. Also, we printed a 3D simulated vein model to improve the illustration of this system. The reconstruction error extends from -3.58 mm to 1.94 mm with a standard deviation of 0.68 mm and a mean of 0.07 mm., Conclusions: The algorithm is validated in terms of homography optimization, matching efficiency, and simulated vascular reconstruction error. The experimental results demonstrate that the veins captured from the left and right views can be accurately matched through the proposed algorithm., (Copyright © 2018 Elsevier B.V. All rights reserved.)

Background and Objective: A framework of sparse deformation prediction using Markove Decision Processes is proposed for achieving a rapid and accurate registration by providing a suitable initial deformation., Methods: In the proposed framework, the tree is built based on the training set for each patch from the template image. The template patch is considered as the root. The node is the patch group in which multiple similar patches are extracted around a key point on the training image. Given the linkages between patch groups in the tree, MDP is introduced to select the optimal path with highest registration accuracy from each training patch to the template patch. The deformation between them is estimated along the selected path by patch-wise registration which can be realized by a non-learning-based method. Given the patches on a testing image, their best matching patches are fast chosen from the training patches and the corresponding deformations constitute a sparse deformation. A dense deformation for the entire test image is subsequently interpolated and used as an initial deformation for further registration., Results: With the non-learning-based registration as the baseline method, the proposed framework is evaluated using three datasets of inter-subject brain MR images with three learning-based methods. Experimental results of the non-learning-based method using the proposed framework reveal that the computation time is reduced by fivefold after using the proposed framework. And, with the same baseline method, the proposed framework demonstrates the higher accuracy than three learning-based methods which predicts the initial deformation at image scale. The mean Dice of three datasets for the tissues of the brain are 73.52%, 70.73% and 64.82%, respectively., Conclusions: The proposed framework rapidly registers the inter-subject brains and achieves the high mean Dice for the tissues of the brain., (Copyright © 2018 Elsevier B.V. All rights reserved.)

Liver metabolic information is considered as a crucial diagnostic marker for the diagnosis of fever of unknown origin, and liver recognition is the basis of automatic diagnosis of metabolic information extraction. However, the poor quality of PET and CT images is a challenge for information extraction and target recognition in PET-CT images. The existing detection method cannot meet the requirement of liver recognition in PET-CT images, which is the key problem in the big data analysis of PET-CT images. A novel texture feature descriptor called multi-layer cube sampling (MLCS) is developed for liver boundary detection in low-dose CT and PET images. The cube sampling feature is proposed for extracting more texture information, which uses a bi-centric voxel strategy. Neighbour voxels are divided into three regions by the centre voxel and the reference voxel in the histogram, and the voxel distribution information is statistically classified as texture feature. Multi-layer texture features are also used to improve the ability and adaptability of target recognition in volume data. The proposed feature is tested on the PET and CT images for liver boundary detection. For the liver in the volume data, mean detection rate (DR) and mean error rate (ER) reached 95.15 and 7.81% in low-quality PET images, and 83.10 and 21.08% in low-contrast CT images. The experimental results demonstrated that the proposed method is effective and robust for liver boundary detection.

Purpose: In liver microwave ablation (MWA) surgery, the ablation area covers the tumor to generate tissue necrosis and treat the cancer. As the liver deforms during the operation, deviation between the target area determined during preoperative planning and the resultant ablation area is inevitable. Therefore, an accurate assessment of tumor coverage is crucial for treatment. Through registration between the pre- and postoperative livers, the ablation area is warped on the preoperative liver for the computation of tumor coverage. However, large deformations between the pre- and postoperative livers are caused by multiple factors, and these diverse deformations make registration a challenging task. The purpose of this paper was to develop an automatic method that can accurately register post- to preoperative livers., Methods: In the proposed method, nonrigid deformations caused by respiratory movement and edema are separately considered and estimated by the local incompressible model in the registration of livers. The pre- and postoperative livers are first aligned by a rigid registration based on a convex hull. In the nonrigid registrations, local incompressible constraints are then set on the liver and the ablation area to estimate the deformations caused by respiratory movement and edema, respectively. The concatenation of the rigid and nonrigid deformations is used to warp the ablation area on the preoperative liver., Results: The proposed method was evaluated using clinical CT datasets from 20 patients. The Dice similarity coefficient (DSC) between the preoperative and warped postoperative livers is 94.35%, the mean surface distance (MSD) between the livers is 1.65 mm, the mean Hausdorff distance (HDD) between the livers is 3.36 mm, and the mean corresponding distance (MCD) between the corresponding landmarks is 1.70 mm. Compared with five other state-of-the-art methods, the proposed method achieves automatic ablation assessment with highly accurate registration., Conclusions: The proposed method achieves a high accuracy for registering the livers. The sizes and positions of the ablation area and tumor are accurately compared for the assessment of ablation surgery., (© 2017 American Association of Physicists in Medicine.)

Background: Automated image segmentation has benefits for reducing clinicians' workload, quicker diagnosis, and a standardization of the diagnosis., Methods: This study proposes an automatic liver segmentation approach based on appearance and context information. The relationship between neighboring pixels in blocks is utilized to estimate appearance information, which is used for training the first classifier and obtaining the probability distribution map. The map is used for extracting context information, along with appearance features, to train the next classifier. The prior probability distribution map is achieved after iterations and refined through an improved random walk for liver segmentation without user interaction., Results: The proposed approach is evaluated using CT images with eight contemporary approaches, and it achieves the highest VOE, RVD, ASD, RMSD and MSD. It also achieves a high average score of 76 using the MICCAI-2007 Grand Challenge scoring system., Conclusions: Experimental results show that the proposed method is superior to eight other state of the art methods.

Liver segmentation is a significant processing technique for computer-assisted diagnosis. This method has attracted considerable attention and achieved effective result. However, liver segmentation using computed tomography (CT) images remains a challenging task because of the low contrast between the liver and adjacent organs. This paper proposes a feature-learning-based random walk method for liver segmentation using CT images. Four texture features were extracted and then classified to determine the classification probability corresponding to the test images. Seed points on the original test image were automatically selected and further used in the random walk (RW) algorithm to achieve comparable results to previous segmentation methods., Competing Interests: The authors have declared that no competing interests exist.

For tooth segmentation problem on the three-dimensional computed tomography(CT)volume data,this paper proposes a regional adaptive deformable model for tooth structure measurement of CT images.The proposed method combines the automatic thresholding segmentation,CV active contour model,and graph-cut.Firstly,we achieved the segmentation and location of dental crowns by automatic thresholding segmentation.And then by using the above segmentation result as the initial contour,we utilized active contour method to slice gradually the segment of remaining tooth.By incorporating active contour and graph-cut then,we realized the accurate segmentation for tooth root,which is the most difficult to be segmented.The experimental results showed that the proposed tooth structure measurement accurately and automatically segmented dental crowns from CT data,and then rapidly and accurately segmented the tooth neck and tooth root.The structure of tooth could be effectively segmented from CT data by using the proposed method.Experimental results indicated that the proposed method was rather robust and accurate,and could effectively assist the doctor for diagnosis in clinical treatment.

Background: Ultrasound images are usually affected by speckle noise, which is a type of random multiplicative noise. Thus, reducing speckle and improving image visual quality are vital to obtaining better diagnosis., Method: In this paper, a novel noise reduction method for medical ultrasound images, called multiresolution generalized N dimension PCA (MR-GND-PCA), is presented. In this method, the Gaussian pyramid and multiscale image stacks on each level are built first. GND-PCA as a multilinear subspace learning method is used for denoising. Each level is combined to achieve the final denoised image based on Laplacian pyramids., Results: The proposed method is tested with synthetically speckled and real ultrasound images, and quality evaluation metrics, including MSE, SNR and PSNR, are used to evaluate its performance., Conclusion: Experimental results show that the proposed method achieved the lowest noise interference and improved image quality by reducing noise and preserving the structure. Our method is also robust for the image with a much higher level of speckle noise. For clinical images, the results show that MR-GND-PCA can reduce speckle and preserve resolvable details.