Search

The clinical success of the percutaneous coronary intervention (PCI) is highly dependent on endovascular manipulation skills and dexterous manipulation strategies of interventionalists. However, the analysis of endovascular manipulations and related discussion for technical skill assessment are limited. In this study, a multilayer and multimodal-fusion architecture is proposed to recognize six typical endovascular manipulations. The synchronously acquired multimodal motion signals from ten subjects are used as the inputs of the architecture independently. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. The recognition metrics under the determined architecture are further used to assess technical skills. The experimental results indicate that the proposed architecture can achieve the overall accuracy of 96.41%, much higher than that of a single-layer recognition architecture (92.85%). In addition, the multimodal fusion brings significant performance improvement in comparison with single-modal schemes. Furthermore, the K -means-based skill assessment can obtain an accuracy of 95% to cluster the attempts made by different skill-level groups. These hopeful results indicate the great possibility of the architecture to facilitate clinical skill assessment and skill learning.

In endovascular and cardiovascular surgery, real-time guidewire morphological and positional analysis is an important prerequisite for robot-assisted intervention, which can aid in reducing the radiation dose, contrast agent, and procedure time. Nevertheless, this task often comes with the challenge of the deformable elongated structure with low contrast in noisy X-ray fluoroscopy. In this article, a real-time multifunctional framework is proposed for fully automatic guidewire morphological and positional analysis, namely, guidewire segmentation, endpoint localization, and angle measurement. In the first stage, the proposed fast attention recurrent network (FAR-Net) achieves real-time and accurate guidewire segmentation. In the second stage, the endpoint localization and angle measurement algorithm robustly obtain subpixel-level endpoint and angle of the guidewire tip. Quantitative and qualitative evaluations on the MSGSeg data set consisting of 180 X-ray sequences from 30 patients demonstrate that the proposed framework significantly outperforms simpler baselines as well as the best previously published result for this task. The proposed approach reached ${F_{1}}$ -Score of 0.938, mean distance error of 0.596 pixels, endpoint localization and angle measurement accuracy of 97.8% and 95.3%, and an inference rate of approximately 13 FPS. The proposed framework not only addresses the issues of extreme class imbalance and misclassified examples but also meets the real-time requirements, achieving state-of-the-art performance. The proposed approach is promising for integration into robotic navigation frameworks to various intravascular applications, enabling robotic-assisted intervention.

Percutaneous coronary intervention (PCI) has increasingly become the main treatment for coronary artery disease. The procedure requires high experienced skills and dexterous manipulations. However, there are few techniques to model PCI skill so far. In this study, a learning framework with local and ensemble learning is proposed to learn skill characteristics of different skill-level subjects from their PCI manipulations. Ten interventional cardiologists (four experts and six novices) were recruited to deliver a medical guidewire to two target arteries on a porcine model for in vivo studies. Simultaneously, translation and twist manipulations of thumb, forefinger, and wrist are acquired with electromagnetic (EM) and fiber-optic bend (FOB) sensors, respectively. These behavior data are then processed with wavelet packet decomposition (WPD) under 1-10 levels for feature extraction. The feature vectors are further fed into three candidate individual classifiers in the local learning layer. Furthermore, the local learning results from different manipulation behaviors are fused in the ensemble learning layer with three rule-based ensemble learning algorithms. In subject-dependent skill characteristics learning, the ensemble learning can achieve 100% accuracy, significantly outperforming the best local result (90%). Furthermore, ensemble learning can also maintain 73% accuracy in subject-independent schemes. These promising results demonstrate the great potential of the proposed method to facilitate skill learning in surgical robotics and skill assessment in clinical practice.

In computer-assisted vascular surgery, real-time multi-instrument segmentation serves as a pre-requisite step. However, a large amount of effort has been dedicated to single-instrument rather than multi-instrument in computer-assisted intervention research to this day. To fill the overlooked gap, this study introduces a Light-Weight Deep Feature Refinement Network (DFR-Net) based on multi-task learning for real-time multi-instrument segmentation. In this network, the proposed feature refinement module (FRM) can capture long-term dependencies while retaining precise positional information, which helps model locate the foreground objects of interest. The designed channel calibration module (CCM) can re-calibrate fusion weights of multi-level features, which helps model balance the importance of semantic information and appearance information. Besides, the connectivity loss function is developed to address fractures in the wire-like structure segmentation results. Extensive experiments on two different types of datasets consistently demonstrate that DFR-Net can achieve state-of-the-art segmentation performance while meeting the real-time requirements.

Objective: In this paper, Keypoint Localization Region-based CNN (KL R-CNN) is proposed, which can simultaneously accomplish the guidewire detection and endpoint localization in a unified model. Methods: KL R-CNN modifies Mask R-CNN by replacing the mask branch with a novel keypoint localization branch. Besides, some settings of Mask R-CNN are also modified to generate the keypoint localization results at a higher detail level. At the same time, based on the existing metrics of Average Precision (AP) and Percentage of Correct Keypoints (PCK), a new metric named AP PCK is proposed to evaluate the overall performance on the multi-guidewire endpoint localization task. Compared with existing metrics, AP PCK is easy to use and its results are more intuitive. Results: Compared with existing methods, KL R-CNN has better performance when the threshold is loose, reaching a mean AP PCK of 90.65% when the threshold is 9 pixels. Conclusion: KL R-CNN achieves the state-of-the-art performance on the multi-guidewire endpoint localization task and has application potentials. Significance: KL R-CNN can achieve the localization of guidewire endpoints in fluoroscopy images, which is a prerequisite for computer-assisted percutaneous coronary intervention. KL R-CNN can also be extended to other multi-instrument localization tasks.

Real-time guidewire segmentation and endpoint localization play a pivotal role in robot-assisted minimally invasive surgery, which is helpful to reduce radiation dose and procedure time. Nevertheless, the tasks often come with the challenge of limited computational resources. For this purpose, a real-time multi-task framework with two stages is developed. In the first stage, a Fast Attention-fused Network (FAD-Net) is proposed to obtain accurate guidewire segmentation masks. In the second stage, a lightweight localization network and a post-processing algorithm are designed to robustly predict the guidewire endpoint position. Quantitative and qualitative evaluations on intraoperative X-ray sequences from 30 patients demonstrate that the developed framework outperforms the previously-published results for the tasks, achieving state-of-the-art performance. Moreover, the inference rate of the developed framework is approximately 10.6 FPS, which meets the real-time requirement of X-ray fluoroscopy. These results indicate the proposed approach has the potential to be integrated into the robotic navigation framework for endovascular interventions, enabling robotic-assisted minimally invasive surgery.

Medical diagnostic robot systems have been paid more and more attention due to its objectivity and accuracy. The diagnosis of mild cognitive impairment (MCI) is considered an effective means to prevent Alzheimer's disease (AD). Doctors diagnose MCI based on various clinical examinations, which are expensive and the diagnosis results rely on the knowledge of doctors. Therefore, it is necessary to develop a robot diagnostic system to eliminate the influence of human factors and obtain a higher accuracy rate. In this paper, we propose a novel Group Feature Domain Adversarial Neural Network (GF-DANN) for amnestic MCI (aMCI) diagnosis, which involves two important modules. A Group Feature Extraction (GFE) module is proposed to reduce individual differences by learning group-level features through adversarial learning. A Dual Branch Domain Adaptation (DBDA) module is carefully designed to reduce the distribution difference between the source and target domain in a domain adaption way. On three types of data set, GF-DANN achieves the best accuracy compared with classic machine learning and deep learning methods. On the DMS data set, GF-DANN has obtained an accuracy rate of 89.47%, and the sensitivity and specificity are 90% and 89%. In addition, by comparing three EEG data collection paradigms, our results demonstrate that the DMS paradigm has the potential to build an aMCI diagnose robot system., This paper has been accepted by 2021 International Conference on Robotics and Automation (ICRA 2021)

Vessel width estimation has a wide range of applications in disease diagnosis and treatment. In this paper, vessel width estimation is cast as a regression problem, and a novel Convolutional Neural Network (CNN) based method is proposed for vessel width estimation. In our CNN-based method, the idea of divide-and-conquer is introduced to solve the challenge of imbalanced training samples. Besides, in order to solve the shortage of training samples required by CNN, a vessel width label generation method is proposed to generate width labels from vessel segmentation labels. In the experiments, we apply our vessel width label generation method and CNN-based width estimation method to two tasks which are retinal vessel width estimation and coronary artery width estimation. Experimental results show that our width label generation method can generate sufficiently realistic width labels using accurate segmentation labels. Also, our CNN-based method can solve the challenge of imbalanced training samples, achieving state-of-the-art performance with less inference time.

The increasingly-used robotic systems can provide precise delivery and reduce X-ray radiation to medical staff in percutaneous coronary interventions (PCI), but natural manipulations of interventionalists are forgone in most robot-assisted procedures. Therefore, it is necessary to explore natural manipulations to design more advanced human-robot interfaces (HRI). In this study, a multilayer-multimodal fusion architecture is proposed to recognize six typical subpatterns of guidewire manipulations in conventional PCI. The synchronously acquired multimodal behaviors from ten subjects are used as the inputs of the fusion architecture. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. Experimental results indicate that the multimodal fusion brings significant accuracy improvement in comparison with single-modal schemes. Furthermore, the proposed architecture can achieve the overall accuracy of 96.90%, much higher than that of a singlelayer recognition architecture (92.56%). These results have indicated the potential of the proposed method for facilitating the development of HRI for robot-assisted PCI.

For endovascular aneurysm repair (EVAR), real-time and accurate segmentation and tracking of interventional instruments can aid in reducing radiation exposure, contrast agents and procedure time. Nevertheless, this task often comes with the challenges of the slender deformable structures with low contrast in noisy X-ray fluoroscopy. In this paper, a novel efficient network architecture, termed FRR-Net, is proposed for real-time catheter segmentation and tracking. The novelties of FRR-Net lie in the manner in which recurrent convolutional layers ensures better feature representation and the pre-trained lightweight components can improve model processing speed while ensuring performance. Quantitative and qualitative evaluation on images from 175 X-ray sequences of 30 patients demonstrate that the proposed approach significantly outperforms simpler baselines as well as the best previously-published result for this task, achieving the state-of-the-art performance.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

In endovascular interventional therapy, the fusion of preoperative data with intraoperative X-ray fluoroscopy has demonstrated the potential to reduce radiation dose, contrast agent and processing time. Real-time intraoperative stent segmentation is an important pre-requisite for accurate fusion. Nevertheless, this task often comes with the challenge of the thin stent wires with low contrast in noisy X-ray fluoroscopy. In this paper, a novel and efficient network, termed Lightweight Double Attention-fused Network (LDA-Net), is proposed for end-to-end stent segmentation in intraoperative X-ray fluoroscopy. The proposed LDA-Net consists of three major components, namely feature attention module, relevance attention module and pre-trained MobileNetV2 encoder. Besides, a hybrid loss function of both reinforced focal loss and dice loss is designed to better address the issues of class imbalance and misclassified examples. Quantitative and qualitative evaluations on 175 intraoperative X-ray sequences demonstrate that the proposed LDA-Net significantly outperforms simpler baselines as well as the best previously-published result for this task, achieving the state-of-the-art performance.