Search

Purpose: There is no objective way to measure the amount of manipulation and retraction of neural tissue by the surgeon. Our goal is to develop metrics quantifying dynamic retraction and manipulation by instruments during neurosurgery. Methods: We trained a convolutional neural network (CNN) to analyze microscopic footage of neurosurgical procedures and thereby generate metrics evaluating the surgeon's dynamic retraction of brain tissue and, using an object tracking process, evaluate the surgeon's manipulation of the instruments themselves. U-Net image segmentation is used to output bounding polygons around cerebral parenchyma of interest, as well as the vascular structures and cranial nerves. A channel and spatial reliability tracker framework is used in conjunction with our CNN to track desired surgical instruments. Results: Our network achieved a state-of-the-art intersection over union ( 72.64 % ) for biological tissue segmentation. Multivariate statistical analysis was used to evaluate dynamic retraction, tissue handling, and instrument manipulation. Conclusion: Our model enables to evaluate dynamic retraction of soft tissue and manipulation of instruments during a surgical procedure, while accounting for movement of the operative microscope. This model can potentially provide the surgeon with objective feedback about the movement of instruments and its effect on brain tissue. [ABSTRACT FROM AUTHOR]

The robot-held laparoscope efficiently maintains the stability of the surgical screen in comparison with the laparoscope held by the assistant. The collaboration between the robot-held laparoscope and the surgeon is achieved without the artificial manipulation by tracking surgical instruments automatically. In this article, a visual tracking space vector method is proposed to autonomously control the field of view by tracking multiple surgical instruments. In the proposed method, a visual tracking space vector is introduced to track the motion of multiple instruments, and a constraint vector is defined to ensure that the laparoscope is always within the safe working space. In addition, a modified YOLOv3 algorithm is proposed to robustly detect and classify the surgical instruments. The proposed method automatically adapts to the changes in the number of surgical instruments, as compared to other visual servoing methods for tracking surgical instruments. The proposed method does not require the depth information of the tracked surgical instruments. Experimental and simulation results verify that the proposed method accurately and efficiently controls the robot-held laparoscope to track the motion of multiple surgical instruments and ensure safe tracking movement. [ABSTRACT FROM AUTHOR]

Nowadays, ultrasound (US) is increasingly being chosen as imaging modality for both diagnostic and interventional applications, owing to its positive characteristics in terms of safety, low footprint, and low cost. The combination of this imaging modality with wearable augmented reality (AR) systems, such as the head-mounted displays (HMD), comes forward as a breakthrough technological solution, as it allows for hands-free interaction with the augmented scene, which is an essential requirement for the execution of high-precision manual tasks, such as in surgery. What we propose in this study is the integration of an AR navigation system (HMD plus dedicated platform) with a 3D US imaging system to guide a dissection task that requires maintaining safety margins with respect to unexposed anatomical or pathological structures. For this purpose, a standard scalpel was sensorized to provide real-time feedback on the position of the instrument during the execution of the task. The accuracy of the system was quantitatively assessed with two different experimental studies: a targeting experiment, which revealed a median error of 2.53 mm in estimating the scalpel to target distance, and a preliminary user study simulating a dissection task that requires reaching a predefined distance to an occult lesion. The second experiment results showed that the system can be used to guide a dissection task with a mean accuracy of 0.65 mm, with a mean angular error between the ideal and actual cutting plane of 2.07°. The results encourage further studies to fully exploit the potential of wearable AR and intraoperative US imaging to accurately guide deep surgical tasks, such as to guide the excision of non-palpable breast tumors ensuring optimal margin clearance.

In the field of computer- and robot-assisted minimally invasive surgery, enormous progress has been made in recent years based on the recognition of surgical instruments in endoscopic images and videos. In particular, the determination of the position and type of instruments is of great interest. Current work involves both spatial and temporal information, with the idea that predicting the movement of surgical tools over time may improve the quality of the final segmentations. The provision of publicly available datasets has recently encouraged the development of new methods, mainly based on deep learning. In this review, we identify and characterize datasets used for method development and evaluation and quantify their frequency of use in the literature. We further present an overview of the current state of research regarding the segmentation and tracking of minimally invasive surgical instruments in endoscopic images and videos. The paper focuses on methods that work purely visually, without markers of any kind attached to the instruments, considering both single-frame semantic and instance segmentation approaches, as well as those that incorporate temporal information. The publications analyzed were identified through the platforms Google Scholar, Web of Science, and PubMed. The search terms used were "instrument segmentation", "instrument tracking", "surgical tool segmentation", and "surgical tool tracking", resulting in a total of 741 articles published between 01/2015 and 07/2023, of which 123 were included using systematic selection criteria. A discussion of the reviewed literature is provided, highlighting existing shortcomings and emphasizing the available potential for future developments., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024. Published by Elsevier Ltd.)

In recent years, the problem of visual occlusion in optical surgical tracking system has attracted widespread attention. To solve this problem, this paper proposes a tracking system for surgical instruments based on data fusion of multicamera modules. Multicamera modules are used to track surgical instruments from different angles to solve the problem of tracking occlusion in monocular tracking. Firstly, occlusion of monocular tracking based on different fiducial marker is analyzed, and a fast calibration method of tracking system for surgical instrument based on multicamera module is proposed. The calibration of the tracking system is mainly divided into the calibration of surgical instruments and the fast calibration between multicamera modules. On this basis, the occlusion detection mechanism is proposed to screen out the data with large errors, and the fuzzy control theory is used to calculate the support degree between different modules. Finally, the serviceable data is fused based on the least square method and the position of surgical instruments is estimated. The average accuracy of static positioning and dynamic trajectory tracking is mm and mm respectively. The real-time performance of the proposed system is 35 frames per second (fps). Compared with the existing optical tracking system, the method proposed in this paper has stronger anti-occlusion and simpler operation. [ABSTRACT FROM AUTHOR]

• A simple but effective colour-based segmentation model is incorporated to assign weight to the patch-based descriptor. • A two-level sampling strategy enables the tracker to handle both incremental and abrupt scale variations. • Achieve superior results on various datasets among top trackers with near real-time performance. • Substantial evaluation is performed on both ex-vivo and in-vivo surgical datasets. Vision-based tracking in an important component for building computer assisted interventions in minimally invasive surgery as it facilitates estimation of motion for instruments and anatomical targets. Tracking-by-detection algorithms are widely used for visual tracking, where the problem is treated as a classification task and a tracking target appearance model is updated over time using online learning. In challenging conditions, like surgical scenes, where tracking targets deform and vary in scale, the update step is prone to include background information in model appearance or to lack the ability to estimate change of scale, which degrades the performance of classifier. In this paper, we propose a Patch-based Adaptive Weighting with Segmentation and Scale (PAWSS) tracking framework that tackles both scale and background problems. A simple but effective colour-based segmentation model is used to suppress background information and multi-scale samples are extracted to enrich the training pool, which allows the tracker to handle both incremental and abrupt scale variations between frames. Experimentally, we evaluate our approach on Online Tracking Benchmark (OTB) dataset and Visual Object Tracking (VOT) challenge datasets, showing that our approach outperforms recent state-of-the-art trackers, and it especially improves successful rate score on OTB dataset, while on VOT datasets, PAWSS ranks among the top trackers while operating at real-time frame rates. Focusing on the application of PAWSS to surgical scenes, we evaluate on MICCAI 2015 challenge instrument tracking challenge and in vivo datasets, showing that our approach performs the best among all submitted methods and also has promising performance on in vivo surgical instrument tracking. [ABSTRACT FROM AUTHOR]

This paper presents a sensor fusion-based surgical instrument tracking system which uses multiple monocular modules. The system is an optical tracking system, which has been widely utilized in the image-guide surgery because of its high accuracy and precision. However, the line-of-sight occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple monocular modules. The rationale is to enable the system to track the surgical instruments inside the surgical site from different views. Three sensor fusion algorithms are proposed to integrate all sensor data from the multimodule system. In order to show the feasibility of the tracking system, simulations and comparison experiments have been carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions. Moreover, an image-guided surgical trial by using a cadaver head has been carried out to validate the feasibility of the proposed system and the tracking algorithms. The results from both the simulation and the cadaver trial have shown the effectiveness of the proposed robust fusion algorithm. Note to Practitioners—Optical tracking system (OTS) for image-guided surgery has suffered from line-of-sight (LoS) occlusion and field-of-view (FoV) limitation problems for a long time; nevertheless, no satisfactory solution has been given yet. In the real-world practice, surgeons and assistants have to manually adjust the position and orientation of the optical tracker during operation. This interrupts the surgery process and degrades the efficiency of the surgical procedure. Instead of using one tracker for the image-guided surgery, our approach is to utilize distributed modular trackers to avoid LoS occlusion and maintain FoV of the OTS. Based on our previous work, this paper is to provide an optimal fusion algorithm that takes the advantage of the multiple modules and validate the system in both the simulation and the real-world surgical environment. [ABSTRACT FROM AUTHOR]

As the number of robotic surgery procedures has increased, so has the importance of evaluating surgical skills in these techniques. It is difficult, however, to automatically and quantitatively evaluate surgical skills during robotic surgery, as these skills are primarily associated with the movement of surgical instruments. This study proposes a deep learning-based surgical instrument tracking algorithm to evaluate surgeons’ skills in performing procedures by robotic surgery. This method overcame two main drawbacks: occlusion and maintenance of the identity of the surgical instruments. In addition, surgical skill prediction models were developed using motion metrics calculated from the motion of the instruments. The tracking method was applied to 54 video segments and evaluated by root mean squared error (RMSE), area under the curve (AUC), and Pearson correlation analysis. The RMSE was 3.52 mm, the AUC of 1 mm, 2 mm, and 5 mm were 0.7, 0.78, and 0.86, respectively, and Pearson’s correlation coefficients were 0.9 on the x-axis and 0.87 on the y-axis. The surgical skill prediction models showed an accuracy of 83% with Objective Structured Assessment of Technical Skill (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS). The proposed method was able to track instruments during robotic surgery, suggesting that the current method of surgical skill assessment by surgeons can be replaced by the proposed automatic and quantitative evaluation method.

Localizing instrument parts in video-assisted surgeries is an attractive and open computer vision problem. A working algorithm would immediately find applications in computer-aided interventions in the operating theater. Knowing the location of tool parts could help virtually augment visual faculty of surgeons, assess skills of novice surgeons, and increase autonomy of surgical robots. A surgical tool varies in appearance due to articulation, viewpoint changes, and noise. We introduce a new method for detection and pose estimation of multiple non-rigid and robotic tools in surgical videos. The method uses a rigidly structured, bipartite model of end-effector and shaft parts that consistently encode diverse, pose-specific appearance mixtures of the tool. This rigid part mixtures model then jointly explains the evolving tool structure by switching between mixture components. Rigidly capturing end-effector appearance allows explicit transfer of keypoint meta-data of the detected components for full 2D pose estimation. The detector can as well delineate precise skeleton of the end-effector by transferring additional keypoints. To this end, we propose effective procedure for learning such rigid mixtures from videos and for pooling the modeled shaft part that undergoes frequent truncation at the border of the imaged scene. Notably, extensive diagnostic experiments inform that feature regularization is a key to fine-tune the model in the presence of inherent appearance bias in videos. Experiments further illustrate that estimation of end-effector pose improves upon including the shaft part in the model. We then evaluate our approach on publicly available datasets of in-vivo sequences of non-rigid tools and demonstrate state-of-the-art results. [ABSTRACT FROM AUTHOR]

In laparoscopic surgery, the surgeon must operate with a limited field of view and reduced depth perception. This makes spatial understanding of critical structures difficult, such as an endophytic tumour in a partial nephrectomy. Such tumours yield a high complication rate of 47%, and excising them increases the risk of cutting into the kidney's collecting system. To overcome these challenges, an augmented reality guidance system is proposed. Using intra-operative ultrasound, a single navigation aid, and surgical instrument tracking, four augmentations of guidance information are provided during tumour excision. Qualitative and quantitative system benefits are measured in simulated robot-assisted partial nephrectomies. Robot-to-camera calibration achieved a total registration error of 1.0 ± 0.4 mm while the total system error is 2.5 ± 0.5 mm. The system significantly reduced healthy tissue excised from an average (±standard deviation) of 30.6 ± 5.5 to 17.5 ± 2.4 cm^3 (p < 0.05) and reduced the depth from the tumor underside to cut from an average (±standard deviation) of 10.2 ± 4.1 to 3.3 ± 2.3 mm (p < 0.05). Further evaluation is required in vivo, but the system has promising potential to reduce the amount of healthy parenchymal tissue excised.

Robot-assisted minimally invasive surgery is effective for operations in limited space. Enhancing safety based on automatic tracking of surgical instrument position to prevent inadvertent harmful events such as tissue perforation or instrument collisions could be a meaningful augmentation to current robotic surgical systems. A vision-based instrument tracking scheme as a core algorithm to implement such functions was developed in this study. An automatic tracking scheme is proposed as a chain of computer vision techniques, including classification of metallic properties using k-means clustering and instrument movement tracking using similarity measures, Euclidean distance calculations, and a Kalman filter algorithm. The implemented system showed satisfactory performance in tests using actual robot-assisted surgery videos. Trajectory comparisons of automatically detected data and ground truth data obtained by manually locating the center of mass of each instrument were used to quantitatively validate the system. Instruments and collisions could be well tracked through the proposed methods. The developed collision warning system could provide valuable information to clinicians for safer procedures. [ABSTRACT FROM AUTHOR]

Tracking instruments during surgery is becoming a useful acquisition tool for different applications. This article presents a tracking system to detect and track instruments in endoscopic video using biocompatible colour markers. The system tracks single or multiple instruments in the video. The originality of this method is that it combines continuously adaptive shift algorithm with Kalman-filter for real-time tracking of single and multiple surgical instruments during surgery. Preliminary results show that the proposed method has a real-time performance. Moreover it is robust to partial occlusion and smoke. The system shows high sensitivity and specificity results for blue, green and yellow colours. The achieved sensitivity and specificity results are sufficient to apply the system for real-time registration of surgical workflow in-vivo during surgery. [ABSTRACT FROM AUTHOR]

As the number of robotic surgery procedures has increased, so has the importance of evaluating surgical skills in these techniques. It is difficult, however, to automatically and quantitatively evaluate surgical skills during robotic surgery, as these skills are primarily associated with the movement of surgical instruments. This study proposes a deep learning-based surgical instrument tracking algorithm to evaluate surgeons' skills in performing procedures by robotic surgery. This method overcame two main drawbacks: occlusion and maintenance of the identity of the surgical instruments. In addition, surgical skill prediction models were developed using motion metrics calculated from the motion of the instruments. The tracking method was applied to 54 video segments and evaluated by root mean squared error (RMSE), area under the curve (AUC), and Pearson correlation analysis. The RMSE was 3.52 mm, the AUC of 1 mm, 2 mm, and 5 mm were 0.7, 0.78, and 0.86, respectively, and Pearson's correlation coefficients were 0.9 on the x-axis and 0.87 on the y-axis. The surgical skill prediction models showed an accuracy of 83% with Objective Structured Assessment of Technical Skill (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS). The proposed method was able to track instruments during robotic surgery, suggesting that the current method of surgical skill assessment by surgeons can be replaced by the proposed automatic and quantitative evaluation method. [ABSTRACT FROM AUTHOR]

In laparoscopic surgery, the surgeon must operate with a limited field of view and reduced depth perception. This makes spatial understanding of critical structures difficult, such as an endophytic tumour in a partial nephrectomy. Such tumours yield a high complication rate of 47%, and excising them increases the risk of cutting into the kidney's collecting system. To overcome these challenges, an augmented reality guidance system is proposed. Using intra-operative ultrasound, a single navigation aid, and surgical instrument tracking, four augmentations of guidance information are provided during tumour excision. Qualitative and quantitative system benefits are measured in simulated robot-assisted partial nephrectomies. Robot-to-camera calibration achieved a total registration error of 1.0 ± 0.4 mm while the total system error is 2.5 ± 0.5 mm. The system significantly reduced healthy tissue excised from an average (±standard deviation) of 30.6 ± 5.5 to 17.5 ± 2.4 cm 3 ( p < 0.05) and reduced the depth from the tumor underside to cut from an average (±standard deviation) of 10.2 ± 4.1 to 3.3 ± 2.3 mm ( p < 0.05). Further evaluation is required in vivo, but the system has promising potential to reduce the amount of healthy parenchymal tissue excised.