Search

The field of ophthalmic surgery demands accurate identification of specialized surgical instruments. Manual recognition can be time-consuming and prone to errors. In recent years neural networks have emerged as promising techniques for automating the classification process. However, the deployment of these advanced algorithms requires the collection of large amounts of data and a painstaking process of tagging selected elements. This paper presents a novel investigation into the application of neural networks for the detection and classification of surgical instruments in ophthalmic surgery. The main focus of the research is the application of active learning techniques, in which the model is trained by selecting the most informative instances to expand the training set. Various active learning methods are compared, with a focus on their effectiveness in reducing the need for significant data annotation -- a major concern in the field of surgery. The use of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to achieve high performance in the task of surgical tool detection is outlined. The combination of artificial intelligence (AI), machine learning, and Active Learning approaches, specifically in the field of ophthalmic surgery, opens new perspectives for improved diagnosis and surgical planning, ultimately leading to an improvement in patient safety and treatment outcomes. [ABSTRACT FROM AUTHOR]

Abstract Background Retained surgical items (RSI) are preventable events that pose a significant risk to patient safety. Current strategies for preventing RSIs rely heavily on manual instrument counting methods, which are prone to human error. This study evaluates the feasibility and performance of a deep learning-based computer vision model for automated surgical tool detection and counting. Methods A novel dataset of 1,004 images containing 13,213 surgical tools across 11 categories was developed. The dataset was split into training, validation, and test sets at a 60:20:20 ratio. An artificial intelligence (AI) model was trained on the dataset, and the model’s performance was evaluated using standard object detection metrics, including precision and recall. To simulate a real-world surgical setting, model performance was also evaluated in a dynamic surgical video of instruments being moved in real-time. Results The model demonstrated high precision (98.5%) and recall (99.9%) in distinguishing surgical tools from the background. It also exhibited excellent performance in differentiating between various surgical tools, with precision ranging from 94.0 to 100% and recall ranging from 97.1 to 100% across 11 tool categories. The model maintained strong performance on a subset of test images containing overlapping tools (precision range: 89.6–100%, and recall range 97.2–98.2%). In a real-time surgical video analysis, the model maintained a correct surgical tool count in all non-transition frames, with a median inference speed of 40.4 frames per second (interquartile range: 4.9). Conclusion This study demonstrates that using a deep learning-based computer vision model for automated surgical tool detection and counting is feasible. The model’s high precision and real-time inference capabilities highlight its potential to serve as an AI safeguard to potentially improve patient safety and reduce manual burden on surgical staff. Further validation in clinical settings is warranted.

Background: Retained surgical items (RSI) are preventable events that pose a significant risk to patient safety. Current strategies for preventing RSIs rely heavily on manual instrument counting methods, which are prone to human error. This study evaluates the feasibility and performance of a deep learning-based computer vision model for automated surgical tool detection and counting. Methods: A novel dataset of 1,004 images containing 13,213 surgical tools across 11 categories was developed. The dataset was split into training, validation, and test sets at a 60:20:20 ratio. An artificial intelligence (AI) model was trained on the dataset, and the model's performance was evaluated using standard object detection metrics, including precision and recall. To simulate a real-world surgical setting, model performance was also evaluated in a dynamic surgical video of instruments being moved in real-time. Results: The model demonstrated high precision (98.5%) and recall (99.9%) in distinguishing surgical tools from the background. It also exhibited excellent performance in differentiating between various surgical tools, with precision ranging from 94.0 to 100% and recall ranging from 97.1 to 100% across 11 tool categories. The model maintained strong performance on a subset of test images containing overlapping tools (precision range: 89.6–100%, and recall range 97.2–98.2%). In a real-time surgical video analysis, the model maintained a correct surgical tool count in all non-transition frames, with a median inference speed of 40.4 frames per second (interquartile range: 4.9). Conclusion: This study demonstrates that using a deep learning-based computer vision model for automated surgical tool detection and counting is feasible. The model's high precision and real-time inference capabilities highlight its potential to serve as an AI safeguard to potentially improve patient safety and reduce manual burden on surgical staff. Further validation in clinical settings is warranted. [ABSTRACT FROM AUTHOR]

Introduction: Collaborative robots, designed to work alongside humans for manipulating end-effectors, greatly benefit from the implementation of active constraints. This process comprises the definition of a boundary, followed by the enforcement of some control algorithm when the robot tooltip interacts with the generated boundary. Contact with the constraint boundary is communicated to the human operator through various potential forms of feedback. In fields like surgical robotics, where patient safety is paramount, implementing active constraints can prevent the robot from interacting with portions of the patient anatomy that shouldn't be operated on. Despite improvements in orthopaedic surgical robots, however, there exists a gap between bulky systems with haptic feedback capabilities and miniaturised systems that only allow for boundary control, where interaction with the active constraint boundary interrupts robot functions. Generally, active constraint generation relies on optical tracking systems and preoperative imaging techniques. Methods: This paper presents a refined version of the Signature Robot, a three degrees-of-freedom, hands-on collaborative system for orthopaedic surgery. Additionally, it presents a method for generating and enforcing active constraints "on-the-fly" using our previously introduced monocular, RGB, camera-based network, SimPS-Net. The network was deployed in real-time for the purpose of boundary definition. This boundary was subsequently used for constraint enforcement testing. The robot was utilised to test two different active constraints: a safe region and a restricted region. Results: The network success rate, defined as the ratio of correct over total object localisation results, was calculated to be 54.7% ± 5.2%. In the safe region case, haptic feedback resisted tooltip manipulation beyond the active constraint boundary, with a mean distance from the boundary of 2.70 mm ± 0.37 mm and a mean exit duration of 0.76 s ± 0.11 s. For the restricted-zone constraint, the operator was successfully prevented from penetrating the boundary in 100% of attempts. Discussion: This paper showcases the viability of the proposed robotic platform and presents promising results of a versatile constraint generation and enforcement pipeline. [ABSTRACT FROM AUTHOR]

Introduction: Collaborative robots, designed to work alongside humans for manipulating end-effectors, greatly benefit from the implementation of active constraints. This process comprises the definition of a boundary, followed by the enforcement of some control algorithm when the robot tooltip interacts with the generated boundary. Contact with the constraint boundary is communicated to the human operator through various potential forms of feedback. In fields like surgical robotics, where patient safety is paramount, implementing active constraints can prevent the robot from interacting with portions of the patient anatomy that shouldn’t be operated on. Despite improvements in orthopaedic surgical robots, however, there exists a gap between bulky systems with haptic feedback capabilities and miniaturised systems that only allow for boundary control, where interaction with the active constraint boundary interrupts robot functions. Generally, active constraint generation relies on optical tracking systems and preoperative imaging techniques.Methods: This paper presents a refined version of the Signature Robot, a three degrees-of-freedom, hands-on collaborative system for orthopaedic surgery. Additionally, it presents a method for generating and enforcing active constraints “on-the-fly” using our previously introduced monocular, RGB, camera-based network, SimPS-Net. The network was deployed in real-time for the purpose of boundary definition. This boundary was subsequently used for constraint enforcement testing. The robot was utilised to test two different active constraints: a safe region and a restricted region.Results: The network success rate, defined as the ratio of correct over total object localisation results, was calculated to be 54.7% ± 5.2%. In the safe region case, haptic feedback resisted tooltip manipulation beyond the active constraint boundary, with a mean distance from the boundary of 2.70 mm ± 0.37 mm and a mean exit duration of 0.76 s ± 0.11 s. For the restricted-zone constraint, the operator was successfully prevented from penetrating the boundary in 100% of attempts.Discussion: This paper showcases the viability of the proposed robotic platform and presents promising results of a versatile constraint generation and enforcement pipeline.

Surgical tool detection in minimally invasive surgery is an essential part of computer-assisted interventions. Current approaches are mostly based on supervised methods requiring large annotated datasets. However, labelled datasets are often scarce. Semi-supervised learning (SSL) has recently emerged as a viable alternative showing promise in producing models retaining competitive performance to supervised methods. Therefore, this paper introduces an SSL framework in the surgical tool detection paradigm, which aims to mitigate training data scarcity and data imbalance problems through a knowledge distillation approach. In the proposed work, we train a model with labelled data which initialises the Teacher-Student joint learning, where the Student is trained on Teacher-generated pseudo-labels from unlabelled data. We also propose a multi-class distance with a margin-based classification loss function in the region-of-interest head of the detector to segregate the foreground-background region effectively. Our results on m2cai16-tool-locations dataset indicates the superiority of our approach on different supervised data settings (1%, 2%, 5% and 10% of annotated data) where our model achieves overall improvements of 8%, 12%, and 27% in mean average precision on 1% labelled data over the state-of-the-art SSL methods and the supervised baseline, respectively. The code is available at . [ABSTRACT FROM AUTHOR]

Purpose: Computer assistance for endoscopic surgery depends on knowledge about the contents in an endoscopic scene. An important step of analysing the video contents is real-time surgical tool detection. Most methods for tool detection nevertheless depend on multi-step algorithms building upon prior knowledge like anchor boxes or non-maximum suppression which ultimately decrease performance. A real-world difficulty encountered by learning-based methods are limited datasets. Training a neural network on data matching a specific distribution (e.g. from a single hospital or showing a specific type of surgery) can result in a lack of generalization. Methods: In this paper, we propose the application of a transformer based architecture for end-to-end tool detection. This architecture promises state-of-the-art accuracy while decreasing the complexity resulting in improved run-time performance. To improve the lack of cross-domain generalization due to limited datasets, we enhance the architecture with a latent feature space via variational encoding to capture common intra-domain information. This feature space models the linear dependencies between domains by constraining their rank. Results: The trained neural networks show a distinct improvement on out-of-domain data indicating better generalization to unseen domains. Inference with the end-to-end architecture can be performed at up to 138 frames per second (FPS) achieving a speedup in comparison to older approaches. Conclusions: Experimental results on three representative datasets demonstrate the performance of the method. We also show that our approach leads to better domain generalization. [ABSTRACT FROM AUTHOR]

Detecting surgical tools is an essential task for analyzing and evaluating surgical videos. However, most studies focus on minimally invasive surgery (MIS) and cataract surgery. Mainly because of a lack of a large, diverse, and well-annotated dataset, research in the area of open surgery has been limited so far. Open surgery video analysis is challenging because of its properties: varied number and roles of people (e.g., main surgeon, assistant surgeons, and nurses), a complex interaction of tools and hands, various operative environments, and lighting conditions. In this paper, to handle these limitations and difficulties, we introduce an egocentric open surgery dataset that includes 15 open surgeries recorded with a head-mounted camera. More than 67k bounding boxes are labeled to 19k images with 31 surgical tool categories. Finally, we present a surgical tool detection baseline model based on recent advances in object detection. The results of our new dataset show that our presented dataset provides enough interesting challenges for future methods and that it can serve as a strong benchmark to address the study of tool detection in open surgery. [ABSTRACT FROM AUTHOR]

Accurate recognition of surgical tools is a crucial component in the development of robust, context-aware systems. Recently, deep learning methods have been increasingly adopted to analyse laparoscopic videos. Existing work mainly leverages the ability of convolutional neural networks (CNNs) to model visual information of laparoscopic images. However, the performance was evaluated only on data belonging to the same dataset used for training. A more comprehensive evaluation of CNN performance on data from other datasets can provide a more rigorous assessment of the approaches. In this work, we investigate the generalisation capability of different CNN architectures to classify surgical tools in laparoscopic images recorded at different institutions. This research highlights the need to determine the effect of using data from different surgical sites on CNN generalisability. Experimental results imply that training a CNN model using data from multiple sites improves generalisability to new surgical locations.

Medical instruments detection in laparoscopic video has been carried out to increase the autonomy of surgical robots, evaluate skills or index recordings. However, it has not been extended to surgical gauzes. Gauzes can provide valuable information to numerous tasks in the operating room, but the lack of an annotated dataset has hampered its research. In this article, we present a segmentation dataset with 4003 hand-labelled frames from laparoscopic video. To prove the dataset potential, we analyzed several baselines: detection using YOLOv3, coarse segmentation, and segmentation with a U-Net. Our results show that YOLOv3 can be executed in real time but provides a modest recall. Coarse segmentation presents satisfactory results but lacks inference speed. Finally, the U-Net baseline achieves a good speed-quality compromise running above 30 FPS while obtaining an IoU of 0.85. The accuracy reached by U-Net and its execution speed demonstrate that precise and real-time gauze segmentation can be achieved, training convolutional neural networks on the proposed dataset. [ABSTRACT FROM AUTHOR]

Surgical tool presence detection in laparoscopic videos is a challenging problem that plays a critical role in developing context-aware systems in operating rooms (ORs). In this work, we propose a deep learning-based approach for detecting surgical tools in laparoscopic images using a convolutional neural network (CNN) in combination with two long short-term memory (LSTM) models. A pre-trained CNN model was trained to learn visual features from images. Then, LSTM was employed to include temporal information through a video clip of neighbour frames. Finally, the second LSTM was utilized to model temporal dependencies across the whole surgical video. Experimental evaluation has been conducted with the Cholec80 dataset to validate our approach. Results show that the most notable improvement is achieved after employing the two-stage LSTM model, and the proposed approach achieved better or similar performance compared with state-of-the-art methods.

To enhance surgeons' efficiency and safety of patients, minimally invasive surgery (MIS) is widely used in a variety of clinical surgeries. Real-time surgical tool detection plays an important role in MIS. However, most methods of surgical tool detection may not achieve a good trade-off between detection speed and accuracy. We propose a real-time attention-guided convolutional neural network (CNN) for frame-by-frame detection of surgical tools in MIS videos, which comprises a coarse (CDM) and a refined (RDM) detection modules. The CDM is used to coarsely regress the parameters of locations to get the refined anchors and perform binary classification, which determines whether the anchor is a tool or background. The RDM subtly incorporates the attention mechanism to generate accurate detection results utilizing the refined anchors from CDM. Finally, a light-head module for more efficient surgical tool detection is proposed. The proposed method is compared to eight state-of-the-art detection algorithms using two public (EndoVis Challenge and ATLAS Dione) datasets and a new dataset we introduced (Cholec80-locations), which extends the Cholec80 dataset with spatial annotations of surgical tools. Our approach runs in real-time at 55.5 FPS and achieves 100, 94.05, and 91.65% mAP for the above three datasets, respectively. Our method achieves accurate, fast, and robust detection results by end-to-end training in MIS videos. The results demonstrate the effectiveness and superiority of our method over the eight state-of-the-art methods.

Robot-assisted surgery (RAS), a type of minimally invasive surgery, is used in a variety of clinical surgeries because it has a faster recovery rate and causes less pain. Automatic video analysis of RAS is an active research area, where precise surgical tool detection in real time is an important step. However, most deep learning methods currently employed for surgical tool detection are based on anchor boxes, which results in low detection speeds. In this paper, we propose an anchor-free convolutional neural network (CNN) architecture, a novel frame-by-frame method using a compact stacked hourglass network, which models the surgical tool as a single point: the center point of its bounding box. Our detector eliminates the need to design a set of anchor boxes, and is end-to-end differentiable, simpler, more accurate, and more efficient than anchor-box-based detectors. We believe our method is the first to incorporate the anchor-free idea for surgical tool detection in RAS videos. Experimental results show that our method achieves 98.5% mAP and 100% mAP at 37.0 fps on the ATLAS Dione and Endovis Challenge datasets, respectively, and truly realizes real-time surgical tool detection in RAS videos.

Detecting surgical tools is an essential task for analyzing and evaluating surgical videos. However, most studies focus on minimally invasive surgery (MIS) and cataract surgery. Mainly because of a lack of a large, diverse, and well-annotated dataset, research in the area of open surgery has been limited so far. Open surgery video analysis is challenging because of its properties: varied number and roles of people (e.g., main surgeon, assistant surgeons, and nurses), a complex interaction of tools and hands, various operative environments, and lighting conditions. In this paper, to handle these limitations and difficulties, we introduce an egocentric open surgery dataset that includes 15 open surgeries recorded with a head-mounted camera. More than 67k bounding boxes are labeled to 19k images with 31 surgical tool categories. Finally, we present a surgical tool detection baseline model based on recent advances in object detection. The results of our new dataset show that our presented dataset provides enough interesting challenges for future methods and that it can serve as a strong benchmark to address the study of tool detection in open surgery.

Medical instruments detection in laparoscopic video has been carried out to increase the autonomy of surgical robots, evaluate skills or index recordings. However, it has not been extended to surgical gauzes. Gauzes can provide valuable information to numerous tasks in the operating room, but the lack of an annotated dataset has hampered its research. In this article, we present a segmentation dataset with 4003 hand-labelled frames from laparoscopic video. To prove the dataset potential, we analyzed several baselines: detection using YOLOv3, coarse segmentation, and segmentation with a U-Net. Our results show that YOLOv3 can be executed in real time but provides a modest recall. Coarse segmentation presents satisfactory results but lacks inference speed. Finally, the U-Net baseline achieves a good speed-quality compromise running above 30 FPS while obtaining an IoU of 0.85. The accuracy reached by U-Net and its execution speed demonstrate that precise and real-time gauze segmentation can be achieved, training convolutional neural networks on the proposed dataset.

Laparoscopic videos are a very important source of information which is inherently available in minimally invasive surgeries. Detecting surgical tools based on that videos have gained increasing interest due to its importance in developing a context-aware system. Such system can provide guidance assistance to the surgical team and optimise the processes inside the operating room. Convolutional neural network is a robust method to learn discriminative visual features and classify objects. As it expects a uniform distribution of data over classes, it fails to identify classes which are under-presented in the training data. In this work, loss-sensitive learning approach and resampling techniques were applied to counter the negative effects of imbalanced laparoscopic data on training the CNN model. The obtained results showed improvement in the classification performance especially for detecting surgical tools which are shortly used in the procedure.

Purpose: A fully automated surgical tool detection framework is proposed for endoscopic video streams. State-of-the-art surgical tool detection methods rely on supervised one-vs-all or multi-class classification techniques, completely ignoring the co-occurrence relationship of the tools and the associated class imbalance. Methods: In this paper, we formulate tool detection as a multi-label classification task where tool co-occurrences are treated as separate classes. In addition, imbalance on tool co-occurrences is analyzed and stratification techniques are employed to address the imbalance during convolutional neural network (CNN) training. Moreover, temporal smoothing is introduced as an online post-processing step to enhance runtime prediction. Results: Quantitative analysis is performed on the M2CAI16 tool detection dataset to highlight the importance of stratification, temporal smoothing and the overall framework for tool detection. Conclusion: The analysis on tool imbalance, backed by the empirical results, indicates the need and superiority of the proposed framework over state-of-the-art techniques. [ABSTRACT FROM AUTHOR]

Surgical tool detection is a recently active research area. It is the foundation to a series of advanced surgical support functions, such as image guided surgical navigation, forming safety zone between surgical tools and sensitive tissues. Previous methods rely on two types of information: tool locating signals and vision features. Collecting tool locating signals requires additional hardware equipments. Vision based methods train their detection models using strong annotations (e.g. bounding boxes), which are quite rare and expensive to acquire in the field of surgical image understanding. In this paper, we propose a Pseudo Supervised surgical Tool detection (PSTD) framework, which performs explicit detection refinement by three levels of associated measures (pseudo bounding box generation, real box regression, weighted boxes fusion) in a weakly supervised manner. On the basis of PSTD, we develop a Bi-directional Adaption Weighting (BAW) mechanism in our tool classifier for contextual information mining by creating competition or cooperation relationships between channels. By only using image-level tool category labels, the proposed method yields state-of-the-art results with 87.0% mAP on a mainstream surgical image dataset: Cheloc80. [ABSTRACT FROM AUTHOR]