Search

To realize a higher-level autonomy of surgical knot tying in minimally invasive surgery (MIS), automated suture grasping, which bridges the suture stitching and looping procedures, is an important yet challenging task needs to be achieved. This paper presents a holistic framework with image-guided and automation techniques to robotize this operation even under complex environments. The whole task is initialized by suture segmentation, in which we propose a novel semi-supervised learning architecture featured with a suture-aware loss to pertinently learn its slender information using both annotated and unannotated data. With successful segmentation in stereo-camera, we develop a Sampling-based Sliding Pairing (SSP) algorithm to online optimize the suture’s 3D shape. By jointly studying the robotic configuration and the suture’s spatial characteristics, a target function is introduced to find the optimal grasping pose of the surgical tool with Remote Center of Motion (RCM) constraints. To compensate for inherent errors and practical uncertainties, a unified grasping strategy with a novel vision-based mechanism is introduced to autonomously accomplish this grasping task. Our framework is extensively evaluated from learning-based segmentation, 3D reconstruction, and image-guided grasping on the da Vinci Research Kit (dVRK) platform, where we achieve high performances and successful rates in perceptions and robotic manipulations. These results prove the feasibility of our approach in automating the suture grasping task, and this work fills the gap between automated surgical stitching and looping, stepping towards a higher-level of task autonomy in surgical knot tying. Note to Practitioners—This paper aims to automate the suture grasping task in surgical knot tying by leveraging stereo visual guidance. To effectively robotize this procedure, it requires multidisciplinary knowledge to achieve suture segmentation, 3D shape reconstruction, and reliable automated grasping, while there are no existing works tackling this procedure especially using robots with RCM kinematics constraints and under complex environments. In this article, we propose a learning-driven method along with a 3D shape optimizer, which can conduct the suture segmentation and output its accurate spatial coordinates, serving as guidance for automated grasping operation. Apart from this, we introduce a unified function to optimize the grasping pose, and a vision-based grasping strategy is also proposed to intelligently complete this task. The experiments extensively validate the feasibility of our framework for automated suture grasp, and its successful completion can serve as a basis for the following looping manipulation, hence filling a step gap in robot-assisted knot tying. This framework can be also encapsulated into the medical robotic system, and by simply indicating (e.g. mouse click) the rough position of the suture’s tip in one camera frame, the overall framework can be initialized and further accomplish the suture grasping task, which further prompts a full autonomy of surgical knot tying in the near future. [ABSTRACT FROM AUTHOR]

In this study, a system of automatic microparticle patterning that could enable the separation, trapping, and translation of single microbeads in liquid suspension using negative dielectrophoresis (DEP) tweezers was presented to form a single-bead pattern. A microchip with integrated electrodes was flipped and placed above the substrate through a micromanipulator. Microparticles laying on the substrate could be displaced to different positions relative to the electrodes on the microchip, and only the selected particles would be trapped by the electric fields generated from electrodes. Vision-based approaches were used to evaluate the necessary information, such as the gap distance and the positions of electrodes and microparticles in the image. A strategy for separating nearby particles was proposed to achieve single-bead patterning with high accuracy. A controller was used to guide the microparticles toward the position for trapping while avoiding flow disturbance. Different strategies were simulated to decrease the patterning time and find the minimum traveling distance and the best route of movement. The optimization problem is NP-hard. Hence, global optimization algorithms, such as genetic algorithm, particle swarm optimization, and ant colony optimization (ACO), were simulated, and the results were compared with those of the local optimization method. The comparison results showed that ACO obtained the best performance among the methods. The strategy for constructing high-quality microparticle patterns was also examined through experiments. Orange fluorescent polystyrene beads suspended in 6-aminohexanoic acid solution were considered and successfully patterned on a glass substrate by using the proposed system. Note to Practitioners—Micropatterning is an effective tool for pharmaceutical research and drug discovery. However, the reliability of results depends on the quality of patterns. Existing approaches, such as microfluidic devices, are limited to create a pattern from one chip for single use, and the entire process is sealed and isolated from the environment. In this study, a multielectrode microchip combined with a vision-based micromanipulator is introduced to create a novel noncontact approach for microparticle patterning, which offers high flexibility and guarantees the quality of the constructed patterns. The electrodes on the chip can be selectively energized to determine the shape of the final pattern. A real-time screening is performed so that the micromanipulator will only guide the particles in good condition for selection. An optimization algorithm is implemented to aid the particle selection with the electrodes, allowing high-quality microparticle patterns to be constructed in a short time for various applications. [ABSTRACT FROM AUTHOR]

A vision-based system presents one of the most reliable methods for achieving an automated robot-assisted manipulation associated with surgical knot tying. However, some challenges in suture thread detection and automated suture thread grasping significantly hinder the realization of a fully automated surgical knot tying. In this article, we propose a novel algorithm that can be used for computing the 3-D coordinates of a suture thread in knot tying. After proper training with our data set, we built a deep-learning model for accurately locating the suture’s tip. By applying a Hessian-based filter with multiscale parameters, the environmental noises can be eliminated while preserving the suture thread information. A multistencils fast marching method was then employed to segment the suture thread, and a precise stereomatching algorithm was implemented to compute the 3-D coordinates of this thread. Experiments associated with the precision of the deep-learning model, the robustness of the 2-D segmentation approach, and the overall accuracy of 3-D coordinate computation of the suture thread were conducted in various scenarios, and the results quantitatively validate the feasibility and reliability of the entire scheme for automated 3-D shape reconstruction. Note to Practitioners—This article was motivated by the challenges of suture thread detection and 3-D coordinate evaluation in a calibrated stereovision system. To precisely detect the suture thread with no distinctive feature in an image, additional information, such as the two ends of the suture thread or its total length, is usually required. This article suggests a new method utilizing a deep-learning model to automate the tip detection process, eliminating the need of manual click in the initial stage. After feature enhancements with image filters, a multistencils fast marching method was incorporated to compute the arrival time from the detected tip to other points on the suture contour. By finding the point that takes the maximal time to travel in a closed contour, the other end of the suture thread can be identified, thereby allowing suture threads of any length to be segmented out from an image. A precise stereomatching method was then proposed to generate matched key points of the suture thread on the image pair, thereby enabling the reconstruction of its 3-D coordinates. The accuracy and robustness of the entire suture detection scheme were validated through experiments with different backgrounds and lengths. This proposed scheme offers a new solution for detecting curvilinear objects and their 3-D coordinates, which shows potential in realizing automated suture grasping with robot manipulators. [ABSTRACT FROM AUTHOR]

Robot-assisted surgery has revolutionized the field of surgery over the past few decades. Despite many successes, achieving full automation in these surgeries remains a challenging task. In this paper, a dynamic approach is proposed to automate knot tying with an in-house robot vision system. Through efficient path planning and coordination between two grippers, the workspace required for constructing the suture loop can be reduced while issues such as suture slippage and collisions between instruments can be eliminated. Visual images were employed to monitor the two grippers in real time and their positions were evaluated using transformation matrices obtained experimentally. A linear quadratic control scheme was applied to optimize the tracking performance of the two grippers. From the experiments, this visual evaluation method can achieve a position accuracy of 1 mm in the workspace. The proposed algorithm was evaluated and automatic suture looping operation was successfully performed in all six trials. Different parameters in the control scheme were also examined by introducing external impulse disturbances during the knot-tying process. This proposed knot-tying approach demonstrates a simple and efficient way to construct a suture knot in a minimal workspace. Note to Practitioners—A surgical operation usually takes several hours to complete, which is a test of surgeon’s endurance. To better assist surgeons, several repetitive surgical tasks can be performed automatically with a robotic system. This paper presents a new method for robot-assisted surgical knot tying in a confined environment. Through simultaneous manipulation and trajectory planning of the two surgical grippers, suture loops were constructed while keeping the suture in tension. Images were employed to guide the gripers in completing the knot-tying process with high precision. Experiments were conducted to demonstrate the automated suture looping operation with the system and the proposed method. The robustness of the method was also examined by introducing impulse disturbances and occlusions during the experiments. This proposed method does not require complicated hardware setup, allowing easy implementations on various surgical systems. [ABSTRACT FROM AUTHOR]