Search

Intra-operative local 3D vessel representations have the potential to significantly decrease the use of contrast agents and exposure to ionizing radiation during endovascular procedures, while overcoming the 2D visualization limitation of fluoroscopic guidance. By fusing intravascular ultrasound (IVUS) imaging and electromagnetic (EM) pose sensing in a robotic catheter tip, a real-time local 3D model of the vasculature could be constructed intra-operatively. This letter proposes the use of a cylinder model to approximate the vessel geometry near the catheter tip. An unscented Kalman filter is employed to robustly estimate the cylinder that best fits IVUS and EM data while navigating through the vessel. This forms a radiation-free alternative to conventional radiation-based guidance. Validation on one in silico and two in vitro models showed median estimation errors of cylinder radius of 0.14 mm, 0.42 mm and 0.70 mm; cylinder position of 0.45 mm, 1.07 mm and 0.96 mm; and cylinder orientation of $2.94^\circ$ , $4.60^\circ$ and $3.03^\circ$ showing great potential for helping interventionists preventing harmful interactions between the instrument tip and the vessel wall.

Force sensing is highly desirable in minimally invasive medical applications, since this feature shows great potential for reducing tissue damage and enhancing manipulation safety. However, embedding force sensors in medical devices is challenging and costly. This article explores the possibility to use shape sensing as a measure to extract force information. In this work, a model-based approach that allows simultaneous shape and force sensing is proposed. Shape information is reconstructed employing a multicore fiber with fiber Bragg grating sensors spaced over the fiber length. This fiber is capable of distributed 3D shape sensing. It is shown how by making use of extended Kalman filter and a mechanics model of the flexible instrument, it becomes possible to estimate both the magnitudes and locations of externally applied forces. Experiments were carried out to validate the proposed method for both one and two external forces applied at arbitrary locations in different directions on a flexible instrument. Results show that one-directional force magnitude and location can be estimated with an average error of 23.08 mN (15.39%) and 11.06 mm (6.51%), respectively. For two-directional forces, results of the load near the base show an average error of 52.01 mN (30.59%) for the magnitude and 29.24 mm (17.20%) for the location. For the load applied simultaneously near the tip, the mean magnitude error is 16.79 mN (11.19%) and the average location error is 10.18 mm (5.99%). The force sensing algorithm can run in real time with an approximate frequency of 59 Hz. In these experiments, it can be observed that the force-sensing accuracy, which depends on the sensitivity of the shape of flexible instruments with respect to the external force, can vary drastically in function of the force application point and force direction.

The emergence of steerable flexible instruments has widened the uptake of minimally invasive surgical techniques. In sinus surgery, such flexible instruments could enable the access to difficult-to-reach anatomical areas. However, design-oriented metrics, essential for the development of steerable flexible instruments for maxillary sinus surgery, are still lacking. This paper proposes a method to process measurements and provides the instrument designer with essential information to develop adapted flexible instruments for limited access surgery. This method was applied to maxillary sinus surgery and showed that an instrument with a diameter smaller than 2.4 mm can be used on more than 72.5% of the subjects’ set. Based on the statistical analysis and provided that this flexible instrument can bend up to $$164.4^\circ,$$ it is estimated that all areas within the maxillary sinus could be reached through a regular antrostomy without resorting to extra incision or tissue removal in 94.9% of the population set. The presented method was partially validated by conducting cadaver experiments.

Purpose Fetoscopic laser photocoagulation is a minimally invasive surgery for the treatment of twin-to-twin transfusion syndrome (TTTS). By using a lens/fibre-optic scope, inserted into the amniotic cavity, the abnormal placental vascular anastomoses are identified and ablated to regulate blood flow to both fetuses. Limited field-of-view, occlusions due to fetus presence and low visibility make it difficult to identify all vascular anastomoses. Automatic computer-assisted techniques may provide better understanding of the anatomical structure during surgery for risk-free laser photocoagulation and may facilitate in improving mosaics from fetoscopic videos. Methods We propose FetNet, a combined convolutional neural network (CNN) and long short-term memory (LSTM) recurrent neural network architecture for the spatio-temporal identification of fetoscopic events. We adapt an existing CNN architecture for spatial feature extraction and integrated it with the LSTM network for end-to-end spatio-temporal inference. We introduce differential learning rates during the model training to effectively utilising the pre-trained CNN weights. This may support computer-assisted interventions (CAI) during fetoscopic laser photocoagulation. Results We perform quantitative evaluation of our method using 7 in vivo fetoscopic videos captured from different human TTTS cases. The total duration of these videos was 5551 s (138,780 frames). To test the robustness of the proposed approach, we perform 7-fold cross-validation where each video is treated as a hold-out or test set and training is performed using the remaining videos. Conclusion FetNet achieved superior performance compared to the existing CNN-based methods and provided improved inference because of the spatio-temporal information modelling. Online testing of FetNet, using a Tesla V100-DGXS-32GB GPU, achieved a frame rate of 114 fps. These results show that our method could potentially provide a real-time solution for CAI and automating occlusion and photocoagulation identification during fetoscopic procedures.

INTRODUCTION: Monopolar electrocautery is the most common dissection and coagulation tool during transoral robotic surgery (TORS) but causes significant collateral tissue damage as opposed to CO2 laser. We aimed at combining both modalities in one robotic instrument arm. METHODS: We developed a steerable CO2 -laser fibre carrier serving as an add-on to the existing Endowrist® monopolar spatula of the Da Vinci Xi. Feasibility and safety were assessed in a preclinical setting. RESULTS: One radical tonsillectomy with monopolar cautery and three with the instrument prototype were performed in two cadavers by two surgeons. No serious prototype-related intra-operative difficulties were observed. Safe and efficient switching between energy sources proved possible in all simulated intra-operative bleeding events. Prototype use allowed for the identification of the majority of key anatomical structures and was scored favourably on NASA-TLX questionnaires. DISCUSSION: The reported prototype successfully combines the advantages of CO2 -laser with the advantages of TORS. ispartof: INTERNATIONAL JOURNAL OF MEDICAL ROBOTICS AND COMPUTER ASSISTED SURGERY vol:18 issue:1 ispartof: location:England status: published

The twin-to-twin transfusion syndrome is a severe fetal anomaly appearing in up to 15% of identical twin pregnancies. This anomaly occurs when twins share blood vessels from a common placenta. The complication leads to an unbalanced blood transfusion between both fetuses. A current surgical treatment consists in coagulating the shared vessels using a fetoscope with an embedded laser. Such treatment is very delicate and constraining due to limited vision and size of the insertion area. The rigidity and lack of controllability of the current used instruments add an additional difficulty and limit the choice in insertion site. This letter proposes an improved flexible fetoscope, offering an enhanced laser controllability and higher versatility regarding the location of the insertion site. A better approach angle can therefore be realized. Also, tissue damage may be further reduced. This single-handed controllable active fetoscope is obtained after adaptation of a LithoVue (Boston Scientific, Natick, MA, USA), a commercially available passive flexible ureteroscope. The LithoVue is fitted with a unique lightweight add-on actuation module foreseen of an artificial muscle and a dedicated control system. Experiments in a mixed reality trainer suggested that the proposed fetoscope is compact, ergonomic, and intuitive in use, allowing an adequate control of the flexible end.

A variety of medical treatment and diagnostic procedures rely on flexible instruments such as catheters and endoscopes to navigate through tortuous and soft anatomies like the vasculature. Knowledge of the interaction forces between these flexible instruments and patient anatomy is extremely valuable. This can aid interventionalists in having improved awareness and decision-making abilities, efficient navigation, and increased procedural safety. In many applications, force interactions are inherently distributed. While knowledge of their locations and magnitudes is highly important, retrieving this information from instruments with conventional dimensions is far from trivial. Robust and reliable methods have not yet been found for this purpose. In this work, we present two new approaches to estimate the location, magnitude, and number of external point and distributed forces applied to flexible and elastic instrument bodies. Both methods employ the knowledge of the instrument's curvature profile. The former is based on piecewise polynomial-based curvature segmentation, whereas the latter on model-based parameter estimation. The proposed methods make use of Cosserat rod theory to model the instrument and provide force estimates at rates over 30 Hz. Experiments on a Nitinol rod embedded with a multi-core fiber, inscribed with fiber Bragg gratings, illustrate the feasibility of the proposed methods with mean force error reaching 7.3% of the maximum applied force, for the point load case. Furthermore, simulations of a rod subjected to two distributed loads with varying magnitudes and locations show a mean force estimation error of 1.6% of the maximum applied force. ispartof: Frontiers in Robotics and AI vol:8 pages:232-232 ispartof: location:Switzerland status: published

Endovascular catheterization is an intervention which offers a low risk alternative to open surgery in many patients. Today’s interventions rely heavily on fluoroscopic imaging to guide interventionalists. Fluoroscopy only produces 2D visualization of the catheter and also exposes both the patient and interventionalists to harmful radiation. Different approaches have been proposed to overcome the limitations of fluoroscopy. Fiber Bragg Grating (FBG)-based shape sensing is becoming popular to reconstruct the catheter shape. Multi-core fibers with parallel optical cores are interesting as they allow 3D shape reconstruction with a single fiber. A common issue with FBG-based shape sensing is its sensitivity to variations in twist. Even small amounts of twist can significantly impact the overall shape reconstruction accuracy. This work proposes a novel approach which combines electromagnetic tracking (EMT), FBG-based shape sensing, and sparse fluoroscopic images. The method provides real-time 3D visualization of the catheter without the need of continuous fluoroscopy. A unique feature of the proposed method is the selective use of imaging for dynamic twist-compensation of the FBG sensor. The proposed sensor-fusion method improved 3D reconstruction accuracy. Real-world in-vitro experiments promising results. For a catheter with an embedded fiber length of 170 mm, the proposed approach the 3D shape with a median root-mean-square (rms) error of 0.39 mm and an interquartile range of 0.10 mm in the 2D experiment in which the catheter was bent in a plane. A median rms error of 0.54 mm and an interquartile range of 0.07 mm were achieved in the 3D experiments. ispartof: Ieee Sensors Journal vol:21 issue:20 pages:1-12 status: Published online

Surface exploration in virtual reality has a large potential to enrich the user’s experience. It could for example be used to train and simulate medical palpation. During palpation, users tap, indent, and rub the surface of a sample to estimate the underlying properties. However, up to now there is no good approach to render such intricate interaction realistically. This paper introduces 6 degrees of freedom (DoF) encountered-type haptic display technology for simulating surface exploration tasks. Among the different phases of exploration, this article focuses on the ‘in-contact sliding’ phase. Two novel control approaches to render sliding over a virtual surface are elaborated. A first rendering method generates lateral frictional forces as the finger slides over the surface. A second method adjusts the inclination of the end-effector to render tissue properties. With both methods a stiff nodule embedded in a soft tissue was prepared. User experiments were carried out to find proper parameter and intensity ranges and to confirm the feasibility of the new rendering schemes. Participants indicated that both rendering schemes felt realistic. Compared to earlier work, where only the vertical stiffness was altered, lower thresholds to detect and localise embedded virtual nodules were found. Users also made fewer errors in detecting nodule edges. Furthermore, the method that used end-effector inclination allowed faster discovery of the nodule’s edges. It is expected that approaches that combine both rendering methods could provide an even more realistic feel.

The Failure Mode and Effect Analysis (FMEA) is a useful instrument born in the aerospace industry and widely used to improve a process or product's efficiency. Over the years, this instrument has been adopted in increasingly different contexts, such as HealthCare. This paper proposes an approach aimed at improving the defects typical of the classic FMEA in the design phase, that is, in a scenario full of uncertainties and with little information available, using a new hybrid Multiple-criteria decision-making (MCDM) method in order to obtain a priority index more performant than Risk Priority Number (RPN). In the proposed method, the three assessment criteria have a different weighting in the index's final computation, differently from the classical RPN. These weights are obtained with a scientific technique, thus avoiding that excessive subjectivity influences the final result. A more efficient priority index is obtained through a new hybrid approach that solves some classical RPN gaps. A case study concerning an endoscope Ear Nose Throat Entropy (ENT) prototype is examined to illustrate the proposed method. FMEA analysis in HealthCare is increasingly used for its flexibility and reliability. This study focuses on using new techniques to eliminate certain defects or exploit some qualities better. The use of a robust and elastic innovative MCDM method to calculate a new priority index and a scientific technique to obtain the weight of the selection are the interesting insights proposed in this paper. © 2021 John Wiley & Sons Ltd.

Purpose Fetoscopic laser photocoagulation is a minimally invasive surgical procedure used to treat twin-to-twin transfusion syndrome (TTTS), which involves localization and ablation of abnormal vascular connections on the placenta to regulate the blood flow in both fetuses. This procedure is particularly challenging due to the limited field of view, poor visibility, occasional bleeding, and poor image quality. Fetoscopic mosaicking can help in creating an image with the expanded field of view which could facilitate the clinicians during the TTTS procedure. Methods We propose a deep learning-based mosaicking framework for diverse fetoscopic videos captured from different settings such as simulation, phantoms, ex vivo, and in vivo environments. The proposed mosaicking framework extends an existing deep image homography model to handle video data by introducing the controlled data generation and consistent homography estimation modules. Training is performed on a small subset of fetoscopic images which are independent of the testing videos. Results We perform both quantitative and qualitative evaluations on 5 diverse fetoscopic videos (2400 frames) that captured different environments. To demonstrate the robustness of the proposed framework, a comparison is performed with the existing feature-based and deep image homography methods. Conclusion The proposed mosaicking framework outperformed existing methods and generated meaningful mosaic, while reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.

F1exible fetoscopes have been introduced in prior literature as a new surgical tool to limit the stresses applied on the fetal membrane during fetoscopic laser surgery and therefore avoid iatrogenic preterm premature rupture of membranes. However, previous flexible fetoscopes are either too large or do not foresee all the tools that are essential to the success of the targeted procedure. This study proposes the design and control of a single-handed active flexible fetoscope. The instrument is equipped with a camera, a light source, and a working channel while remaining compact (350 mm length and 30 mm wide, with a shaft of 3 mm diameter) and lightweight (87g). It is proposed to use a modified generalized PrandtlIshlinski model for hysteresis compensation as control method. The experimental results indicate that this method allows the instrument to reach a positioning accuracy of up to 4.5%. This suggests that a modified generalized Prandtl-Ishlinski model can therefore be used to compensate for an entire system (e.g. a fetoscope) affected by hysteresis and friction.

Hysteroscopy is a widely used gynaecological procedure to evaluate and treat cervical and intra-uterine pathology. In the last few decades, technical refinements in the optics technology, surgical accessories and the reduction of the outer diameter of the instrument have made it possible to perform many hysteroscopic procedures, including some operative procedures, in the office setting and without any anesthesia. Mini-hysteroscopic procedures in the office setting are associated with less pain, lower complication rate and faster recovery compared hysteroscopic procedures in day surgery under general anesthesia.

During fetoscopic laser photocoagulation, a treatment for twin-to-twin transfusion syndrome (TTTS), the clinician first identifies abnormal placental vascular connections and laser ablates them to regulate blood flow in both fetuses. The procedure is challenging due to the mobility of the environment, poor visibility in amniotic fluid, occasional bleeding, and limitations in the fetoscopic field-of-view and image quality. Ideally, anastomotic placental vessels would be automatically identified, segmented and registered to create expanded vessel maps to guide laser ablation, however, such methods have yet to be clinically adopted. We propose a solution utilising the U-Net architecture for performing placental vessel segmentation in fetoscopic videos. The obtained vessel probability maps provide sufficient cues for mosaicking alignment by registering consecutive vessel maps using the direct intensity-based technique. Experiments on 6 different in vivo fetoscopic videos demonstrate that the vessel intensity-based registration outperformed image intensity-based registration approaches showing better robustness in qualitative and quantitative comparison. We additionally reduce drift accumulation to negligible even for sequences with up to 400 frames and we incorporate a scheme for quantifying drift error in the absence of the ground-truth. Our paper provides a benchmark for fetoscopy placental vessel segmentation and registration by contributing the first in vivo vessel segmentation and fetoscopic videos dataset.

During laparoscopic sacrocolpopexy, pelvic organ prolapse is repaired by suturing one side of a synthetic mesh around the vaginal vault while stapling the other end to the sacrum, restoring the anatomical position of the vagina. A perineal assistant positions and tensions the vault with a vaginal manipulator instrument to properly expose the vaginal tissue to the laparoscopic surgeon. A technical difficulty during this surgery is the loss of depth perception due to visualization of the patient's internals on a 2D screen. Especially during precise surgical tasks, a more natural way to understand the distance between the laparoscopic instruments and the surgical region of interest could be advantageous. This work describes an exploratory study to investigate the potential of introducing 3D visualization into this surgical intervention. More in particular, experimentation is conducted with autostereoscopic display technology. A mixed reality setup was constructed featuring a virtual reality model of the vagina, 2D and 3D visualization, a physical interface representing the tissue of the body wall and a tracking system to track instrument motion. An experiment was conducted whereby the participants had to navigate the instrument to a number of pre-defined locations under 2D or 3D visualization. Compared to 2D, a considerable reduction in average task time (-42.9 %), travelled path lenght (-31.8 %) and errors (-52.2 %) was observed when performing the experiment in 3D. Where this work demonstrated a potential benefit of autostereoscopic visualization with respect to 2D visualization, in future work we wish to investigate if there also exists a benefit when comparing this technology with conventional stereoscopic visualization and whether stereoscopy can be used for (semi-) automated guidance during robotic laparoscopy.

Depth perception remains a key challenge in vitreoretinal surgery. Currently users only have the view from an overhead stereo-microscope available, but this means of visualisation is quite restrictive. Optical Coherence Tomography (OCT) has been introduced in the past and is even integrated in a number of commercial systems to provide a more detailed depth vision, even showing subsurface structures up to a few millimeters below the surface. The intra-operative use of OCT, especially in combination with robotics, is still sub-optimal. At present one can get either a very slowly updating large volume scan (C-scan) or a faster but not aligned cross-sectional B-scan or an even more local single point A-scan at very high update rate. In this work we propose a model-mediated approach. As the posterior eye segment can be approximated as a sphere, we propose to model the retina with this simplified sphere model, the center and radius of which can be estimated in real time. A time-varying Kalman filter is proposed here in combination with an instrument-integrated optical fiber that provides high-frequency A-scans along the longitudinal instrument direction. The model and convergence of the model has been validated first in a simulation environment and subsequently in-silico using an OCT A-scan probe mounted on a co-manipulated vitreoretinal robotic system. The probe was manipulated to measure a 3D stereo lithographically printed spherical model of the posterior eye segment. The feasibility of the proposed method was demonstrated in various scenarios. For the in-silico validation a 20 micrometer error and convergence speed of 2.0 seconds was found when sampling A-scans at 200Hz.

During robot-assisted surgery, a Remote Center of Motion (RCM) is often implemented to constrain instrument motion through and about a specific point in space. Aligning and re-positioning this point during surgery is not trivial, as this is a defined, yet often non-visualised, point in space. When misaligned with the patient surrounding anatomy is excessively strained, potentially causing post-operative complications. Not being able to safely re-position the RCM for these purposes limits the use of surgical robotics. This work introduces a general approach relying on anatomy-based haptic fixtures to simplify and improve RCM positioning during robot-assisted surgery. The proposed approach is extended with a novel method and mechanism to mechanically implement such fixtures. This is applied for the use case of vitreoretinal surgery, for which purpose a dedicated robotic setup and fixture mechanism was developed. These were used to conduct an initial experimental validation, during which the feasibility of both a virtual and mechanical implementation is reviewed. Initial outcomes suggest that both implementations are feasible. With the currently used impedance-type system, the mechanical implementation is shown to offer at least one order of magnitude stiffness increase when compared to an equivalent virtual implementation.

The instruments currently used by surgeons for in utero treatment of the twin-to-twin transfusion syndrome (TTTS) are rigid or semi-rigid. Their poor dexterity makes this surgical intervention risky and the surgeon's work very complex. This paper proposes the design, assembly and quantitative evaluation of an add-on system intended to be placed on a commercialized cable-driven flexible endoscope. The add-on system is lightweight and easily exchangeable thanks to the McKibben muscle actuators embedded in its system. The combination of the flexible endoscope and the new add-on unit results in an easy controllable flexible instrument with great potential use in TTTS treatment, and especially for regions that are hard to reach with conventional instruments. The fetoscope has a precision of 7.4% over its entire bending range and allows to decrease the maximum planar force on the body wall of 6.15% compared to the original endoscope. The add-on control system also allows a more stable and precise actuation of the endoscope flexible tip. ispartof: FRONTIERS IN ROBOTICS AND AI vol:6 ispartof: location:Switzerland status: published

In minimally invasive surgery (MIS), the incision point acts as a fulcrum about which the surgical instrument pivots. Most robotic MIS systems foresee procedures to carefully align the robot with the incision in the patient. Such procedures disrupt the normal workflow. This hampers the clinical translation of such assistive technology. In contrast, backdrivable, wristed (BW) robots, especially when operated as comanipulation devices, have a minimal impact on the traditional surgical workflow in MIS, as they are readily equipped with algorithms that automate fulcrum point estimation. To improve safety, accuracy and/or task completion time, virtual fixtures can be implemented at the distal instrument tip, using an assistive BW system. This letter formalizes the problems related to such distal virtual fixtures. It then develops a hybrid active/passive wrist control strategy to improve the performance of virtual fixtures with BW systems while minimizing stability issues. Aside from an analytical example, experimental validation is added to show that the new algorithm increases the achievable stiffness of distal virtual fixtures.

Hysteresis phenomenon limits the effective use of pneumatic artificial muscles in numerous driving applications. Experimental results show widening hysteresis loops and lowering displacement amplitudes by increasing excitation frequency. In this study, a feedforward controller is proposed to compensate for this hysteretic behavior. This proposed controller is based on a rate-dependent Prandtl–Ishlinskii model. In order to avoid analytical calculation of inverse model, a direct inverse model is proposed. To identify unknown parameters of a direct inverse rate-dependent model, an inverse model of the rate-independent Generalized Prandtl-Ishlinskii (GPI) is used. The inverse GPI model can provide a fairly accurate output to diminish the hysteresis effect at a single-frequency hysteresis. The genetic algorithm combined with an interior-point method as a gradient-based optimization, is proposed to estimate the parameters of the proposed controller quickly. Simulation and experimental results show that the proposed feedforward controller that is identified along this new approach is able to compensate for the hysteresis in the PAM-driven catheter successfully. The identification procedure to optimize the parameters takes less than five minutes and is thus very convenient in use. It should be noted that while demonstrated on a single PAM, the approach is believed to be general and hence also applicable to other actuators that possess the complex hysteresis behavior.

This work reports on the synthesis of a parallel Remote Center of Motion (RCM) mechanism, and an optimized design for the use-case of robot-assisted vitreoretinal surgery. A 2-DoF planar RCM mechanism is proposed and synthesised as part of a 4-DoF RCM mechanism. The proposed design substantially reduces the occupied volume at the end-effector. This solves a major problem present in related state-of-the-art, which poses limitations on sterile end-effector design and surgical instrument compatibility. Subsequently, an optimal design algorithm is proposed and implemented for the given use-case. The workspace is determined within mechanism constraints, after which performance parameters such as workspace coverage, potential energy, manipulability, reflected stiffness are determined for specific areas of interest within a desired workspace. Subsequently, these parameters are combined in a score function identifying an optimal kinematic design. When compared with the closely-related prior art, the resulting design shows improvements in relevant workspace coverage, reduced gravity compensation effort, and more isotropic manipulability. Overall reflected stiffness is reduced and should be taken into account in future design phases. Future work includes the integration of the kinematic design into a detailed conceptual design and a first prototype development. ispartof: Mechanism And Machine Theory vol:151 status: Published online

Twin-to-twin transfusion syndrome treatment requires fetoscopic laser photocoagulation of placental vascular anastomoses to regulate blood flow to both fetuses. Limited field-of-view (FoV) and low visual quality during fetoscopy make it challenging to identify all vascular connections. Mosaicking can align multiple overlapping images to generate an image with increased FoV, however, existing techniques apply poorly to fetoscopy due to the low visual quality, texture paucity, and hence fail in longer sequences due to the drift accumulated over time. Deep learning techniques can facilitate in overcoming these challenges. Therefore, we present a new generalized Deep Sequential Mosaicking (DSM) framework for fetoscopic videos captured from different settings such as simulation, phantom, and real environments. DSM extends an existing deep image-based homography model to sequential data by proposing controlled data augmentation and outlier rejection methods. Unlike existing methods, DSM can handle visual variations due to specular highlights and reflection across adjacent frames, hence reducing the accumulated drift. We perform experimental validation and comparison using 5 diverse fetoscopic videos to demonstrate the robustness of our framework., Comment: Accepted at MICCAI 2019

In collaborative human-robot navigation such as when driving semi-autonomous robotic wheelchairs, intuitive control of the mobile robot is only possible if the robot understands its user. This becomes especially important as users present varying levels of abilities and heterogeneous driving styles. Furthermore, the robot needs to consider the inherent uncertainty on its navigation task because the user may not be able to communicate his or her plans explicitly. In order to address these requirements, we have adopted a probabilistic framework to recognise navigation plans. A key component in this framework is a personalised driver model, which captures how a particular user transforms his or her mental navigation plan into inputs to the robot. In this work, we evaluate the use of Gaussian Processes to implement and calibrate this probabilistic, user-specific driver model, and this for use with haptic joysticks. Furthermore, special care was taken to obtain fast online evaluation of this user model through sparse approximation and parallel computation on a GPU. This resulted in an achievable user model evaluation frequency of 40 Hz, which is far above the navigation assistance frequency we aimed for, i.e. 5 Hz. We illustrate the validity of the approach by recognising the navigation plans of a spastic wheelchair user.

International audience; Purpose Minimally invasive surgery is becoming the standard treatment of care for a variety of procedures. Surgeons need to display a high level of proficiency to overcome the challenges imposed by the minimal access. Especially when operating on a dynamic organ, it becomes very difficult to align instruments reliably and precisely. In this paper, a hybrid ro-botic system and a dedicated robotic control approach are proposed to assist the surgeon performing complex surgical gestures in a dynamic environment. Methods The proposed hybrid robotic system consists of a rigid robot arm on top of which a continuum robot is mounted in series. The continuum robot is locally actuated with McKibben muscles. A control scheme is adopted based on quadratic programming framework. It is shown that this framework allows enforcing a set of constraints on the pose of the tip, as well as of the instrument shaft, which is commanded to slide in and out through the entry point. Results Through simulation and experiments it is shown how the robot tool-tip is able to follow sinus-oidal trajectories of 0.37 Hz and 2 Hz, corresponding to motion due to breathing and heartbeat respectively, while maintaining the instrument shaft pivoting nicely about the entry point. The positioning and tracking accuracy of such system is shown to lie below 3mm in position and 5 • in angle. Herbert De Praetere is with UZ Leuven, Cardiac surgery, Conclusion The results suggest a good potential for applying the proposed technology to assist the surgeon during complex robot-assisted interventions. It is also illustrated that even when using flexible hence relatively safe end-effectors, it is possible to reach acceptable tracking behaviour at relatively high frequencies.

In robotic Minimally Invasive Surgery (MIS) the incision point acts as a fulcrum around which the surgical instrument pivots. The fulcrum point has been the topic of much research. Mechanisms have been invented to enforce instrument motion about such a fulcrum. Other systems establish a fulcrum through coordinated control of their joints. For laparoscopists, the fulcrum point is an obstacle to overcome through a lot of training. For robots, it is a hurdle requiring careful consideration. In this paper, new estimation methods are proposed to exploit the properties of a fulcrum and turn its presence into an advantage. The paper starts by presenting a novel fulcrum estimation method that is robust against measurement noise and quantization effects. A general fulcrum refinement method is proposed next. This method can be used as add-on to ameliorate alternative estimation approaches. It is shown how the fulcrum can also be leveraged to get accurate, high-bandwidth estimates of the instrument tip. The quality with which the instrument tip is estimated has a large impact on the performance of advanced guidance schemes, such as haptic virtual walls. User tests are included, demonstrating substantially improved guidance thanks to the algorithms presented in this work. ispartof: IEEE Robotics and Automation Letters vol:3 issue:3 pages:2071-2078 status: published

Surgical interventions are increasingly executed minimal invasively. Surgeons insert instruments through tiny incisions in the body and pivot slender instruments to treat organs or tissue below the surface. While a blessing for patients, surgeons need to pay extra attention to overcome the fulcrum effect, reduced haptic feedback and deal with lost hand-eye coordination. The mental load makes it difficult to pay sufficient attention to the forces that are exerted on the body wall. In delicate procedures such as fetal surgery, this might be problematic as irreparable damage could cause premature delivery. As a first attempt to quantify the interaction forces applied on the patient's body wall, a novel 6 degrees of freedom force sensor was developed for an ex-vivo set up. The performance of the sensor was characterised. User experiments were conducted by 3 clinicians on a set up simulating a fetal surgical intervention. During these simulated interventions, the interaction forces were recorded and analysed when a normal instrument was employed. These results were compared with a session where a flexible instrument under haptic guidance was used. The conducted experiments resulted in interesting insights in the interaction forces and stresses that develop during such difficult surgical intervention. The results also implicated that haptic guidance schemes and the use of flexible instruments rather than rigid ones could have a significant impact on the stresses that occur at the body wall.

Real-time tool segmentation from endoscopic videos is an essential part of many computer-assisted robotic surgical systems and of critical importance in robotic surgical data science. We propose two novel deep learning architectures for automatic segmentation of non-rigid surgical instruments. Both methods take advantage of automated deep-learning-based multi-scale feature extraction while trying to maintain an accurate segmentation quality at all resolutions. The two proposed methods encode the multi-scale constraint inside the network architecture. The first proposed architecture enforces it by cascaded aggregation of predictions and the second proposed network does it by means of a holistically-nested architecture where the loss at each scale is taken into account for the optimization process. As the proposed methods are for real-time semantic labeling, both present a reduced number of parameters. We propose the use of parametric rectified linear units for semantic labeling in these small architectures to increase the regularization ability of the design and maintain the segmentation accuracy without overfitting the training sets. We compare the proposed architectures against state-of-the-art fully convolutional networks. We validate our methods using existing benchmark datasets, including ex vivo cases with phantom tissue and different robotic surgical instruments present in the scene. Our results show a statistically significant improved Dice Similarity Coefficient over previous instrument segmentation methods. We analyze our design choices and discuss the key drivers for improving accuracy., Paper accepted at IROS 2017

Fiber optic shape sensing is gaining popularity within areas such as medical catheterization where catheters and guidewires are used to navigate through tortuous vascular paths. Shape sensing can aid medical interventionalists by reducing exposure to damaging radiation and providing a more detailed real-time understanding of the 3-dimensional shape of the catheter/guidewire. However, despite the technology existing for several years, there is still room for improvement and steps to follow to reach the accuracy and robustness needed for these safety-critical applications. This letter discusses and provides methods for fiber integration within catheters to improve shape estimation accuracy and repeatability. A two-step calibration process is introduced for intrinsic twist compensation, which results in significant improvements in estimation accuracy. Additionally, a practical method for fiber parameter identification is introduced. The importance of estimating these parameters was found to be paramount for reaching adequate shape estimation. Further improvements to the reconstruction algorithm are proposed. Experimental validations with ground truth shapes are performed to assess the overall accuracy for static and dynamic configurations. For complex geometrical shapes and a fiber length of 170 mm, experiments show a mean spatial error of 0.70 mm (0.41%), a maximum of 2.52 mm (1.48%), and repeatability of ± 0.82 mm.

The determination of an optimal pivot point ( $${ OPP}$$ ) is important for instrument manipulation in minimally invasive surgery. Such knowledge is of particular importance for robotic-assisted surgery where robots need to rotate precisely around a specific point in space in order to minimize trauma to the body wall while maintaining position control. Remote center of motion (RCM) mechanisms are commonly used, where the RCM point is manually and visually aligned. If not positioned appropriately, this misalignment might lead to intolerably high forces on the body wall with increased risk of postoperative complications or instrument damage. An automated method to align the RCM with the $${ OPP}$$ was developed and tested. Computer vision and a lightweight calibration procedure are used to estimate the optimal pivot point. One or two pre-calibrated cameras viewing the surgical scene are employed. The surgeon is asked to make short pivoting movements, applying as little torque as possible, with an instrument of choice passing through the insertion point while camera images are being recorded. The physical properties of an instrument rotating around a pivot point are exploited in a random sample consensus scheme to robustly estimate the ideal position of the RCM in the image planes. Triangulation is used to estimate the RCM position in 3D. Experiments were performed on a specially designed mockup to test the method. The position of the pivot point is estimated with an average error less than 1.85 mm using two webcams placed from approximately 30 cm to 1 m away from the scene. The entire procedure was completed in a few seconds. An automated method to estimate the ideal position of the RCM was shown to be reliable. The method can be implemented within a visual servoing approach to automatically place the RCM point, or the results can be displayed on a screen to provide guidance to the surgeon. Further work includes the development of an image-guided alignment method and validation with in vivo experiments.

Computer vision and robotics are being increasingly applied in medical interventions. Especially in interventions where extreme precision is required they could make a difference. One such application is robot-assisted retinal microsurgery. In recent works, such interventions are conducted under a stereo-microscope, and with a robot-controlled surgical tool. The complementarity of computer vision and robotics has however not yet been fully exploited. In order to improve the robot control we are interested in 3D reconstruction of the anatomy and in automatic tool localization using a stereo microscope. In this paper, we solve this problem for the first time using a single pipeline, starting from uncalibrated cameras to reach metric 3D reconstruction and registration, in retinal microsurgery. The key ingredients of our method are: (a) surgical tool landmark detection, and (b) 3D reconstruction with the stereo microscope, using the detected landmarks. To address the former, we propose a novel deep learning method that detects and recognizes keypoints in high definition images at higher than real-time speed. We use the detected 2D keypoints along with their corresponding 3D coordinates obtained from the robot sensors to calibrate the stereo microscope using an affine projection model. We design an online 3D reconstruction pipeline that makes use of smoothness constraints and performs robot-to-camera registration. The entire pipeline is extensively validated on open-sky porcine eye sequences. Quantitative and qualitative results are presented for all steps., Comment: Accepted in Robotics and Automation Letters (RA-L). Project page: http://www.vision.ee.ethz.ch/~kmaninis/keypoints2stereo/index.html

Real-time tool segmentation is an essential component in computer-assisted surgical systems. We propose a novel real-time automatic method based on Fully Convolutional Networks (FCN) and optical flow tracking. Our method exploits the ability of deep neural networks to produce accurate segmentations of highly deformable parts along with the high speed of optical flow. Furthermore, the pre-trained FCN can be fine-tuned on a small amount of medical images without the need to hand-craft features. We validated our method using existing and new benchmark datasets, covering both ex vivo and in vivo real clinical cases where different surgical instruments are employed. Two versions of the method are presented, non-real-time and real-time. The former, using only deep learning, achieves a balanced accuracy of 89.6% on a real clinical dataset, outperforming the (non-real-time) state of the art by 3.8% points. The latter, a combination of deep learning with optical flow tracking, yields an average balanced accuracy of 78.2% across all the validated datasets.

Accurate and precise control of catheters inside a vasculature is a difficult yet important task. Current manual approaches require significant surgical skill. Over the years, surgeons build up a sort of mental kinematic map telling them how to handle the catheter in order to steer the catheter tip safely through the vessel system. The input-output behaviour of the catheter is complex and depends heavily on its configuration within and contacts with the vasculature. This paper introduces an alternative approach to control robotic catheters. The input-output behaviour or so-called differential kinematics are derived from a patient-specific vasculature model, following a minimum-energy argumentation. The validity of the proposed approach is demonstrated experimentally. Whereas the performance of model-based approaches is obviously greatly influenced by the correctness of estimated parameters, within this work we show experimentally how reasonable performance can already be achieved within the setup that was constructed. We expect that there is still ample room for improvement by, e.g., putting more sophisticated identification, modeling and collision detection schemes into place.

Advances in technology and computing play an increasingly important role in the evolution of modern surgical techniques and paradigms. This article reviews the current role of machine learning (ML) techniques in the context of surgery with a focus on surgical robotics (SR). Also, we provide a perspective on the future possibilities for enhancing the effectiveness of procedures by integrating ML in the operating room. The review is focused on ML techniques directly applied to surgery, surgical robotics, surgical training and assessment. The widespread use of ML methods in diagnosis and medical image computing is beyond the scope of the review. Searches were performed on PubMed and IEEE Explore using combinations of keywords: ML, surgery, robotics, surgical and medical robotics, skill learning, skill analysis and learning to perceive. Studies making use of ML methods in the context of surgery are increasingly being reported. In particular, there is an increasing interest in using ML for developing tools to understand and model surgical skill and competence or to extract surgical workflow. Many researchers begin to integrate this understanding into the control of recent surgical robots and devices. ML is an expanding field. It is popular as it allows efficient processing of vast amounts of data for interpreting and real-time decision making. Already widely used in imaging and diagnosis, it is believed that ML will also play an important role in surgery and interventional treatments. In particular, ML could become a game changer into the conception of cognitive surgical robots. Such robots endowed with cognitive skills would assist the surgical team also on a cognitive level, such as possibly lowering the mental load of the team. For example, ML could help extracting surgical skill, learned through demonstration by human experts, and could transfer this to robotic skills. Such intelligent surgical assistance would significantly surpass the state of the art in surgical robotics. Current devices possess no intelligence whatsoever and are merely advanced and expensive instruments.

This paper introduces a conceptual design of a hybrid mechatronic system consisting of a rigid robotic arm and a flexible link. The hybrid system represents an intermediate step towards a new type of instruments to be employed in the field of robotic surgery. These instruments are characterized by high distal structural flexibility allowing for safer interaction with the tissue. This paper shows that despite of the flexibility it is possible to accurately control the interaction forces between the robot and the soft and delicate tissue. To this end, a force control method is suggested, featuring quasi-static system model that is resolved in real-time inside the control law. The paper describes the overall system architecture, explains the basic concept behind the force control law and includes experimental validation of the force control of the hybrid mechatronic system. Conclusions are drawn and some indications for further work are formulated.

In current practice, fluoroscopy remains the gold standard for guiding surgeons during endovascular catheterization. The poor visibility of anatomical structures and the absence of depth information make accurate catheter localization and manipulation a difficult task. Overexposure to radiation and use of risk-prone contrast agent also compromise surgeons’ and patients’ health. Alternative approaches using embedded electromagnetic (EM) sensors have been developed to overcome the limitations of fluoroscopy-based interventions. As only a finite number of sensors can be integrated within a catheter, methods that rely on such sensors require the use of interpolation schemes to recover the catheter shape. Since EM sensors are sensitive to external interferences, the outcome is not robust. This paper introduces a probabilistic framework that improves the catheter localization and reduces the dependency on fluoroscopy and contrast agents. Within this framework, the dense 2D information extracted from fluoroscopic images is combined with the discrete pose information of EM sensors to provide a reliable reconstruction of the full three-dimensional catheter shape. Validation in a physics-based simulation environment and in a real-world experimental setup provides promising results and indicates that the proposed framework allows reconstructing the 3D catheter shape with a median root-mean-square error of 3.7[Formula: see text]mm with an interquartile range of 0.3[Formula: see text]mm.

Virtual Reality technology takes in an increasingly prominent role in modern medical training programs. The fidelity of the Virtual Reality environment is of crucial importance and directly relates to the quality of skill transfer. To train interventions that rely on physical interaction with the patient, high-quality haptic feedback should be provided. Palpation is a prototypical example of an important medical skill relying heavily on physical interaction. Since the interaction is quite involved and thus complex to render, very few training systems have attempted to train palpation so far and none of those provide a truly natural experience. In this work a novel haptic interface for free-hand palpation is presented. The interface follows the principle of an encountered-type robot replicating more naturally the actual palpation procedure. The concept and design of this new development are introduced. A control method to render spatially distributed kinaesthetic information is explained next. Experimental results give a proof-of-principle indicating that this approach might lead towards a more natural virtual palpation experience.

This paper describes a novel haptic guidance scheme that helps powered wheelchair users steer their wheelchair through narrow and complex environments. The proposed scheme encodes the local environment of the wheelchair as a set of collision-free circular paths. An adaptive impedance controller is constructed upon these circular paths. The controller increases resistance when nearing obstacles and simultaneously helps the user to change motion towards a safer circular path. To test the algorithm, a commercial powered wheelchair was interfaced and equipped with necessary sensors and an in-house built haptic joystick. The user was asked to drive backwards into a narrow elevator with and without navigation assistance. Although there is still room for improvements, the first results are promising. Thanks to the assistance the user can perform this maneuver successfully in most of the cases without even looking backwards.

In robotics and in many other fields, time synchronization between a host computer and sensors or other computers is essential for many reasons. Whereas past approaches have focused on phase and frequency estimation, little discussion seems to be available in literature on adopting uncertainty on these estimates to determine whether synchronization is out of bounds and to actively decide when to synchronize again to reduce uncertainty. This paper presents an algorithm to perform such active synchronization based on a Kalman filtering approach. The performance of the algorithm is illustrated by synchronizing a Hokuyo URG laserscanner with a computer. ispartof: pages:501-506 ispartof: The 15th International Conference on Advanced Robotics pages:501-506 ispartof: ICAR location:Tallinn, Estonia date:20 Jun - 23 Jun 2011 status: published

This paper presents a mechatronic analysis of a teleoperation system that provides haptic feedback. The analysis, based on the Bounded Environment Passivity method, describes the combined effect of the teleoperation controller and the master and slave hardware, including the effect of structural flexibilities, which have often been neglected in the domain of teleoperation. This results in rules of thumb for the design of the hardware and the tuning of the controller. As two of these rules of thumb pose a trade-off in hardware design, an unprecedented conceptual design optimization is described in a second part of this paper. The objective is to demonstrate the need for a mechatronic approach to the design of teleoperation systems in order to obtain maximal system performance. ispartof: pages:161-168 ispartof: EuroHaptics 2010 vol:6191 issue:PART 1 pages:161-168 ispartof: EuroHaptics location:Amsterdam date:8 Jul - 10 Jul 2010 status: published

The main goal of this chapter is to demonstrate the potential benefits of controllers of the third concept, i.e. controllers with model-based haptic feedback, especially for telesurgical applications. Hereto, this chapter describes the practical implementation of the Stiffness Reflecting Controller. The experiments described in Section 5 support the claim that such controllers show good robustness properties. It is shown that, for the SRC, the compliance of the position controller does not influence the stiffness felt by the operator. It is also shown that the introduction of a low-pass filter or non-negligible time-delay only minimally affects the transparency and stability for the SRC. Although not explicitly demonstrated in this chapter, controllers of the third concept can also behave more robust with respect to other hardware-related issues of surgical slave robots that traditionally restrict the applicability of bilateral controllers on such robots. Willaert et al. (2009b) show e.g. that the inertia of the slave has a large influence on the stability properties of the DFF controller and conclude that the slave inertia should be as low as possible. Since current commercial surgical robots are mostly not lightweight robots, the SRC can be a useful controller for these robots. Another, hardware aspect of current surgical robots is the restricted structural stiffness, which influences both transparency and stability (Christiansson & van der Helm, 2007; Tavakoli & Howe, 2009). For the DFF and a slave with flexibilities, the stiffness that the human operator feels is a series connection of the real environment stiffness, the stiffness of the position controller and the structural stiffness of the slave. In this work, the estimation of the environment stiffness is based on the force measurement at the end-effector (Fe ) and the position measurement at the motor (xs xe ). As the 1 d.o.f master and slave behave as a rigid-body for frequencies below 100 Hz, the correct environment stiffness can be estimated. For flexible multi-d.o.f. systems, however, the estimation of k e should be based on the force measurement at the end-effector (Fe ) and the position measurement at the end-effector (xe = xs ). In future research, it will be investigated how the position of the end-effector can be measured or estimated. By doing so, the SRC can be made insensitive to both the compliance of the position controller and the compliance of the slave robot itself. Based on the detailed stability analysis of the DFF and the experiments presented here, it is clear that, compared to the DFF, the SRC will have significantly better stability properties when implemented on multi-d.o.f master-slave setups. But, a detailed analysis of the stability properties of the SRC is not straightforward due to the presence of the Extended Kalman Filter. Thanks to the introduction of a stiffness-depending damping term (see 12), the SRC implemented on the setup described in this chapter is stable for environment stiffnesses up to

In this paper, the Bounded Environment Passivity method [1] is applied to a 3-channel controller. This method enables the design of teleoperation controllers that show passive behaviour for interactions with a bounded range of environments. The resulting tuning guidelines, derived analytically, provide interesting tuning flexibility, which allows to focus on different aspects of transparency. As telesurgery is the motivation behind this work, the focus lies on correctly reflecting the stiffness properties of the environment. A comparison between the transparency and stability properties of this 3- channel controller and the same properties of the Position- Force controller demonstrates the interesting properties of the 3-channel controller. The theoretical results are verified experimentally on a 1 d.o.f. master-slave setup. ispartof: pages:66-72 ispartof: Proceedings of the Third International Conferences on Advances in Computer-Human Interactions pages:66-72 ispartof: ACHI location:St. Maarten date:10 Feb - 16 Feb 2010 status: published

At present, surgical master-slave systems lack any kind of force feedback. Typically, controllers giving good stiffness transparency for soft environments cannot guarantee stability during hard contact. This paper presents a pragmatic method to avoid instability of a asterslave system during hard contacts, which does not affect the stiffness reflection for soft environments. The time derivative of the interaction force with the environment is used to detect a hard contact. Upon detection of a hard contact the force feedback is switched off and a virtual wall is activated at the master side in order to guarantee the perception of hard contact by the operator. The experiments demonstrate good stiffness transparency for soft environments, while the system remains stable for both soft and hard environments. ispartof: pages:73-82 ispartof: Haptics: Perception, Devices and Scenarios (EuroHaptics 2008) vol:5024 pages:73-82 uri: http://www.disam.upm.es/~eurohaptics2008/ ispartof: 6th International Conference on Haptics: Perception, devices and scenarios location:Madrid, Spain date:11 Jun - 13 Jun 2008 status: published

Catheters are increasingly being used to tackle problems in the cardiovascular system. However, positioning precision of the catheter tip is negatively affected by hysteresis. To ensure tissue damage due to imprecise positioning is avoided, hysteresis is to be understood and compensated for. This work investigates the feasibility to model hysteresis with a Long Short-Term Memory (LSTM) network. A bench-top setup containing a catheter distal segment was developed for model evaluation. The LSTM was first tested using four groups of test datasets containing diverse patterns. To compare with the LSTM, a Deadband Rate-Dependent Prandtl-Ishlinskii (DRDPI) model and a Support Vector Regression (SVR) model were established. The results demonstrated that the LSTM is capable of predicting the tip bending angle with sub-degree precision. The LSTM outperformed the DRDPI model and the SVR model by 60.1% and 36.0%, respectively, in arbitrarily varying signals. Next, the LSTM was further validated in a 3D reconstruction experiment using Forward-Looking Optical Coherence Tomography (FL-OCT). The results revealed that the LSTM was able to accurately reconstruct the environment with a reconstruction error below 0.25 mm. Overall, the proposed LSTM enabled precise free-space control of a robotic catheter in the presence of severe hysteresis. The LSTM predicted the catheter tip response precisely based on proximal input pressure, minimizing the need to install sensors at the catheter tip for localization. ispartof: IEEE Robotics and Automation Letters vol:6 issue:2 pages:2106-2113 status: Published online