Search

Medical image segmentation and registration are two tasks to analyze the anatomical structures in clinical research. Still, deep-learning solutions utilizing the connections between segmentation and registration remain underdiscovered. This article designs a joint learning framework named RSegNet that can realize concurrent deformable registration and segmentation by minimizing an integrated loss function, including three parts: diffeomorphic registration loss, segmentation similarity loss, and dual-consistency supervision loss. The probabilistic diffeomorphic registration branch could benefit from the auxiliary segmentations available from the segmentation branch to achieve anatomical consistency and better deformation regularity by dual-consistency supervision. Simultaneously, the segmentation performance could also be improved by data augmentation based on the registration with well-behaved diffeomorphic guarantees. Experiments on the human brain 3-D magnetic resonance images have been implemented to demonstrate the effectiveness of our approach. We trained and validated RSegNet with 1000 images and tested its performances on four public datasets, which shows that our method successfully yields concurrent improvements of both segmentation and registration compared with separately trained networks. Specifically, our method can increase the accuracy of segmentation and registration by 7.0% and 1.4%, respectively, in terms of Dice scores. Note to Practitioners—Registration and segmentation of medical images are two significant tasks in medical research and clinical application. However, most existing approaches consider these two tasks independently while neglecting the potential association between them. Therefore, we suggest a new approach that combines these two tasks into one joint deep learning framework, boosting registration, and segmentation performance by introducing dual-consistency supervision. Besides, our framework could generate outputs within 1 s by taking an affinely aligned medical image pair as input, which is suitable for time-critical requirements in a clinic. We tested it on four public datasets and achieved state-of-the-art performance to demonstrate the proposed method’s feasibility and robustness. Furthermore, our proposed RSegNet is a general learning framework suitable for various image modalities and anatomical structures. Hence, we expect our framework to serve as a practical clinical tool to speed up medical image analysis procedures and improve diagnostic accuracy. [ABSTRACT FROM AUTHOR]

Performing a successful robotic grasping to uncertain objects in unstructured environments is challenging. This study presents a new compliant soft robotic gripper for objects handling and cap manipulation through the coordination of three soft fingers and in-hand manipulation. The experiments are conducted to validate that the soft robotic gripper can successfully realize simultaneous grasping and capping manipulations with only one flexible shaft actuation for every single soft finger. Note to Practitioners—Uncertain object manipulation tasks pose significant challenges to a robotic gripper while grasping and capping unknown objects without damaging them. The existing rigid grippers have experienced flexible manipulation through multiple degrees of freedom (DoFs) by complex mechanical structures, and the soft gripper can realize stiffness-compliant manipulation differently. The proposed novel robotic in-hand manipulation can execute grasping and cap manipulation by a single flexible shaft to simultaneously achieve bending and rotational movements. The relationship between stretching force and finger’s curvature can enable a custom design for user-specific applications. [ABSTRACT FROM AUTHOR]

Registration is an essential problem in image-guided surgery (IGS) since it brings different involved coordinate frames together. Nonrigid or deformable registration still faces many challenges, such as two point sets (PSs) are partially overlapped. To tackle the challenges in the nonrigid registration, we introduce a new two-step point-based registration pipeline that includes two steps. In the first step, the rigid transformation between the two spaces is recovered where the orientation vectors are adopted. In the second step, built on the nonrigid coherent point drift (CPD) approach, the anisotropic positional noise is also assumed. Registration results on the human liver verify the proposed approach’ great improvements over the other methods. First, the rotation and translation are recovered with smaller error values than the existing methods. Second, our registration method’s performance is much more robust to the partial overlapping between two PSs. Third, the two-step registration framework achieves the best performances in most test cases when there is a localization error in acquiring the intraoperative data. Note to Practitioners—A novel registration approach is presented for image-guided liver surgery (LGLS). Compared with existing nonrigid registration methods, two significant changes (or improvements) exist in the proposed registration framework: 1) the normal vectors are extracted and utilized in the rigid registration step and 2) the anisotropic positional uncertainties are considered. In both steps, the registration problems are formulated as a maximum likelihood (ML) problems and dealt with the expectation-maximization (EM) technique. In both steps, the matrix form of the updated positional covariance is provided and can speed up the computational process. The readers are reminded that with extra information and a more general positional error assumption, our approach demonstrates improved performances in the case of partial-to-full alignment. [ABSTRACT FROM AUTHOR]

Accurately detecting tumors and estimating the depth of tumors is essential in the surgical removal of tumors. In robotic-assisted surgery, autonomous robotic palpation has the potential to provide more precise detection, tumors’ depth estimation, and less intrusion when normal tissues surround tumors. In this article, by mimicking the human finger touch, we propose a tactile sensing-based deep recurrent neural network (DRNN) with long short-term memory (LSTM) architecture to improve the accuracy of the detection and depth estimation of tumors embedded in soft tissue. In the experimental setup, the hard inclusions simulate the tumors, while the phantom tissue is fabricated by silicon to simulate the soft tissue. During the experiment, the data from the force sensor and displacement of the robot palpation probe are for detection and depth estimation purposes. The collected sequential data set of the force and the displacement of the probe during one completed palpation process will go through the proposed DRNN network with deep LSTM architecture, in which the temporal dependencies of the sequential data will be captured in the cell states in the deep LSTM layers. Subsequently, the softmax classifier is adopted to determine if there is any hard inclusion exists and offer the depth estimation of the hard inclusions. Experiments based on 396 real data sets demonstrate that the detection accuracy for the testing data set is 99.2% and the depth estimation accuracy for the testing data set is 95.8%. The accuracy of the proposed method is best when comparing with other widely used methods. Note to Practitioners—The palpation of tumors motivated this article in the robot-assisted surgical systems through tactile feedback. In order to mimic the human touch on the soft tissue, this article presents a deep-learning-based approach to estimate the depth of the hard inclusions in the phantom tissue through force information. The displacement of the palpation probe and the touch force during one palpation are recorded as data sequences to train the deep model, which aims to capture dynamics and long-term dependence of the palpation process. In this article, we made the first successful attempt to accurately estimate the depth of the hard inclusions buried at different locations of the phantom tissue using only force information. The proposed approach can work in different robot-assisted scenarios, such as master–slave robotic surgery. In the clinic applications, the force sensor will be integrated at the end-effector of the robotic manipulator. According to the specific requirements, the force sensor and the robotic manipulator might be different from those used in this article. For some applications, such as the laparoscopic interventions, the complete vertical contact tends to be difficult to obtain due to the laparoscopic port effects. The projection of the recorded force data and displacement can obtain the information in the normal direction. The future work is going to be extended to tissue environments with arbitrary surface and tumors with various shapes/depths for more complex and prospective clinical applications. [ABSTRACT FROM AUTHOR]

In a paired-point rigid registration, target registration error (TRE) is deemed to be the most important quality metric. TRE usually cannot be directly measured, and thus many TRE estimation algorithms have been proposed. However, target localization errors (TLEs) in two spaces are not considered in the definition of TRE. In this paper, we propose a new type of evaluation metric that is referred to as total TRE (TTRE) at a given target point with TLE incorporated. Statistics including mean, root mean square (rms), and covariance matrix of TTRE are derived without making any assumption of the TLE magnitude. TTRE and fiducial registration error (FRE) are proved to be uncorrelated when an ideal weighting scheme is adopted in solving the registration problem. The proposed error model is validated through extensive experiments. In the first experiment with random fiducials and targets, in 90% of the test cases, there shows no difference between the predicted and simulated TTRE statistics when six fiducials are used. In the second experiment of deep-brain stimulation surgery, the mean value of CC(TTRE,FRE) being $8.9246\times 10^{-4}\pm 0.0389 $ was observed, which indicates that TTRE and FRE are uncorrelated. In the third experiment of surgical tool-tip tracking, the mean and standard deviation of percentage differences between predicted and simulated TTRE rms values are $2.22\%\pm 0.77\%$ for the planar tool and $2.62\%\pm 0.59\%$ for the textral tool. In summary, our proposed algorithm can well model the TTRE metric. Note to Practitioners—This paper was motivated by considering the existence of target localization error in two spaces to be registered in a rigid point-based registration. Given fiducial localization error (FLE) and TLE probability distributions, the proposed TTRE model gives the covariance matrix and root mean square of TTRE at a given target point. The possible applications of the proposed TTRE model in an image-guided surgery (IGS) include: 1) estimation of surgical tool-tip tracking error distribution when pivot calibration uncertainty is considered; 2) accuracy evaluation of an image-to-patient registration; 3) updating of a preoperative surgical plan on a computed tomography/MRI image to consider the registration error at a given target point; and 4) estimation of real-time tool-tip error in touching a desired “true” surgical target, such as a tumor during surgery. To use the proposed error estimation algorithm, FLE and TLE in two spaces to be registered have to be known or estimated in advance to a registration process. The proposed error model is validated through extensive simulations. [ABSTRACT FROM AUTHOR]

This paper presents a sensor fusion-based surgical instrument tracking system which uses multiple monocular modules. The system is an optical tracking system, which has been widely utilized in the image-guide surgery because of its high accuracy and precision. However, the line-of-sight occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple monocular modules. The rationale is to enable the system to track the surgical instruments inside the surgical site from different views. Three sensor fusion algorithms are proposed to integrate all sensor data from the multimodule system. In order to show the feasibility of the tracking system, simulations and comparison experiments have been carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions. Moreover, an image-guided surgical trial by using a cadaver head has been carried out to validate the feasibility of the proposed system and the tracking algorithms. The results from both the simulation and the cadaver trial have shown the effectiveness of the proposed robust fusion algorithm. Note to Practitioners—Optical tracking system (OTS) for image-guided surgery has suffered from line-of-sight (LoS) occlusion and field-of-view (FoV) limitation problems for a long time; nevertheless, no satisfactory solution has been given yet. In the real-world practice, surgeons and assistants have to manually adjust the position and orientation of the optical tracker during operation. This interrupts the surgery process and degrades the efficiency of the surgical procedure. Instead of using one tracker for the image-guided surgery, our approach is to utilize distributed modular trackers to avoid LoS occlusion and maintain FoV of the OTS. Based on our previous work, this paper is to provide an optimal fusion algorithm that takes the advantage of the multiple modules and validate the system in both the simulation and the real-world surgical environment. [ABSTRACT FROM AUTHOR]

This paper introduces a systematical interval-based framework of inherent uncertainties and pose evaluation for a class of soft continuum robots driven by flexible shafts. A more general model of continuum robots driven by shaft tendons is extended from prior kinematic models. On top of the proposed model, the interval-based analysis is presented to analyze and characterize the pose of continuum robots considering uncertainties in kinematic parameters and joint inputs. A 3-D printed bending actuator driven by a flexible shaft is evaluated for case study based on the proposed interval-valued framework. This paper investigates and compares a couple of refinement methods and proposes a new way of sensitivity analysis of model parameters based on interval arithmetic. The kinematic and mechanics parameters are measured and identified experimentally with a representation of intervals. The in-plane motion experiment validates that the computed bounds can enclose all the measured tip positions with consideration of the measurement uncertainty. The method is also validated when external loading is exerted. Note to Practitioners—With the fast development of soft robotics, the increasing number of soft robotic manipulators show great potentials of application in industries such as agriculture, biomedicine, home automation, manufacturing, logistics, and domestic service. Thanks to the mechanical compliance, soft manipulators demonstrate environmental adaptability at the cost of very precise positioning. However, the guaranteed bounds of the reaching range are valuable in order to provide users a good knowledge of the product performance, which is the motivation of this paper. This paper provides not only a method and framework based on interval analysis to evaluate the pose of soft actuators, but also a case study of uncertainty interval characterization procedure which is domain-specific for soft actuators. The presented method framework can be implemented for continuum manipulators and soft actuators as well as any other robotic devices featured by such flexible components. [ABSTRACT FROM AUTHOR]

Image guidance during minimally invasive intravascular interventions is primarily achieved based on X-ray fluoroscopy, which has several limitations including limited 3-D imaging capability, significant doses of radiation to operators, and lack of contact force measurement between the cardiovascular tissue and interventional tools. Ultrasound imaging can be adopted to complement or possibly replace 2-D fluoroscopy for intravascular interventions due to its portability, safety to use, and the ability of providing depth information. However, it is challenging to precisely visualize catheters and guidewires in the ultrasound images. In this paper, we propose a novel method to figure out both the position and orientation of the catheter tip in 2-D ultrasound images in real time by detecting and tracking a passive marker attached to the catheter tip. Moreover, the contact force can be estimated simultaneously as well via measuring the length variation of the marker. A geometrical model-based method is introduced to detect the initial position of the marker, and a Kanade–Lucas–Tomasi-based algorithm is developed to track the position, orientation, and length of the marker. The ex vivo experiment results validate the effectiveness of the proposed approach in automatically locating the catheter tip in ultrasound images and its capability of sensing the contact force. Therefore, it can be concluded that the presented method can be utilized to better facilitate operators during intravascular interventions. Note to Practitioners—This paper is motivated by the vision-based force sensing mechanism. A coil spring marker is used to visualize and locate the catheter tip in ultrasound images. The metallic helical structure of the marker can result in remarkable features in 2-D in-plane ultrasound images. A geometrical model-based detection algorithm is proposed to detect the passive marker in initial frame regardless of its length and orientation variation. A prediction–detection-combined tracking algorithm is developed to automatically track the catheter tip for the subsequent frames in real time. Most importantly, the distal force can be simultaneously estimated due to the length variation of the passive marker. The sensed force can be fed back to a robotic catheterization system during intravascular interventions. [ABSTRACT FROM AUTHOR]

Data-defect would affect the data quality and the analysis results of data mining. This paper presents a data-defect inspection method with kernel-neighbor-density-change outlier factor (KNDCOF). The definition of kernel neighbor density is proposed to represent the density of each object in database, and the ascending distance series (ADS) of each object is calculated based on the kernel distance between the object and its neighbors. Then, the average density fluctuation (ADF) of the object is established according to the weighted sum of the square of density difference between the object and others in ADS. Finally, the KNDCOF of the object is equal to the ratios of the ADF of the object and the average ADF of neighbors of the object. The degree of the object being an outlier is indicated by the KNDCOF value. The experiments are performed on three real data sets to evaluate the effectiveness of the proposed method. The experimental results verify that the proposed method has higher quality of data-defect inspection and does not increase the time complexity.

Note to Practitioners–Data-defect inspection is an important procedure of data preprocessing for a real industrial process. This paper presents a data-defect inspection method with kernel-neighbor-density-change outlier factor to identify the outliers, and addresses the challenges associated with the strong correlation and the nonlinearity of the industrial data. The proposed method calculates the outlier factor for each object, which quantifies how outlying it is. The outlier factor is based on the density difference between the object and its neighbors. The larger the outlier factor of an object is, the higher the outlierness of the object is. The proposed method could be wildly used in an industrial complex data set with different density regions. In the industrial field, engineers can deal with the objects with high outlier factor values based on the actual requirements. [ABSTRACT FROM AUTHOR]

When a robot is required to perform specific tasks defined in the world frame, there is a need for finding the coordinate transformation between the kinematic base frame of the robot and the world frame. The kinematic base frame used by the robot controller to define and evaluate the kinematics may deviate from the mechanical base frame constructed based on structural features. Besides, by using kinematic modeling rules such as the product of exponentials (POE) formula, the base frame can be arbitrarily located, and does not have to be related to any feature of the mechanical structure. As a result, the kinematic base frame cannot be measured directly. This paper proposes to find the kinematic base frame by solving a hand-eye calibration problem using 3D position measurements only, which avoids the inconvenience and inaccuracy of measuring orientations and thus significantly facilitates practical operations. A closed-form solution and an iterative solution are explicitly formulated and proved effective by simulations. Comprehensive analyses of the impact of key parameters to the accuracy of the solution are also carried out, providing four guidelines to better conduct practical operations. Finally, experiments on a 7-DOF industrial robot are performed with an optical tracking system to demonstrate the superiority of the proposed method using position data only over the method using full pose data. [ABSTRACT FROM AUTHOR]

Wheeled mobile robots (WMRs) are often used for maintenance of round pipes or ducts, which can typically be represented as a cylindrical workspace. Working in round pipes or ducts, kinematic models of WMRs are different from those applying on a plane and thus pose significant challenges in terms of kinematic analysis and motion control. To address these challenges, the kinematic properties of WMRs in a cylindrical workspace are analyzed in this paper. First, we discuss the kinematic properties of a single wheel in a cylindrical workspace. Then, we analyze the geometric constraints of WMRs in round pipes or ducts with analytical geometry. Based on these analyses, kinematic properties of WMRs in cylindrical workspaces are discussed with screw theory. A control law based on biaxial clinometer information is proposed, and it enables the robot to move horizontally in round pipes or ducts. Finally, the motion of a single wheel purely rolling in a cylindrical workspace is simulated. Experiments using a car-like mobile robot moving in round ducts are carried out to show the feasibility of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Central catadioptric omnidirectional images exhibit serious nonlinear distortions due to the involved quadratic mirrors. Therefore, features based on the conventional pin-hole model are hard to achieve satisfactory performances when directly applied to the distorted omnidirectional images. This paper analyzes the catadioptric geometry to facilitate modeling the nonlinear distortions of omnidirectional images. Different to the conventional imaging model, the prior information is considered in catadioptric system. A parameterized neighborhood mapping model is proposed to efficiently calculate the neighborhood of an object based on its measurable radial distance in the image plane. On the basis of the parameterized nonlinear model, a distortion-invariant fragment-based joint-feature mixture model of Gaussian is presented for human target tracking in omnidirectional vision. Under the framework of Gaussian Mixture Model, the problem of feature matching is converted into the feature clustering. The joint probability distribution of a joint-feature class is modeled by a mixture of Gaussian. A weight contribution mechanism is designed to flexibly weight the fragments contribution based on their responses, which leads to a robust tracking even under serious partial occlusion. Finally, experiments validate the advantage of the proposed algorithm over other conventional approaches. [ABSTRACT FROM AUTHOR]

Optical means of instrument tracking has been widely used in image-guided interventions and considered the de facto standard for tracking rigid bodies with a direct line-of-sight. However, the occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple reconfigurable monocular modules. The main approach is to enable the system to dynamically reconfigure the multiple monocular modules when occlusion occurs partially within the workspace. In this paper, we focus on the system architecture and an agile multicamera calibration method which only uses the customized tool for the surgical instrument tracking scenario. Additionally, two fast non-iterative algorithms are proposed and studied. In order to show the feasibility and superiority of the corresponding multicamera calibration algorithm, comparison experiments have carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions. [ABSTRACT FROM PUBLISHER]

This paper proposes an algorithm for robotic kinematic calibration based on a minimal product of exponentials (POE)-based model for the applications where only position measurements are required. Both joint zero-offset errors and initial frame twist error can be involved in this model. Analysis of the identifiability of these errors shows that at most six elements of these parameters can be identified. It also suggests that at least three noncollinear points on the end-effector should be measured to maximize the identifiability. Compared with the traditional POE-based model with full pose (position and orientation) measurements, the minimal model with only position measurements outperforms in terms of convenience, efficiency, and accuracy. [ABSTRACT FROM PUBLISHER]

This paper presents a three dimensional ultrasound (3DUS)-based visual servoing technique for intraoperative tracking of the motion of both surgical instruments and tissue targets. In the proposed approach, visual servoing techniques are used to control the position of a virtual ultrasound probe so as to keep a target centered within the virtual probe's field-of-view. Multiple virtual probes can be servoed in parallel to provide simultaneous tracking of instruments and tissue. The technique is developed in the context of robotic beating-heart intracardiac surgery in which the goal of tracking is to both provide guidance to the operator as well as to provide the means to automate the surgical procedure. To deal with the low signal-to-noise ratio (SNR) of the 3DUS volumes, an intensity-based method is proposed that requires no primitive extraction or image segmentation since it directly utilizes the image intensity information as a visual feature. This approach is computationally efficient and can be applied to a wide range of tissue types and medical instruments. This paper presents the first validation of these techniques through offline robot and tissue tracking using actual in vivo cardiac volume sequences from a robotic beating-heart surgery. [ABSTRACT FROM PUBLISHER]

This paper presents a passive-marker-based optical tracking system utilizing dual Kinect sensors and additional custom optical tracking components. To obtain sub-millimeter tracking accuracy, we introduce robust calibration of dual infrared sensors and point correspondence establishment in a stereo configuration. The 3D localization is subsequently accomplished using multiple back projection lines. The proposed system extends existing inexpensive consumer electronic devices, implements tracking algorithms, and shows the feasibility of applying the proposed low-cost system to surgical training for computer assisted surgeries. [ABSTRACT FROM AUTHOR]