Search

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548

This article describes a novel system for quantitative and volumetric measurement of tissue elasticity in the prostate using simultaneous multi-frequency tissue excitation. Elasticity is computed by using a local frequency estimator to measure the three-dimensional local wavelengths of steady-state shear waves within the prostate gland. The shear wave is created using a mechanical voice coil shaker which transmits simultaneous multi-frequency vibrations transperineally. Radio frequency data is streamed directly from a BK Medical 8848 transrectal ultrasound transducer to an external computer where tissue displacement due to the excitation is measured using a speckle tracking algorithm. Bandpass sampling is used that eliminates the need for an ultra-fast frame rate to track the tissue motion and allows for accurate reconstruction at a sampling frequency that is below the Nyquist rate. A roll motor with computer control is used to rotate the transducer and obtain 3D data. Two commercially available phantoms were used to validate both the accuracy of the elasticity measurements as well as the functional feasibility of using the system for in vivo prostate imaging. The phantom measurements were compared with 3D Magnetic Resonance Elastography (MRE), where a high correlation of 96% was achieved. In addition, the system has been used in two separate clinical studies as a method for cancer identification. Qualitative and quantitative results of 11 patients from these clinical studies are presented here. Furthermore, an AUC of 0.87±0.12 was achieved for malignant vs. benign classification using a binary support vector machine classifier trained with data from the latest clinical study with leave one patient out cross-validation.

Real-time ultrasound imaging plays an important role in ultrasound-guided interventions. The 3-D imaging provides more spatial information compared to conventional 2-D frames by considering the volumes of data. One of the main bottlenecks of 3-D imaging is the long data acquisition time, which reduces practicality and can introduce artifacts from unwanted patient or sonographer motion. This article introduces the first shear wave absolute vibro-elastography (S-WAVE) method with real-time volumetric acquisition using a matrix array transducer. In S-WAVE, an external vibration source generates mechanical vibrations inside the tissue. The tissue motion is then estimated and used in solving a wave equation inverse problem to provide the tissue elasticity. A matrix array transducer is used with a Verasonics ultrasound machine and a frame rate of 2000 volumes/s to acquire 100 radio frequency (RF) volumes in 0.05 s. Using plane wave (PW) and compounded diverging wave (CDW) imaging methods, we estimate axial, lateral, and elevational displacements over 3-D volumes. The curl of the displacements is used with local frequency estimation to estimate elasticity in the acquired volumes. Ultrafast acquisition extends substantially the possible S-WAVE excitation frequency range, now up to 800 Hz, enabling new tissue modeling and characterization. The method was validated on three homogeneous liver fibrosis phantoms and on four different inclusions within a heterogeneous phantom. The homogeneous phantom results show less than 8% (PW) and 5% (CDW) difference between the manufacturer values and the corresponding estimated values over a frequency range of 80–800 Hz. The estimated elasticity values for the heterogeneous phantom at 400-Hz excitation frequency show the average errors of 9% (PW) and 6% (CDW) compared to the provided average values by magnetic resonance elastography (MRE). Furthermore, both imaging methods were able to detect the inclusions within the elasticity volumes. An ex vivo study on a bovine liver sample shows less than 11% (PW) and 9% (CDW) difference between the estimated elasticity ranges by the proposed method and the elasticity ranges provided by MRE and acoustic radiation force impulse (ARFI).

Quantitative tissue stiffness characterization using ultrasound (US) has been shown to improve prostate cancer (PCa) detection in multiple studies. Shear wave absolute vibro-elastography (SWAVE) allows quantitative and volumetric assessment of tissue stiffness using external multifrequency excitation. This article presents a proof of concept of a first-of-a-kind 3-D hand-operated endorectal SWAVE system designed to be used during systematic prostate biopsy. The system is developed with a clinical US machine, requiring only an external exciter that can be mounted directly to the transducer. Subsector acquisition of radio frequency (RF) data allows imaging of shear waves with a high effective frame rate (up to 250 Hz). The system was characterized using eight different quality assurance phantoms. Due to the invasive nature of prostate imaging, at this early stage of development, validation of in vivo human tissue was instead carried out by intercostally scanning the livers of $n = 7$ healthy volunteers. The results are compared with 3-D magnetic resonance elastography (MRE) and an existing 3-D SWAVE system with a matrix array transducer (M-SWAVE). High correlations were found with MRE (99% in phantoms, 94% in liver data) and with M-SWAVE (99% in phantoms, 98% in liver data).

Purpose: Trans-oral robotic surgery (TORS) using the da Vinci surgical robot is a new minimally-invasive surgery method to treat oropharyngeal tumors, but it is a challenging operation. Augmented reality (AR) based on intra-operative ultrasound (US) has the potential to enhance the visualization of the anatomy and cancerous tumors to provide additional tools for decision-making in surgery. Methods: We propose a US-guided AR system for TORS, with the transducer placed on the neck for a transcervical view. Firstly, we perform a novel MRI-to-transcervical 3D US registration study, comprising (i) preoperative MRI to preoperative US registration, and (ii) preoperative to intraoperative US registration to account for tissue deformation due to retraction. Secondly, we develop a US-robot calibration method with an optical tracker and demonstrate its use in an AR system that displays anatomy models in the surgeon’s console in real-time. Results: Our AR system achieves a projection error from the US to the stereo cameras of 27.14 and 26.03 pixels (image is 540$\times$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document}960) in a water bath experiment. The average target registration error (TRE) for MRI to 3D US is 8.90 mm for the 3D US transducer and 5.85 mm for freehand 3D US, and the TRE for pre-intra operative US registration is 7.90 mm. Conclusion: We demonstrate the feasibility of each component of the first complete pipeline for MRI-US-robot-patient registration for a proof-of-concept transcervical US-guided AR system for TORS. Our results show that trans-cervical 3D US is a promising technique for TORS image guidance.

Purpose: Prostate imaging to guide biopsy remains unsatisfactory, with current solutions suffering from high complexity and poor accuracy and reliability. One novel entrant into this field is micro-ultrasound (microUS), which uses a high-frequency imaging probe to achieve very high spatial resolution, and achieves prostate cancer detection rates equivalent to multiparametric magnetic resonance imaging (mpMRI). However, the ExactVu transrectal microUS probe has a unique geometry that makes it challenging to acquire controlled, repeatable three-dimensional (3D) transrectal ultrasound (TRUS) volumes. We describe the design, fabrication, and validation of a 3D acquisition system that allows for the accurate use of the ExactVu microUS device for volumetric prostate imaging. Methods: The design uses a motorized, computer-controlled brachytherapy stepper to rotate the ExactVu transducer about its axis. We perform geometric validation using a phantom with known dimensions and compare performance with magnetic resonance imaging (MRI) using a commercial quality assurance anthropomorphic prostate phantom. Results: Our geometric validation shows accuracy of 1 mm or less in all three directions, and images of an anthropomorphic phantom qualitatively match those acquired using MRI and show good agreement quantitatively. Conclusion: We describe the first system to acquire robotically controlled 3D microUS images using the ExactVu microUS system. The reconstructed 3D microUS images are accurate, which will allow for future applications of the ExactVu microUS system in prostate specimen and in vivo imaging.

Purpose: To achieve effective robot-assisted laparoscopic prostatectomy, the integration of transrectal ultrasound (TRUS) imaging system which is the most widely used imaging modality in prostate imaging is essential. However, manual manipulation of the ultrasound transducer during the procedure will significantly interfere with the surgery. Therefore, we propose an image co-registration algorithm based on a photoacoustic marker (PM) method, where the ultrasound/photoacoustic (US/PA) images can be registered to the endoscopic camera images to ultimately enable the TRUS transducer to automatically track the surgical instrument. Methods: An optimization-based algorithm is proposed to co-register the images from the two different imaging modalities. The principle of light propagation and an uncertainty in PM detection were assumed in this algorithm to improve the stability and accuracy of the algorithm. The algorithm is validated using the previously developed US/PA image-guided system with a da Vinci surgical robot. Results: The target-registration-error (TRE) is measured to evaluate the proposed algorithm. In both simulation and experimental demonstration, the proposed algorithm achieved a sub-centimeter accuracy which is acceptable in practical clinics (i.e., 1.15 ± 0.29 mm from the experimental evaluation). The result is also comparable with our previous approach (i.e., 1.05 ± 0.37 mm), and the proposed method can be implemented with a normal white light stereo camera and does not require highly accurate localization of the PM. Conclusion: The proposed frame registration algorithm enabled a simple yet efficient integration of commercial US/PA imaging system into laparoscopic surgical setting by leveraging the characteristic properties of acoustic wave propagation and laser excitation, contributing to automated US/PA image-guided surgical intervention applications.

Purpose: Pathology from trans-perineal template mapping biopsy (TTMB) can be used as labels to train prostate cancer classifiers. In this work, we propose a framework to register TTMB cores to advanced volumetric ultrasound data such as multi-parametric transrectal ultrasound (mpTRUS). Methods: The framework has mainly two steps. First, needle trajectories are calculated with respect to the needle guiding template—considering deflections in their paths. In standard TTMB, a sparsely sampled ultrasound volume is taken prior to the procedure which contains the template overlaid on top of it. The position of this template is detected automatically, and the cores are mapped following the calculated needle trajectories. Second, the TTMB volume is aligned to the mpTRUS volume by a two-step registration method. Using the same transformations from the registration step, the cores are registered from the TTMB volume to the mpTRUS volume. Results: TTMB and mpTRUS of 10 patients were available for this work. The target registration errors (TRE) of the volumes using landmarks picked by three research assistants (RA) and one radiation oncologist (RO) were on average 1.32 ± 0.7 mm and 1.03 ± 0.6 mm, respectively. Additionally, on average, our framework takes only 97 s to register the cores. Conclusion: Our proposed framework allows a quick way to find the spatial location of the cores with respect to volumetric ultrasound. Furthermore, knowing the correct location of the pathology will facilitate focal treatment and will aid in training imaging-based cancer classifiers.

We have trained generative adversarial networks (GANs) to mimic both the effect of temporal averaging and of singular value decomposition (SVD) denoising. This effectively removes noise and acquisition artifacts and improves signal-to-noise ratio (SNR) in both the radio-frequency (RF) data and in the corresponding photoacoustic reconstructions. The method allows a single frame acquisition instead of averaging multiple frames, reducing scan time and total laser dose significantly. We have tested this method on experimental data, and quantified the improvement over using either SVD denoising or frame averaging individually for both the RF data and the reconstructed images. We achieve a mean squared error (MSE) of 0.05%, structural similarity index measure (SSIM) of 0.78, and a feature similarity index measure (FSIM) of 0.85 compared to our ground-truth RF results. In the subsequent reconstructions using the denoised data we achieve a MSE of 0.05%, SSIM of 0.80, and a FSIM of 0.80 compared to our ground-truth reconstructions.

Purpose: In low-dose-rate prostate brachytherapy (LDR-PB), treatment planning is the process of determining the arrangement of implantable radioactive sources that radiates the prostate while sparing healthy surrounding tissues. Currently, these plans are prepared manually by experts incorporating the centre’s planning style and guidelines. In this article, we develop a novel framework that can learn a centre’s planning strategy and automatically reproduce rapid clinically acceptable plans. Methods: The proposed framework is based on conditional generative adversarial networks that learn our centre’s planning style using a pool of 931 historical LDR-PB planning data. Two additional losses that help constrain prohibited needle patterns and produce similar-looking plans are also proposed. Once trained, this model generates an initial distribution of needles which is passed to a planner. The planner then initializes the sources based on the predicted needles and uses a simulated annealing algorithm to optimize their locations further. Results: Quantitative analysis was carried out on 170 cases which showed the generated plans having similar dosimetry to that of the manual plans but with significantly lower planning durations. Indeed, on the test cases, the clinical target volumes achieving $100\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$100\%$$\end{document}of the prescribed dose for the generated plans was on average $98.98\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$98.98\%$$\end{document}($99.36\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$99.36\%$$\end{document}for manual plans) with an average planning time of $3.04\pm 1.1$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3.04\pm 1.1$$\end{document}min ($20\pm 10$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\pm 10$$\end{document}min for manual plans). Further qualitative analysis was conducted by an expert planner who accepted $90\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$90\%$$\end{document}of the plans with some changes ($60\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$60\%$$\end{document}requiring minor changes & $30\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$30\%$$\end{document}requiring major changes). Conclusion: The proposed framework demonstrated the ability to rapidly generate quality treatment plans that not only fulfil the dosimetric requirements but also takes into account the centre’s planning style. Adoption of such a framework would save significant amount of time and resources spent on every patient; boosting the overall operational efficiency of this treatment.

Purpose: This scoping review covers needle visualization and localization techniques in ultrasound, where localization-based approaches mostly aim to compute the needle shaft (and tip) location while potentially enhancing its visibility too. Methods: A literature review is conducted on the state-of-the-art techniques, which could be divided into five categories: (1) signal and image processing-based techniques to augment the needle, (2) modifications to the needle and insertion to help with needle-transducer alignment and visibility, (3) changes to ultrasound image formation, (4) motion-based analysis and (5) machine learning. Results: Advantages, limitations and challenges of representative examples in each of the categories are discussed. Evaluation techniques performed in ex vivo, phantom and in vivo studies are discussed and summarized. Conclusion: Greatest limitation of the majority of the literature is that they rely on original visibility of the needle in the static image. Need for additional/improved apparatus is the greatest limitation toward clinical utility in practice. Significance: Ultrasound-guided needle placement is performed in many clinical applications, including biopsies, treatment injections and anesthesia. Despite the wide range and long history of this technique, an ongoing challenge is needle visibility in ultrasound. A robust technique to enhance ultrasonic needle visibility, especially for steeply inserted hand-held needles, and while maintaining clinical utility requirements is needed.

Purpose: This paper introduces the concept of using an additional intracorporeal camera for the specific goal of training and skill assessment and explores the benefits of such an approach. This additional camera can provide an additional view of the surgical scene, and we hypothesize that this additional view would improve surgical training and skill assessment in robot-assisted surgery. Methods: We developed a multi-camera, multi-view system, and we conducted two user studies ($N=12$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N = 12$$\end{document}) to evaluate its effectiveness for training and skill assessment. In the training user study, subjects were divided into two groups: a single-view group and a dual-view group. The skill assessment study was a within-subject study, in which every subject was shown single- and dual view recorded videos of a surgical training task, and the goal was to count the number of errors committed in each video. Results: The results show the effectiveness of using an additional intracorporeal camera view for training and skill assessment. The benefits of this view are modest for skill assessment as it improves the assessment accuracy by approximately 9%. For training, the additional camera view is clearly more effective. Indeed, the dual-view group is 57% more accurate than the single-view group in a retention test. In addition, the dual-view group is 35% more accurate and 25% faster than the single-view group in a transfer test. Conclusion: A multi-camera, multi-view system has the potential to significantly improve training and moderately improve skill assessment in robot-assisted surgery. One application of our work is to include an additional camera view in existing virtual reality surgical training simulators to realize its benefits in training. The views from the additional intracorporeal camera can also be used to improve on existing surgical skill assessment criteria used in training systems for robot-assisted surgery.

Purpose: Eye gaze tracking is proving to be beneficial in many biomedical applications. The performance of systems based on eye gaze tracking is very much dependent on how accurate their calibration is. It has been reported that the gaze tracking accuracy deteriorates cumulatively and significantly with usage time. This impedes the wide use of gaze tracking in user interfaces. Methods: Explicit re-calibration, typically requiring the user’s active attention, is time-consuming and can interfere with the user’s main activity. Therefore, we propose an implicit re-calibration method, which can rectify the deterioration of the gaze tracking accuracy without bringing about the user’s deliberate attention. We make use of hand-eye coordination, with the reasonable assumption that the eye gaze follows the pointer during a selection task, to acquire additional calibration points during normal usage of a gaze-contingent system. We construct a statistical model for the calibration and the hand-eye coordination and apply the Gaussian process regression framework to perform the re-calibration. Results: To validate our model and method, we performed a user study on ultrasonography tasks on a gaze-contingent interface for ultrasound machines. Results suggest that our method can rectify the tracking accuracy deterioration for $75\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$75\%$$\end{document}of all cases where deterioration occurs in our user study. With another benchmark dataset, our method can redress tracking accuracy to a level comparable to the initial calibration in more than $80\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$80\%$$\end{document}of the cases. Conclusions: Our implicit re-calibration method is a practical and convenient fix for tracking accuracy deterioration in gaze-contingent user interfaces, and in particular for gaze-contingent ultrasound machines.

ABSTRACTThis paper presents an automatic pathology (AutoPath) approach to detect prostatic adenocarcinoma based on morphological analysis of high resolution whole mount (WM) histopathology images of the prostate. In the first stage of the cancer detection algorithm, a pre-screening of cancerous regions is performed at low magnification (5×) based on regional features. In the second stage, we propose a novel technique of labelling individual glands as benign or malignant using gland specific features at high magnification (20×). Two new features, Number of Nuclei Layers and Epithelial Layer Density, are proposed to label individual glands. We validate the approach on 70 WM slides, obtained from 30 patients, and achieve average sensitivity of 90%, specificity of 93% and accuracy of 93%. The main advantage of the approach is that detection of individual malignant gland units, irrespective of neighbouring histology and/or the spatial extent of the cancer, allows a finer annotation of cancer. The AutoPath method performs well on slides with low Gleason grades (3 and 4), but is currently limited in its ability to detect cancer in higher Gleason grades.

The original version of this article unfortunately contained a mistake. The presentation of Table 1 was incorrect.

An excavator simulator has been developed to facilitate the training of human operators and to evaluate control strategies for heavy-duty hydraulic machines. The operator controls a virtual excavator by means of a joystick while experiencing visual and force feedback generated by environment and machine models. The simulator comprises an impedance model of the excavator arm, a model for the bucket-ground interaction forces, a graphically rendered visual environment, and a haptic interface. This paper describes the simulator components and their integration.

Focuses on the development of an adaptive motion/force controller for unilateral or bilateral teleoperation systems. Features of a typical teleoperation system; Advantages of the motion controller; Requirements needed to construct the dynamics of the master or slave robots.