Search

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation. [Display omitted] • We refresh tracking and mapping in medical computer vision, reviewing 515 papers. • We summarize limitations and future directions to enable growth in this field. • Machine learning is increasing in impact for tracking and mapping. • Having more publicly available datasets and robust models is essential. • Modern computer vision methods are frequently adapted to the specifics of medical scenes. [ABSTRACT FROM AUTHOR]

Magnetic resonance elastography (MRE) is commonly regarded as the imaging-based gold-standard for liver fibrosis staging, comparable to biopsy. While ultrasound-based elastography methods for liver fibrosis staging have been developed, they are confined to a 1D or a 2D region of interest and to a limited depth. 3D Shear Wave Absolute Vibro-Elastography (S-WAVE) is a steady-state, external excitation, volumetric elastography technique that is similar to MRE, but has the additional advantage of multi-frequency excitation. We present a novel ultrasound matrix array implementation of S-WAVE that takes advantage of 3D imaging. We use a matrix array transducer to sample axial multi-frequency steady-state tissue motion over a volume, using a Color Power Angiography sequence. Tissue motion with the frequency components {40,50,60} and {45,55,65} Hz are acquired over a (90° lateral) $\times $ (40° elevational) $\times $ (16 cm depth) sector with an acquisition time of 12 seconds. We compute the elasticity map in 3D using local spatial frequency estimation. We characterize this new approach in tissue phantoms against measurements obtained with transient elastography and MRE. Six healthy volunteers and eight patients with chronic liver disease were imaged. Their MRE and S-WAVE volumes were aligned using T1 to B-mode registration for direct comparison in common regions of interest. S-WAVE and MRE results are correlated with R2 = 0.92, while MRE and TE results are correlated with R2 = 0.71. Our findings show that S-WAVE with matrix array has the potential to deliver a similar assessment of liver fibrosis as MRE in a more accessible, inexpensive way, to a broader set of patients. [ABSTRACT FROM AUTHOR]

Optimizing Prostate Cancer Treatment in Men with Advanced Local disease (OPTiMAL) is a single-arm phase II study targeted towards unfavourable risk patients. As a successor to the largest randomized clinical trial (ASCENDE) comparing low-dose-rate prostate brachytherapy (LDR-PB) boost to external beam radiation therapy (EBRT) boost for prostate cancer (PCa) treatment, this study is designed to continue the use of the LDR-PB boost that resulted in high levels of biochemical progression-free survival (b-PFS) in ASCENDE while also aiming to reduce adverse side effects by lowering the overall dose and also limiting high dose regions to identified sites of disease. The central treatment policy in OPTiMAL is to give 100% of the minimum prescribed brachytherapy dose (mpd) to the whole prostate gland while limiting regions with ≥150% of the mpd−as much as possible−to areas containing PCa as determined from transperineal template-guided mapping biopsy (TTMB). In addition, the mpd has been lowered to 100 Gy relative to the 115 Gy ASCENDE dose, and EBRT is delivered after the implant to enable the implanted seeds to be used for prostate localization on verification images. Treatment also includes androgen deprivation therapy (ADT) between TTMB and implant. An important additional aim of the OPTiMAL study is to use the TTMB results as a "ground truth" for investigating the potential of multi-modal advanced imaging of the prostate to better delineate tumours and target them appropriately during treatment. Imaging modalities included in the study are as follows: (1) multi-parametric magnetic resonance imaging (mpMRI) following a protocol compatible with PI-RADs V2, (2) magnetic resonance elastography (MRE) (in the same mpMRI session), (3) multi-parametric transrectal ultrasound (mpTRUS), including shear wave absolute vibro-elastography imaging (SWAVE), strain elastography, and time-series B-mode, all acquired immediately prior to and with the patient in position for the TTMB. Although OPTiMAL does not specifically use the findings from these multi-modal imaging studies for brachytherapy treatment planning, the potential of these multi-modal images for PCa localization is being studied by comparing the findings with the TTMB results. Biopsy core locations in the prostate are tracked during the procedure using ultrasound imaging and the VariPath module in VariSeed software (Varian Medical Systems, Palo Alto, CA, United States). Once available, pathology findings are allocated to the tracked cores within the VariPath software and then localized in the multi-modal images using intensity and surface-based image registration to the TTMB ultrasound images and labels. The OPTiMAL study is ongoing and 11 patients have been enrolled so far with 10 patients fulfilling all requirements for LDR-PB boost treatment. Advanced imaging data have been collected for 5 of the patients. Initial analysis of the elastography data (MRE, SWAVE) indicates a positive correlation with the TTMB ground truth. Fig 1 illustrates the correlation of the TTMB findings with the elastography images for P10. Initial analysis suggests that such multi-modal imaging data has the potential to localize tumours reliably which can eventually replace the invasive TTMB procedure altogether. We are currently recruiting more patients and with more imaging and follow-up (relapse rate, side effects endured) data available, a deeper analysis will be conducted. [ABSTRACT FROM AUTHOR]

The purpose of the study was to assess the feasibility of performing intraoperative dosimetry for permanent prostate brachytherapy by combining transrectal ultrasound (TRUS) and fluoroscopy/cone beam CT [CBCT] images and accounting for the effect of prostate deformation. 13 patients underwent TRUS and multiview two-dimensional fluoroscopic imaging partway through the implant, as well as repeat fluoroscopic imaging with the TRUS probe inserted and retracted, and finally three-dimensional CBCT imaging at the end of the implant. The locations of all the implanted seeds were obtained from the fluoroscopy/CBCT images and were registered to prostate contours delineated on the TRUS images based on a common subset of seeds identified on both image sets. Prostate contours were also deformed, using a finite-element model, to take into account the effect of the TRUS probe pressure. Prostate dosimetry parameters were obtained for fluoroscopic and CBCT-dosimetry approaches and compared with the standard-of-care Day-0 postimplant CT dosimetry. High linear correlation (R2 > 0.8) was observed in the measured values of prostate D 90% , V 100% , and V 150% , between the two intraoperative dosimetry approaches. The prostate D 90% and V 100% obtained from intraoperative dosimetry methods were in agreement with the postimplant CT dosimetry. Only the prostate V 150% was on average 4.1% (p -value <0.05) higher in the CBCT-dosimetry approach and 6.7% (p -value <0.05) higher in postimplant CT dosimetry compared with the fluoroscopic dosimetry approach. Deformation of the prostate by the ultrasound probe appeared to have a minimal effect on prostate dosimetry. The results of this study have shown that both of the proposed dosimetric evaluation approaches have potential for real-time intraoperative dosimetry. [ABSTRACT FROM AUTHOR]

• Uncertainty estimation can benefit deep learning-based medical image segmentation. • Disagreement among an ensemble of models provides a good estimation of uncertainty. • Prior shape information can improve uncertain prostate segmentations in ultrasound. • Uncertainty in medical image segmentation is more due to limited data than noise. The goal of this work was to develop a method for accurate and robust automatic segmentation of the prostate clinical target volume in transrectal ultrasound (TRUS) images for brachytherapy. These images can be difficult to segment because of weak or insufficient landmarks or strong artifacts. We devise a method, based on convolutional neural networks (CNNs), that produces accurate segmentations on easy and difficult images alike. We propose two strategies to achieve improved segmentation accuracy on difficult images. First, for CNN training we adopt an adaptive sampling strategy, whereby the training process is encouraged to pay more attention to images that are difficult to segment. Secondly, we train a CNN ensemble and use the disagreement among this ensemble to identify uncertain segmentations and to estimate a segmentation uncertainty map. We improve uncertain segmentations by utilizing the prior shape information in the form of a statistical shape model. Our method achieves Hausdorff distance of 2.7 ± 2.3 mm and Dice score of 93.9 ± 3.5%. Comparisons with several competing methods show that our method achieves significantly better results and reduces the likelihood of committing large segmentation errors. Furthermore, our experiments show that our approach to estimating segmentation uncertainty is better than or on par with recent methods for estimation of prediction uncertainty in deep learning models. Our study demonstrates that estimation of model uncertainty and use of prior shape information can significantly improve the performance of CNN-based medical image segmentation methods, especially on difficult images. [ABSTRACT FROM AUTHOR]

Highlights • A system for automatic histological grading of prostate cancer was developed. • The classifier was trained based on detailed annotations of six pathologists. • The inter-observer variability was addressed by adapting a crowdsourcing approach. • Novel features based on spatial statistics of the nuclei were proposed. • The performance of the classifier was within agreement levels among pathologists. Graphical abstract Abstract Prostate cancer (PCa) is a heterogeneous disease that is manifested in a diverse range of histologic patterns and its grading is therefore associated with an inter-observer variability among pathologists, which may lead to an under- or over-treatment of patients. In this work, we develop a computer aided diagnosis system for automatic grading of PCa in digitized histopathology images using supervised learning methods. Our pipeline comprises extraction of multi-scale features that include glandular, cellular, and image-based features. A number of novel features are proposed based on intra- and inter-nuclei properties; these features are shown to be among the most important ones for classification. We train our classifiers on 333 tissue microarray (TMA) cores that were sampled from 231 radical prostatectomy patients and annotated in detail by six pathologists for different Gleason grades. We also demonstrate the TMA-trained classifier's performance on additional 230 whole-mount slides of 56 patients, independent of the training dataset, by examining the automatic grading on manually marked lesions and randomly sampled 10% of the benign tissue. For the first time, we incorporate a probabilistic approach for supervised learning by multiple experts to account for the inter-observer grading variability. Through cross-validation experiments, the overall grading agreement of the classifier with the pathologists was found to be an unweighted kappa of 0.51, while the overall agreements between each pathologist and the others ranged from 0.45 to 0.62. These results suggest that our classifier's performance is within the inter-observer grading variability levels across the pathologists in our study, which are also consistent with those reported in the literature. [ABSTRACT FROM AUTHOR]

Abstract: Tracking the position and orientation of a 3-D ultrasound transducer has many clinical applications. Tracking requires calibration to find the transformation between the tracking sensor and the ultrasound coordinates. Typically the set of image slice data are scan converted to a Cartesian volume using assumed motor geometry and a single transformation to the sensor. We propose, instead, the calibration of individual slices using a 2-D calibration technique. A best fit to a subset of slices is performed to decrease data collection time compared with that for calibration of all slices, and to reduce the influence of random errors in individual calibrations. We compare our technique with four scan conversion-based techniques: 2-D N-wire on the center slice, N-wire using a 3-D volume, N-wire using a 3-D volume including the edge points and a new closed-form planar method using a 3-D volume. The proposed multi-slice technique produced the smallest point reconstruction error (0.82 mm using a tracked stylus). [Copyright &y& Elsevier]

Abstract: Purpose: To demonstrate that manual prostate segmentation in transrectal ultrasound images can be replaced with semiautomatic segmentation. Methods and Materials: Semiautomatic segmentation using a tapered ellipsoid model was applied to transrectal ultrasound images. Region-based volumetric evaluation was performed between original and physician-reviewed semiautomatic contours. For dosimetric assessment, treatment plans generated on semiautomatic contours were overlaid on physician-reviewed semiautomatic contours and dose parameters were computed. To establish a threshold for the acceptable amount of dosimetric degradation below which the adoption of semiautomatic planning is unacceptable, the range of variability in dosimetric quality attributed to manual variability was obtained and compared with that of semiautomatic contours. Results: An average volume error (1—Dice similarity coefficient) of less than 7% between semiautomatic and manual volumes (140 cases) was obtained. The difference between the mean V 100 of plans created for semiautomatic contours then overlaid on physician-reviewed semiautomatic contours and the original V 100 values, that is, before overlaying on the physician-reviewed contours (41 cases) was lower than 5%. An average total duration of 2–4min, which includes algorithm initialization, 11.67±3.57s algorithm time, and contour modification is required per case. This algorithm is being used at the British Columbia Cancer Agency and to this date has been applied for the treatment of more than 600 patients. Conclusions: In terms of volumetric and dosimetric accuracy, the proposed algorithm is a suitable replacement for manual segmentation in the context of our planning technique. The benefits are shorter segmentation times; greater consistency; less reliance on user experience; and smooth, symmetric contours. [Copyright &y& Elsevier]

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548.

Purpose: To achieve effective robot-assisted laparoscopic prostatectomy, the integration of transrectal ultrasound (TRUS) imaging system which is the most widely used imaging modality in prostate imaging is essential. However, manual manipulation of the ultrasound transducer during the procedure will significantly interfere with the surgery. Therefore, we propose an image co-registration algorithm based on a photoacoustic marker (PM) method, where the ultrasound/photoacoustic (US/PA) images can be registered to the endoscopic camera images to ultimately enable the TRUS transducer to automatically track the surgical instrument., Methods: An optimization-based algorithm is proposed to co-register the images from the two different imaging modalities. The principle of light propagation and an uncertainty in PM detection were assumed in this algorithm to improve the stability and accuracy of the algorithm. The algorithm is validated using the previously developed US/PA image-guided system with a da Vinci surgical robot., Results: The target-registration-error (TRE) is measured to evaluate the proposed algorithm. In both simulation and experimental demonstration, the proposed algorithm achieved a sub-centimeter accuracy which is acceptable in practical clinics (i.e., 1.15 ± 0.29 mm from the experimental evaluation). The result is also comparable with our previous approach (i.e., 1.05 ± 0.37 mm), and the proposed method can be implemented with a normal white light stereo camera and does not require highly accurate localization of the PM., Conclusion: The proposed frame registration algorithm enabled a simple yet efficient integration of commercial US/PA imaging system into laparoscopic surgical setting by leveraging the characteristic properties of acoustic wave propagation and laser excitation, contributing to automated US/PA image-guided surgical intervention applications., (© 2023. CARS.)

This article describes a novel system for quantitative and volumetric measurement of tissue elasticity in the prostate using simultaneous multi-frequency tissue excitation. Elasticity is computed by using a local frequency estimator to measure the three-dimensional local wavelengths of steady-state shear waves within the prostate gland. The shear wave is created using a mechanical voice coil shaker which transmits simultaneous multi-frequency vibrations transperineally. Radio frequency data is streamed directly from a BK Medical 8848 transrectal ultrasound transducer to an external computer where tissue displacement due to the excitation is measured using a speckle tracking algorithm. Bandpass sampling is used that eliminates the need for an ultra-fast frame rate to track the tissue motion and allows for accurate reconstruction at a sampling frequency that is below the Nyquist rate. A roll motor with computer control is used to rotate the transducer and obtain 3D data. Two commercially available phantoms were used to validate both the accuracy of the elasticity measurements as well as the functional feasibility of using the system for in vivo prostate imaging. The phantom measurements were compared with 3D Magnetic Resonance Elastography (MRE), where a high correlation of 96% was achieved. In addition, the system has been used in two separate clinical studies as a method for cancer identification. Qualitative and quantitative results of 11 patients from these clinical studies are presented here. Furthermore, an AUC of 0.87±0.12 was achieved for malignant vs. benign classification using a binary support vector machine classifier trained with data from the latest clinical study with leave one patient out cross-validation.

Real-time transrectal ultrasound (TRUS) image guidance during robot-assisted laparoscopic radical prostatectomy has the potential to enhance surgery outcomes. Whether conventional or photoacoustic TRUS is used, the robotic system and the TRUS must be registered to each other. Accurate registration can be performed using photoacoustic (PA markers). However, this requires a manual search by an assistant [IEEE Robot. Autom. Lett8, 1287 (2023).10.1109/LRA.2022.3191788]. This paper introduces the first automatic search for PA markers using a transrectal ultrasound robot. This effectively reduces the challenges associated with the da Vinci-TRUS registration. This paper investigated the performance of three search algorithms in simulation and experiment: Weighted Average (WA), Golden Section Search (GSS), and Ternary Search (TS). For validation, a surgical prostate scenario was mimicked and various ex vivo tissues were tested. As a result, the WA algorithm can achieve 0.53°±0.30° average error after 9 data acquisitions, while the TS and GSS algorithm can achieve 0.29 ∘ ± 0.31 ∘ and 0.48°±0.32° average errors after 28 data acquisitions., Competing Interests: The authors declare no conflicts of interest., (© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement.)

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Nassir Navab, Regular Member of the Editorial board of Medical image analysis., (Copyright © 2023. Published by Elsevier B.V.)

Quantitative tissue stiffness characterization using ultrasound (US) has been shown to improve prostate cancer (PCa) detection in multiple studies. Shear wave absolute vibro-elastography (SWAVE) allows quantitative and volumetric assessment of tissue stiffness using external multifrequency excitation. This article presents a proof of concept of a first-of-a-kind 3-D hand-operated endorectal SWAVE system designed to be used during systematic prostate biopsy. The system is developed with a clinical US machine, requiring only an external exciter that can be mounted directly to the transducer. Subsector acquisition of radio frequency (RF) data allows imaging of shear waves with a high effective frame rate (up to 250 Hz). The system was characterized using eight different quality assurance phantoms. Due to the invasive nature of prostate imaging, at this early stage of development, validation of in vivo human tissue was instead carried out by intercostally scanning the livers of n = 7 healthy volunteers. The results are compared with 3-D magnetic resonance elastography (MRE) and an existing 3-D SWAVE system with a matrix array transducer (M-SWAVE). High correlations were found with MRE (99% in phantoms, 94% in liver data) and with M-SWAVE (99% in phantoms, 98% in liver data).

Real-time ultrasound imaging plays an important role in ultrasound-guided interventions. The 3-D imaging provides more spatial information compared to conventional 2-D frames by considering the volumes of data. One of the main bottlenecks of 3-D imaging is the long data acquisition time, which reduces practicality and can introduce artifacts from unwanted patient or sonographer motion. This article introduces the first shear wave absolute vibro-elastography (S-WAVE) method with real-time volumetric acquisition using a matrix array transducer. In S-WAVE, an external vibration source generates mechanical vibrations inside the tissue. The tissue motion is then estimated and used in solving a wave equation inverse problem to provide the tissue elasticity. A matrix array transducer is used with a Verasonics ultrasound machine and a frame rate of 2000 volumes/s to acquire 100 radio frequency (RF) volumes in 0.05 s. Using plane wave (PW) and compounded diverging wave (CDW) imaging methods, we estimate axial, lateral, and elevational displacements over 3-D volumes. The curl of the displacements is used with local frequency estimation to estimate elasticity in the acquired volumes. Ultrafast acquisition extends substantially the possible S-WAVE excitation frequency range, now up to 800 Hz, enabling new tissue modeling and characterization. The method was validated on three homogeneous liver fibrosis phantoms and on four different inclusions within a heterogeneous phantom. The homogeneous phantom results show less than 8% (PW) and 5% (CDW) difference between the manufacturer values and the corresponding estimated values over a frequency range of 80-800 Hz. The estimated elasticity values for the heterogeneous phantom at 400-Hz excitation frequency show the average errors of 9% (PW) and 6% (CDW) compared to the provided average values by magnetic resonance elastography (MRE). Furthermore, both imaging methods were able to detect the inclusions within the elasticity volumes. An ex vivo study on a bovine liver sample shows less than 11% (PW) and 9% (CDW) difference between the estimated elasticity ranges by the proposed method and the elasticity ranges provided by MRE and acoustic radiation force impulse (ARFI).

Purpose: Prostate imaging to guide biopsy remains unsatisfactory, with current solutions suffering from high complexity and poor accuracy and reliability. One novel entrant into this field is micro-ultrasound (microUS), which uses a high-frequency imaging probe to achieve very high spatial resolution, and achieves prostate cancer detection rates equivalent to multiparametric magnetic resonance imaging (mpMRI). However, the ExactVu transrectal microUS probe has a unique geometry that makes it challenging to acquire controlled, repeatable three-dimensional (3D) transrectal ultrasound (TRUS) volumes. We describe the design, fabrication, and validation of a 3D acquisition system that allows for the accurate use of the ExactVu microUS device for volumetric prostate imaging., Methods: The design uses a motorized, computer-controlled brachytherapy stepper to rotate the ExactVu transducer about its axis. We perform geometric validation using a phantom with known dimensions and compare performance with magnetic resonance imaging (MRI) using a commercial quality assurance anthropomorphic prostate phantom., Results: Our geometric validation shows accuracy of 1 mm or less in all three directions, and images of an anthropomorphic phantom qualitatively match those acquired using MRI and show good agreement quantitatively., Conclusion: We describe the first system to acquire robotically controlled 3D microUS images using the ExactVu microUS system. The reconstructed 3D microUS images are accurate, which will allow for future applications of the ExactVu microUS system in prostate specimen and in vivo imaging., (© 2023. CARS.)

Purpose: Trans-oral robotic surgery (TORS) using the da Vinci surgical robot is a new minimally-invasive surgery method to treat oropharyngeal tumors, but it is a challenging operation. Augmented reality (AR) based on intra-operative ultrasound (US) has the potential to enhance the visualization of the anatomy and cancerous tumors to provide additional tools for decision-making in surgery., Methods: We propose a US-guided AR system for TORS, with the transducer placed on the neck for a transcervical view. Firstly, we perform a novel MRI-to-transcervical 3D US registration study, comprising (i) preoperative MRI to preoperative US registration, and (ii) preoperative to intraoperative US registration to account for tissue deformation due to retraction. Secondly, we develop a US-robot calibration method with an optical tracker and demonstrate its use in an AR system that displays anatomy models in the surgeon's console in real-time., Results: Our AR system achieves a projection error from the US to the stereo cameras of 27.14 and 26.03 pixels (image is 540[Formula: see text]960) in a water bath experiment. The average target registration error (TRE) for MRI to 3D US is 8.90 mm for the 3D US transducer and 5.85 mm for freehand 3D US, and the TRE for pre-intra operative US registration is 7.90 mm., Conclusion: We demonstrate the feasibility of each component of the first complete pipeline for MRI-US-robot-patient registration for a proof-of-concept transcervical US-guided AR system for TORS. Our results show that trans-cervical 3D US is a promising technique for TORS image guidance., (© 2023. CARS.)

A new development in photoacoustic (PA) imaging has been the use of compact, portable and low-cost laser diodes (LDs), but LD-based PA imaging suffers from low signal intensity recorded by the conventional transducers. A common method to improve signal strength is temporal averaging, which reduces frame rate and increases laser exposure to patients. To tackle this problem, we propose a deep learning method that will denoise point source PA radio-frequency (RF) data before beamforming with a very few frames, even one. We also present a deep learning method to automatically reconstruct point sources from noisy pre-beamformed data. Finally, we employ a strategy of combined denoising and reconstruction, which can supplement the reconstruction algorithm for very low signal-to-noise ratio inputs.

This paper introduces the first integrated real-time intraoperative surgical guidance system, in which an endoscope camera of da Vinci surgical robot and a transrectal ultrasound (TRUS) transducer are co-registered using photoacoustic markers that are detected in both fluorescence (FL) and photoacoustic (PA) imaging. The co-registered system enables the TRUS transducer to track the laser spot illuminated by a pulsed-laser-diode attached to the surgical instrument, providing both FL and PA images of the surgical region-of-interest (ROI). As a result, the generated photoacoustic marker is visualized and localized in the da Vinci endoscopic FL images, and the corresponding tracking can be conducted by rotating the TRUS transducer to display the PA image of the marker. A quantitative evaluation revealed that the average registration and tracking errors were 0.84 mm and 1.16°, respectively. This study shows that the co-registered photoacoustic marker tracking can be effectively deployed intraoperatively using TRUS+PA imaging providing functional guidance of the surgical ROI.

Purpose: Pathology from trans-perineal template mapping biopsy (TTMB) can be used as labels to train prostate cancer classifiers. In this work, we propose a framework to register TTMB cores to advanced volumetric ultrasound data such as multi-parametric transrectal ultrasound (mpTRUS)., Methods: The framework has mainly two steps. First, needle trajectories are calculated with respect to the needle guiding template-considering deflections in their paths. In standard TTMB, a sparsely sampled ultrasound volume is taken prior to the procedure which contains the template overlaid on top of it. The position of this template is detected automatically, and the cores are mapped following the calculated needle trajectories. Second, the TTMB volume is aligned to the mpTRUS volume by a two-step registration method. Using the same transformations from the registration step, the cores are registered from the TTMB volume to the mpTRUS volume., Results: TTMB and mpTRUS of 10 patients were available for this work. The target registration errors (TRE) of the volumes using landmarks picked by three research assistants (RA) and one radiation oncologist (RO) were on average 1.32 ± 0.7 mm and 1.03 ± 0.6 mm, respectively. Additionally, on average, our framework takes only 97 s to register the cores., Conclusion: Our proposed framework allows a quick way to find the spatial location of the cores with respect to volumetric ultrasound. Furthermore, knowing the correct location of the pathology will facilitate focal treatment and will aid in training imaging-based cancer classifiers., (© 2022. CARS.)

We have trained generative adversarial networks (GANs) to mimic both the effect of temporal averaging and of singular value decomposition (SVD) denoising. This effectively removes noise and acquisition artifacts and improves signal-to-noise ratio (SNR) in both the radio-frequency (RF) data and in the corresponding photoacoustic reconstructions. The method allows a single frame acquisition instead of averaging multiple frames, reducing scan time and total laser dose significantly. We have tested this method on experimental data, and quantified the improvement over using either SVD denoising or frame averaging individually for both the RF data and the reconstructed images. We achieve a mean squared error (MSE) of 0.05%, structural similarity index measure (SSIM) of 0.78, and a feature similarity index measure (FSIM) of 0.85 compared to our ground-truth RF results. In the subsequent reconstructions using the denoised data we achieve a MSE of 0.05%, SSIM of 0.80, and a FSIM of 0.80 compared to our ground-truth reconstructions., Competing Interests: The authors declare that there are no conflicts of interest related to this article., (© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement.)

Purpose: In low-dose-rate prostate brachytherapy (LDR-PB), treatment planning is the process of determining the arrangement of implantable radioactive sources that radiates the prostate while sparing healthy surrounding tissues. Currently, these plans are prepared manually by experts incorporating the centre's planning style and guidelines. In this article, we develop a novel framework that can learn a centre's planning strategy and automatically reproduce rapid clinically acceptable plans., Methods: The proposed framework is based on conditional generative adversarial networks that learn our centre's planning style using a pool of 931 historical LDR-PB planning data. Two additional losses that help constrain prohibited needle patterns and produce similar-looking plans are also proposed. Once trained, this model generates an initial distribution of needles which is passed to a planner. The planner then initializes the sources based on the predicted needles and uses a simulated annealing algorithm to optimize their locations further., Results: Quantitative analysis was carried out on 170 cases which showed the generated plans having similar dosimetry to that of the manual plans but with significantly lower planning durations. Indeed, on the test cases, the clinical target volumes achieving [Formula: see text] of the prescribed dose for the generated plans was on average [Formula: see text] ([Formula: see text] for manual plans) with an average planning time of [Formula: see text] min ([Formula: see text] min for manual plans). Further qualitative analysis was conducted by an expert planner who accepted [Formula: see text] of the plans with some changes ([Formula: see text] requiring minor changes & [Formula: see text] requiring major changes)., Conclusion: The proposed framework demonstrated the ability to rapidly generate quality treatment plans that not only fulfil the dosimetric requirements but also takes into account the centre's planning style. Adoption of such a framework would save significant amount of time and resources spent on every patient; boosting the overall operational efficiency of this treatment.

Purpose: This scoping review covers needle visualization and localization techniques in ultrasound, where localization-based approaches mostly aim to compute the needle shaft (and tip) location while potentially enhancing its visibility too., Methods: A literature review is conducted on the state-of-the-art techniques, which could be divided into five categories: (1) signal and image processing-based techniques to augment the needle, (2) modifications to the needle and insertion to help with needle-transducer alignment and visibility, (3) changes to ultrasound image formation, (4) motion-based analysis and (5) machine learning., Results: Advantages, limitations and challenges of representative examples in each of the categories are discussed. Evaluation techniques performed in ex vivo, phantom and in vivo studies are discussed and summarized., Conclusion: Greatest limitation of the majority of the literature is that they rely on original visibility of the needle in the static image. Need for additional/improved apparatus is the greatest limitation toward clinical utility in practice., Significance: Ultrasound-guided needle placement is performed in many clinical applications, including biopsies, treatment injections and anesthesia. Despite the wide range and long history of this technique, an ongoing challenge is needle visibility in ultrasound. A robust technique to enhance ultrasonic needle visibility, especially for steeply inserted hand-held needles, and while maintaining clinical utility requirements is needed.

Purpose: This paper introduces the concept of using an additional intracorporeal camera for the specific goal of training and skill assessment and explores the benefits of such an approach. This additional camera can provide an additional view of the surgical scene, and we hypothesize that this additional view would improve surgical training and skill assessment in robot-assisted surgery., Methods: We developed a multi-camera, multi-view system, and we conducted two user studies ([Formula: see text]) to evaluate its effectiveness for training and skill assessment. In the training user study, subjects were divided into two groups: a single-view group and a dual-view group. The skill assessment study was a within-subject study, in which every subject was shown single- and dual view recorded videos of a surgical training task, and the goal was to count the number of errors committed in each video., Results: The results show the effectiveness of using an additional intracorporeal camera view for training and skill assessment. The benefits of this view are modest for skill assessment as it improves the assessment accuracy by approximately 9%. For training, the additional camera view is clearly more effective. Indeed, the dual-view group is 57% more accurate than the single-view group in a retention test. In addition, the dual-view group is 35% more accurate and 25% faster than the single-view group in a transfer test., Conclusion: A multi-camera, multi-view system has the potential to significantly improve training and moderately improve skill assessment in robot-assisted surgery. One application of our work is to include an additional camera view in existing virtual reality surgical training simulators to realize its benefits in training. The views from the additional intracorporeal camera can also be used to improve on existing surgical skill assessment criteria used in training systems for robot-assisted surgery.

Purpose: Eye gaze tracking is proving to be beneficial in many biomedical applications. The performance of systems based on eye gaze tracking is very much dependent on how accurate their calibration is. It has been reported that the gaze tracking accuracy deteriorates cumulatively and significantly with usage time. This impedes the wide use of gaze tracking in user interfaces., Methods: Explicit re-calibration, typically requiring the user's active attention, is time-consuming and can interfere with the user's main activity. Therefore, we propose an implicit re-calibration method, which can rectify the deterioration of the gaze tracking accuracy without bringing about the user's deliberate attention. We make use of hand-eye coordination, with the reasonable assumption that the eye gaze follows the pointer during a selection task, to acquire additional calibration points during normal usage of a gaze-contingent system. We construct a statistical model for the calibration and the hand-eye coordination and apply the Gaussian process regression framework to perform the re-calibration., Results: To validate our model and method, we performed a user study on ultrasonography tasks on a gaze-contingent interface for ultrasound machines. Results suggest that our method can rectify the tracking accuracy deterioration for [Formula: see text] of all cases where deterioration occurs in our user study. With another benchmark dataset, our method can redress tracking accuracy to a level comparable to the initial calibration in more than [Formula: see text] of the cases., Conclusions: Our implicit re-calibration method is a practical and convenient fix for tracking accuracy deterioration in gaze-contingent user interfaces, and in particular for gaze-contingent ultrasound machines.

Purpose: The current state-of-the-art surgical robotic systems use only a single endoscope to view the surgical field. Research has been conducted to introduce additional cameras to the surgical system, giving rise to new camera angles that cannot be achieved using the endoscope alone. While this additional visualization certainly aids in surgical performance, current systems lack visual-motor compatibility with respect to the additional camera views. We propose a new system that overcomes this limitation., Methods: In this paper, we introduce a novel design of an additional "pickup" camera that can be integrated into the da Vinci Surgical System. We also introduce a solution to work comfortably in the various arbitrary views this camera provides by eliminating visual-motor misalignment. This is done by changing the working frame of the surgical instruments to work with respect to the coordinate system at the "pickup" camera instead of the endoscope., Results: Human user trials ([Formula: see text]) were conducted to evaluate the effect of visual-motor alignment with respect to the "pickup" camera on surgical performance. An inanimate surgical peg transfer task from the validated Fundamentals of Laparoscopic Surgery (FLS) Training Curriculum was used, and an improvement of 73% in task completion time and 80% in accuracy was observed with the visual-motor alignment over the case without it., Conclusion: Our study shows that there is a requirement to achieve visual-motor alignment when utilizing views from external cameras in current clinical surgical robotics setups. We introduce a complete system that provides additional camera views with visual-motor aligned control. Such a system would be useful in existing surgical procedures and could also impact surgical planning and navigation.

Traumatic brain injury (TBI) is one of the leading causes of death and disability worldwide. Detailed studies of the microglial response after TBI require high throughput quantification of changes in microglial count and morphology in histological sections throughout the brain. In this paper, we present a fully automated end-to-end system that is capable of assessing microglial activation in white matter regions on whole slide images of Iba1 stained sections. Our approach involves the division of the full brain slides into smaller image patches that are subsequently automatically classified into white and grey matter sections. On the patches classified as white matter, we jointly apply functional minimization methods and deep learning classification to identify Iba1-immunopositive microglia. Detected cells are then automatically traced to preserve their complex branching structure after which fractal analysis is applied to determine the activation states of the cells. The resulting system detects white matter regions with 84% accuracy, detects microglia with a performance level of 0.70 (F1 score, the harmonic mean of precision and sensitivity) and performs binary microglia morphology classification with a 70% accuracy. This automated pipeline performs these analyses at a 20-fold increase in speed when compared to a human pathologist. Moreover, we have demonstrated robustness to variations in stain intensity common for Iba1 immunostaining. A preliminary analysis was conducted that indicated that this pipeline can identify differences in microglia response due to TBI. An automated solution to microglia cell analysis can greatly increase standardized analysis of brain slides, allowing pathologists and neuroscientists to focus on characterizing the associated underlying diseases and injuries.

Purpose: Conventional ultrasound (US) machines employ a physical control panel (PCP) as the primary user interface for machine control. This panel is adjacent to the main machine display that requires the operator's constant attention. The switch of attention to the control panel can lead to interruptions in the flow of the medical examination. Some ultraportable machines also lack many physical controls. Furthermore, the need to both control the US machine and observe the US image may lead the practitioners to adopt unergonomic postures and repetitive motions that can lead to work-related injuries. Therefore, there is a need for a more efficient human-computer interaction method on US machines., Methods: To tackle some of the limitations with the PCP, we propose to merge the PCP into the main screen of the US machines. We propose to use gaze tracking and a handheld controller so that machine control can be achieved via a multimodal human-computer interaction (HCI) method that does not require one to touch the screen or look away from the US image. As a first step, a pop-up menu and measurement tool were designed on top of the US image based on gaze position for efficient machine control., Results: A comparative study was performed on the BK Medical SonixTOUCH US machine. Participants were asked to complete the task of measuring the area of an ellipse-shaped tumor in a phantom using our gaze-supported HCI method as well as the traditional method. The user study indicates that the task completion time can be reduced by [Formula: see text] when using our gaze-supported HCI, while no extra workload is imposed on the operators., Conclusions: Our preliminary study suggests that, when combined with a simple handheld controller, eye gaze tracking can be integrated into the US machine HCI for more efficient machine control.

Artificial intelligence (AI) - the ability of a machine to perform cognitive tasks to achieve a particular goal based on provided data - is revolutionizing and reshaping our health-care systems. The current availability of ever-increasing computational power, highly developed pattern recognition algorithms and advanced image processing software working at very high speeds has led to the emergence of computer-based systems that are trained to perform complex tasks in bioinformatics, medical imaging and medical robotics. Accessibility to 'big data' enables the 'cognitive' computer to scan billions of bits of unstructured information, extract the relevant information and recognize complex patterns with increasing confidence. Computer-based decision-support systems based on machine learning (ML) have the potential to revolutionize medicine by performing complex tasks that are currently assigned to specialists to improve diagnostic accuracy, increase efficiency of throughputs, improve clinical workflow, decrease human resource costs and improve treatment choices. These characteristics could be especially helpful in the management of prostate cancer, with growing applications in diagnostic imaging, surgical interventions, skills training and assessment, digital pathology and genomics. Medicine must adapt to this changing world, and urologists, oncologists, radiologists and pathologists, as high-volume users of imaging and pathology, need to understand this burgeoning science and acknowledge that the development of highly accurate AI-based decision-support applications of ML will require collaboration between data scientists, computer researchers and engineers.

Purpose: Most of the existing convolutional neural network (CNN)-based medical image segmentation methods are based on methods that have originally been developed for segmentation of natural images. Therefore, they largely ignore the differences between the two domains, such as the smaller degree of variability in the shape and appearance of the target volume and the smaller amounts of training data in medical applications. We propose a CNN-based method for prostate segmentation in MRI that employs statistical shape models to address these issues., Methods: Our CNN predicts the location of the prostate center and the parameters of the shape model, which determine the position of prostate surface keypoints. To train such a large model for segmentation of 3D images using small data (1) we adopt a stage-wise training strategy by first training the network to predict the prostate center and subsequently adding modules for predicting the parameters of the shape model and prostate rotation, (2) we propose a data augmentation method whereby the training images and their prostate surface keypoints are deformed according to the displacements computed based on the shape model, and (3) we employ various regularization techniques., Results: Our proposed method achieves a Dice score of 0.88, which is obtained by using both elastic-net and spectral dropout for regularization. Compared with a standard CNN-based method, our method shows significantly better segmentation performance on the prostate base and apex. Our experiments also show that data augmentation using the shape model significantly improves the segmentation results., Conclusions: Prior knowledge about the shape of the target organ can improve the performance of CNN-based segmentation methods, especially where image features are not sufficient for a precise segmentation. Statistical shape models can also be employed to synthesize additional training data that can ease the training of large CNNs.

Purpose: In the current standard of care, real-time transrectal ultrasound (TRUS) is commonly used for prostate brachytherapy guidance. As TRUS provides limited soft tissue contrast, segmenting the prostate gland in TRUS images is often challenging and subject to inter-observer and intra-observer variability, especially at the base and apex where the gland boundary is hard to define. Magnetic resonance imaging (MRI) has higher soft tissue contrast allowing the prostate to be contoured easily. In this paper, we aim to show that prostate segmentation in TRUS images informed by MRI priors can improve on prostate segmentation that relies only on TRUS images., Methods: First, we compare the TRUS-based prostate segmentation used in the treatment of 598 patients with a high-quality MRI prostate atlas and observe inconsistencies at the apex and base. Second, motivated by this finding, we propose an alternative TRUS segmentation technique that is fully automatic and uses MRI priors. The algorithm uses a convolutional neural network to segment the prostate in TRUS images at mid-gland, where the gland boundary can be clearly seen. It then reconstructs the gland boundary at the apex and base with the aid of a statistical shape model built from an MRI atlas of 78 patients., Results: Compared to the clinical TRUS segmentation, our method achieves similar mid-gland segmentation results in the 598-patient database. For the seven patients who had both TRUS and MRI, our method achieved more accurate segmentation of the base and apex with the MRI segmentation used as ground truth., Conclusion: Our results suggest that utilizing MRI priors in TRUS prostate segmentation could potentially improve the performance at base and apex.

Purpose: This paper presents a new micro-motion-based approach to track a needle in ultrasound images captured by a handheld transducer., Methods: We propose a novel learning-based framework to track a handheld needle by detecting microscale variations of motion dynamics over time. The current state of the art on using motion analysis for needle detection uses absolute motion and hence work well only when the transducer is static. We have introduced and evaluated novel spatiotemporal and spectral features, obtained from the phase image, in a self-supervised tracking framework to improve the detection accuracy in the subsequent frames using incremental training. Our proposed tracking method involves volumetric feature selection and differential flow analysis to incorporate the neighboring pixels and mitigate the effects of the subtle tremor motion of a handheld transducer. To evaluate the detection accuracy, the method is tested on porcine tissue in-vivo, during the needle insertion in the biceps femoris muscle., Results: Experimental results show the mean, standard deviation and root-mean-square errors of [Formula: see text], [Formula: see text] and [Formula: see text] in the insertion angle, and 0.82, 1.21, 1.47 mm, in the needle tip, respectively., Conclusions: Compared to the appearance-based detection approaches, the proposed method is especially suitable for needles with ultrasonic characteristics that are imperceptible in the static image and to the naked eye.

PURPOSE  : Temporal Enhanced Ultrasound (TeUS) has been proposed as a new paradigm for tissue characterization based on a sequence of ultrasound radio frequency (RF) data. We previously used TeUS to successfully address the problem of prostate cancer detection in the fusion biopsies. METHODS  : In this paper, we use TeUS to address the problem of grading prostate cancer in a clinical study of 197 biopsy cores from 132 patients. Our method involves capturing high-level latent features of TeUS with a deep learning approach followed by distribution learning to cluster aggressive cancer in a biopsy core. In this hypothesis-generating study, we utilize deep learning based feature visualization as a means to obtain insight into the physical phenomenon governing the interaction of temporal ultrasound with tissue. RESULTS  : Based on the evidence derived from our feature visualization, and the structure of tissue from digital pathology, we build a simulation framework for studying the physical phenomenon underlying TeUS-based tissue characterization. CONCLUSION  : Results from simulation and feature visualization corroborated with the hypothesis that micro-vibrations of tissue microstructure, captured by low-frequency spectral features of TeUS, can be used for detection of prostate cancer.

Purpose: This paper presents a new approach to detect a standard handheld needle in ultrasound-guided interventions., Methods: Our proposal is to use natural hand tremor, which causes minute displacement of the needle, to detect the needle in ultrasound B-mode images. Subtle displacements arising from tremor motion have a periodic pattern which is usually imperceptible to the naked eye in the B-mode image. We use these displacement measurements in a spatiotemporal framework to detect linear structures with periodic pattern among a sequence of frames. The needle trajectory is estimated as a linear path in the image having maximum spectral correlation with the time trace of displacement due to tremor. A coarse estimation process is followed by a fine estimation step, where the motion pattern is analyzed along spatiotemporal linear paths with various angles originating from the estimated puncture site, within the trajectory channel. Spectral coherency is derived for each sample path versus the reference path, and the needle trajectory is identified as the mean of the sample paths with the maximum coherence within the tremor frequency range., Results: To evaluate the detection accuracy, we tested the method in vivo on porcine tissue, where the needle was inserted into the biceps femoris muscle. To understand whether tremor itself affects needle position, the maximum angular change due to tremor was calculated: mean, standard deviation (SD) and root-mean-square (RMS) measurement of [Formula: see text] and [Formula: see text]. The accuracy of the needle trajectory was calculated by comparing to an expert manual segmentation, averaged over the captured data and presented in mean, SD and RMS error of [Formula: see text] and [Formula: see text], respectively., Conclusion: Results demonstrate that natural tremor motion creates minute coherent motion along the needle, which could be used to localize the needle trajectory within the acceptable accuracy. This method is suitable for standard needles used clinically.

Purpose: The purpose of this research was to determine whether combined ultrasound- and sensor-based compressibility and augmented blood flow measures yielded better results for DVT detection than for the individual measures alone., Methods: Twenty-six limbs from 19 patients were scanned using a sensorized ultrasound DVT screening system, and compressibility and flow measures were obtained at 125 locations. Results from conventional compression ultrasound examination were used as gold standard, with seven vessels (four patients) positive for DVT. A classification approach was used to combine the individual DVT measures per vessel and generate an optimal feature for every possible combination of individual measures. Sensitivity and specificity were calculated for the individual measures and for all combined measures, as was a usefulness criteria [Formula: see text] for measuring class separability., Results: Seven optimal combined features were found with 100% sensitivity and 100% specificity, with the best combined feature having a [Formula: see text] value over two orders of magnitude greater than the best individual DVT measure., Conclusions: The proposed approach for DVT detection combines different aspects of thrombus detection in a novel way generating a quantifiable measure and outperforms any of the individual measures when used independently. All of the combined measures included the flow measure as well as the slope compressibility measure, which uses the magnitude of the force applied by the ultrasound probe, suggesting that these measurements provide important information when characterizing DVT.

Purpose: Dynamic dosimetry is becoming the standard to evaluate the quality of radioactive implants during brachytherapy. For this, it is essential to obtain a 3D visualization of the implanted seeds and their relative position to the prostate. A method was developed to obtain a robust and precise segmentation of seeds in C-arm images, and this approach was tested using clinical datasets., Method: A region-based implicit active contour approach was used to delineate implanted seeds. Then, a template-based matching was employed to segment iodine implants whereas a K-means algorithm is implemented to resolve palladium seed clusters. To validate the method, 55 C-arm images from 10 patients were used for the segmentation of iodine sources, whereas 225 C-arm images from 16 patients were used for the palladium case., Results: Compared to manual ground truth segmentation of 6,002 iodine seeds and 15,354 palladium seeds, 98.7 % of iodine sources were automatically detected and declustered showing a false-positive rate of only 1.7 %. A total of 98.7 % of palladium sources were automatically detected and declustered with a false-positive rate of only 2.0 %., Conclusion: An automated segmentation method was developed that is able to perform the identification and annotation processes of seeds on par with a human expert. This method was shown to be robust and suitable for integration in the dynamic dosimetry workflow of prostate brachytherapy interventions.

In this paper, a novel computer-based virtual training system for prostate brachytherapy is presented. This system incorporates, in a novel way, prior methodologies of ultrasound image synthesis and haptic transrectal ultrasound (TRUS) transducer interaction in a complete simulator that allows a trainee to maneuver the needle and the TRUS, to see the resulting patient-specific images and feel the interaction forces. The simulated TRUS images reflect the volumetric tissue deformation and comprise validated appearance models for the needle and implanted seeds. Rendered haptic forces use validated models for needle shaft flexure and friction, tip cutting, and deflection due to bevel. This paper also presents additional new features that make the simulator complete, in the sense that all aspects of the brachytherapy procedure as practiced at many cancer centers are simulated, including simulations of seed unloading, fluoroscopy imaging, and transversal/sagittal TRUS plane switching. For real-time rendering, methods for fast TRUS-needle-seed image formation are presented. In addition, the simulator computes real-time dosimetry, allowing a trainee to immediately see the consequence of planning changes. The simulation is also patient specific, as it allows the user to import the treatment plan for a patient together with the imaging data in order for a physician to practice an upcoming procedure or for a medical resident to train using typical implant scenarios or rarely encountered cases.

Prostate segmentation in B-mode images is a challenging task even when done manually by experts. In this paper we propose a 3D automatic prostate segmentation algorithm which makes use of information from both ultrasound B-mode and vibro-elastography data.We exploit the high contrast to noise ratio of vibro-elastography images of the prostate, in addition to the commonly used B-mode images, to implement a 2D Active Shape Model (ASM)-based segmentation algorithm on the midgland image. The prostate model is deformed by a combination of two measures: the gray level similarity and the continuity of the prostate edge in both image types. The automatically obtained mid-gland contour is then used to initialize a 3D segmentation algorithm which models the prostate as a tapered and warped ellipsoid. Vibro-elastography images are used in addition to ultrasound images to improve boundary detection.We report a Dice similarity coefficient of 0.87±0.07 and 0.87±0.08 comparing the 2D automatic contours with manual contours of two observers on 61 images. For 11 cases, a whole gland volume error of 10.2±2.2% and 13.5±4.1% and whole gland volume difference of -7.2±9.1% and -13.3±12.6% between 3D automatic and manual surfaces of two observers is obtained. This is the first validated work showing the fusion of B-mode and vibro-elastography data for automatic 3D segmentation of the prostate.

The characterization of tissue viscoelastic properties requires the measurement of tissue motion over a region of interest at frequencies that significantly exceed the frame rates of conventional ultrasound systems. In this paper, we propose that the bandpass sampling technique be applied to tissue motion sampling. With this approach, high-frequency signals limited to a frequency band can be sampled and reconstructed without aliasing at a sampling frequency that is lower than the Nyquist rate. We first review this approach and discuss the selection of the tissue excitation frequency band and of the feasible sampling frequencies that allow signal reconstruction without aliasing. We then demonstrate the approach using simulations based on the finite element method and ultrasound simulations. Finally, we perform experiments on tissue-mimicking materials and demonstrate accurate motion estimation using a lower sampling rate than that required by the conventional sampling theorem. The estimated displacements were used to measure the elasticity and viscosity in a phantom in which an inclusion has been correctly delineated. Thus, with bandpass sampling, it is feasible to use conventional beamforming on diagnostic ultrasound systems to perform high-frequency dynamic elastography. The method is simple to implement because it does not require beam interleaving, additional hardware, or synchronization.

We aim to compute radioactive stranded-implant displacement during and after prostate brachytherapy. We present the methods used to identify corresponding seeds in planned, intra-operative and postimplant patient data that enable us to compute seed displacements. A minimum cost network flow algorithm is used, on 8 patients, for needle track detection to group seeds into needles that can be matched between datasets. An iterative best line detection algorithm is used both to help with needle detection and to register the different datasets. Our results show that there was an average seed misplacement of 5.08 +/- 2.35 mm during the procedure, which then moved another 3.10 +/- 1.91 mm by the time the quality assurance CT was taken. Several directional trends in different regions of the prostate were noted and commented on.

Intraoperative dosimetry during prostate brachytherapy is a long standing clinical problem. We propose a novel framework to address this problem by reliable detection of a subset of seeds from 3D transrectal ultrasound and registration to fluoroscopy. Seed detection in ultrasound is achieved through template matching in the RF ultrasound domain followed by thresholding and spatial filtering based on the fixed distance between stranded seeds. This subset of seeds is registered to the complete reconstruction of the implant in C-arm fluoroscopy. We report results, validated with a leave-one-needle-out approach, both in a phantom (average post-registration seed distance of 2.5 mm) and in three clinical patient datasets (average error: 3.9 mm over 113 seeds).

This paper describes a new ultrasound-based system for high-frame-rate measurement of periodic motion in 2-D for tissue elasticity imaging. Similarly to conventional 2-D flow vector imaging, the system acquires the RF signals from the region of interest at multiple steering angles. A custom sector subdivision technique is used to increase the temporal resolution while keeping the total acquisition time within the range suitable for real-time applications. Within each sector, 1-D motion is estimated along the beam direction. The intra- and inter-sector delays are compensated using our recently introduced delay compensation algorithm. In-plane 2-D motion vectors are then reconstructed from these delay-compensated 1-D motions. We show that Young's modulus images can be reconstructed from these 2-D motion vectors using local inversion algorithms. The performance of the system is validated quantitatively using a commercial flow phantom and a commercial elasticity phantom. At the frame rate of 1667 Hz, the estimated flow velocities with the system are in agreement with the velocity measured with a pulsed-wave Doppler imaging mode of a commercial ultrasound machine with manual angle correction. At the frame rate of 1250 Hz, phantom Young's moduli of 29, 6, and 54 kPa for the background, the soft inclusion, and the hard inclusion, are estimated to be 30, 11, and 53 kPa, respectively.

A widely used time-domain technique for motion or delay estimation between digitized ultrasound RF signals involves the maximization of a discrete pattern-matching function, usually the cross-correlation. To achieve sub-sample accuracy, the discrete pattern-matching function is interpolated using the values at the discrete maximizer and adjacent samples. In prior work, only 1-D fit, applied separately along the axial, lateral, and elevational axes, has been used to estimate the sub-sample motion in 1-D, 2-D, and 3-D. In this paper, we explore the use of 2-D and 3-D polynomial fitting for this purpose. We quantify the estimation error in noise-free simulations using Field II and experiments with a commercial ultrasound machine. In simulated 2-D translational motions, function fitting with quartic spline polynomials leads to maximum bias of 0.2% of the sample spacing in the axial direction and 0.4% of the sample spacing in the lateral direction, corresponding to 38 nm and 1.31 μm, respectively. The maximum standard deviations were approximately 1% of the sample spacing in both the axial and the lateral directions, corresponding to 193 nm axially and 4.43 μm laterally. In simulated 1% axial strain, the same function fitting leads to mean absolute displacement estimation errors of 255 nm in the axial direction and 4.77 μm in the lateral direction. In experiments with a linear array transducer, 2-D quartic spline fitting leads to maximum bias of 458 nm and 6.27 μm in the axial and the lateral directions, respectively. These results are more than one order of magnitude smaller than those obtained with separate 1-D fit when applied to the same data set. Simulations and experiments in 3-D yield similar results when comparing 3-D polynomial fitting with 1-D fitting along the axial, lateral, and elevational directions.