Search

Purpose: Real-time assessment of surgical margins is critical for favorable outcomes in cancer patients. The iKnife is a mass spectrometry device that has demonstrated potential for margin detection in cancer surgery. Previous studies have shown that using deep learning on iKnife data can facilitate real-time tissue characterization. However, none of the existing literature on the iKnife facilitate the use of publicly available, state-of-the-art pretrained networks or datasets that have been used in computer vision and other domains. Methods: In a new framework we call ImSpect, we convert 1D iKnife data, captured during basal cell carcinoma (BCC) surgery, into 2D images in order to capitalize on state-of-the-art image classification networks. We also use self-supervision to leverage large amounts of unlabeled, intraoperative data to accommodate the data requirements of these networks. Results: Through extensive ablation studies, we show that we can surpass previous benchmarks of margin evaluation in BCC surgery using iKnife data, achieving an area under the receiver operating characteristic curve (AUC) of 81%. We also depict the attention maps of the developed DL models to evaluate the biological relevance of the embedding space Conclusions: We propose a new method for characterizing tissue at the surgical margins, using mass spectrometry data from cancer surgery.

Purpose: The standard of care for prostate cancer (PCa) diagnosis is the histopathological analysis of tissue samples obtained via transrectal ultrasound (TRUS) guided biopsy. Models built with deep neural networks (DNNs) hold the potential for direct PCa detection from TRUS, which allows targeted biopsy and subsequently enhances outcomes. Yet, there are ongoing challenges with training robust models, stemming from issues such as noisy labels, out-of-distribution (OOD) data, and limited labeled data. Methods: This study presents LensePro, a unified method that not only excels in label efficiency but also demonstrates robustness against label noise and OOD data. LensePro comprises two key stages: first, self-supervised learning to extract high-quality feature representations from abundant unlabeled TRUS data and, second, label noise-tolerant prototype-based learning to classify the extracted features. Results: Using data from 124 patients who underwent systematic prostate biopsy, LensePro achieves an AUROC, sensitivity, and specificity of 77.9%, 85.9%, and 57.5%, respectively, for detecting PCa in ultrasound. Our model shows it is effective for detecting OOD data in test time, critical for clinical deployment. Ablation studies demonstrate that each component of our method improves PCa detection by addressing one of the three challenges, reinforcing the benefits of a unified approach. Conclusion: Through comprehensive experiments, LensePro demonstrates its state-of-the-art performance for TRUS-based PCa detection. Although further research is necessary to confirm its clinical applicability, LensePro marks a notable advancement in enhancing automated computer-aided systems for detecting prostate cancer in ultrasound.

Purpose: Deep learning-based analysis of micro-ultrasound images to detect cancerous lesions is a promising tool for improving prostate cancer (PCa) diagnosis. An ideal model should confidently identify cancer while responding with appropriate uncertainty when presented with out-of-distribution inputs that arise during deployment due to imaging artifacts and the biological heterogeneity of patients and prostatic tissue. Methods: Using micro-ultrasound data from 693 patients across 5 clinical centers who underwent micro-ultrasound guided prostate biopsy, we train and evaluate convolutional neural network models for PCa detection. To improve robustness to out-of-distribution inputs, we employ and comprehensively benchmark several state-of-the-art uncertainty estimation methods. Results: PCa detection models achieve performance scores up to $76\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$76\%$$\end{document}average AUROC with a 10-fold cross validation setup. Models with uncertainty estimation obtain expected calibration error scores as low as $2\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2\%$$\end{document}, indicating that confident predictions are very likely to be correct. Visualizations of the model output demonstrate that the model correctly identifies healthy versus malignant tissue. Conclusion: Deep learning models have been developed to confidently detect PCa lesions from micro-ultrasound. The performance of these models, determined from a large and diverse dataset, is competitive with visual analysis of magnetic resonance imaging, the clinical benchmark to identify PCa lesions for targeted biopsy. Deep learning with micro-ultrasound should be further studied as an avenue for targeted prostate biopsy.

Lung ultrasound (LUS) is an important imaging modality used by emergency physicians to assess pulmonary congestion at the patient bedside. B-line artifacts in LUS videos are key findings associated with pulmonary congestion. Not only can the interpretation of LUS be challenging for novice operators, but visual quantification of B-lines remains subject to observer variability. In this work, we investigate the strengths and weaknesses of multiple deep learning approaches for automated B-line detection and localization in LUS videos. We curate and publish, BEDLUS, a new ultrasound dataset comprising 1,419 videos from 113 patients with a total of 15,755 expert-annotated B-lines. Based on this dataset, we present a benchmark of established deep learning methods applied to the task of B-line detection. To pave the way for interpretable quantification of B-lines, we propose a novel “single-point” approach to B-line localization using only the point of origin. Our results show that (a) the area under the receiver operating characteristic curve ranges from 0.864 to 0.955 for the benchmarked detection methods, (b) within this range, the best performance is achieved by models that leverage multiple successive frames as input, and (c) the proposed single-point approach for B-line localization reaches an F$_{1}$-score of 0.65, performing on par with the inter-observer agreement. The dataset and developed methods can facilitate further biomedical research on automated interpretation of lung ultrasound with the potential to expand the clinical utility.

Deep learning-based analysis of high-frequency, high-resolution micro-ultrasound data shows great promise for prostate cancer (PCa) detection. Previous approaches to analysis of ultrasound data largely follow a supervised learning (SL) paradigm. Ground truth labels for ultrasound images used for training deep networks often include coarse annotations generated from the histopathological analysis of tissue samples obtained via biopsy. This creates inherent limitations on the availability and quality of labeled data, posing major challenges to the success of SL methods. However, unlabeled prostate ultrasound data are more abundant. In this work, we successfully apply self-supervised representation learning to micro-ultrasound data. Using ultrasound data from 1028 biopsy cores of 391 subjects obtained in two clinical centers, we demonstrate that feature representations learned with this method can be used to classify cancer from noncancer tissue, obtaining an AUROC score of 91% on an independent test set. To the best of our knowledge, this is the first successful end-to-end self-SL (SSL) approach for PCa detection using ultrasound data. Our method outperforms baseline SL approaches, generalizes well between different data centers, and scales well in performance as more unlabeled data are added, making it a promising approach for future research using large volumes of unlabeled data. Our code is publicly available at https://www.github.com/MahdiGilany/SSL_micro_ultrasound.

ABSTRACTUltrasound imaging is commonly used to guide sampling the prostate tissue in transrectal biopsies, followed by detection of cancer through histopathological analysis and coarse labelling of sampled tissue. Ideally, the procedure should be improved by developing machine learning solutions that can identify the presence of cancer in ultrasound images to guide the biopsy procedure. Training a fully supervised learning model using coarse histopathology labels suffers from weakly annotated data which introduce label noise for each image pixel. To address this challenge, we propose a semi-supervised framework for learning with noisy labels. We leverage a two-component mixture model to cluster the training data into clean and noisy label samples based on their loss values. Then, during the semi-supervised training phase, we utilise the well-known MixMatch algorithm which incorporates consistency regularisation, entropy minimisation, and the Mixup regularisation as well as the cross-entropy loss function for noisy and clean sets, respectively. We evaluate the proposed framework with prostate ultrasound data obtained from 71 subjects, while sampling 264 biopsy cores. We achieve balanced accuracy, sensitivity, and specificity of 78.6%, 80.0%, and 77.1%, respectively. In a detailed comparison study, we demonstrate that our proposed framework outperforms the fully supervised method with state-of-the-art robust loss functions.

Purpose: A large body of previous machine learning methods for ultrasound-based prostate cancer detection classify small regions of interest (ROIs) of ultrasound signals that lie within a larger needle trace corresponding to a prostate tissue biopsy (called biopsy core). These ROI-scale models suffer from weak labeling as histopathology results available for biopsy cores only approximate the distribution of cancer in the ROIs. ROI-scale models do not take advantage of contextual information that are normally considered by pathologists, i.e., they do not consider information about surrounding tissue and larger-scale trends when identifying cancer. We aim to improve cancer detection by taking a multi-scale, i.e., ROI-scale and biopsy core-scale, approach. Methods: Our multi-scale approach combines (i) an “ROI-scale” model trained using self-supervised learning to extract features from small ROIs and (ii) a “core-scale” transformer model that processes a collection of extracted features from multiple ROIs in the needle trace region to predict the tissue type of the corresponding core. Attention maps, as a by-product, allow us to localize cancer at the ROI scale. Results: We analyze this method using a dataset of micro-ultrasound acquired from 578 patients who underwent prostate biopsy, and compare our model to baseline models and other large-scale studies in the literature. Our model shows consistent and substantial performance improvements compared to ROI-scale-only models. It achieves $80.3\%$\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$80.3\%$$\end{document}AUROC, a statistically significant improvement over ROI-scale classification. We also compare our method to large studies on prostate cancer detection, using other imaging modalities. Conclusions: Taking a multi-scale approach that leverages contextual information improves prostate cancer detection compared to ROI-scale-only models. The proposed model achieves a statistically significant improvement in performance and outperforms other large-scale studies in the literature. Our code is publicly available at www.github.com/med-i-lab/TRUSFormer.

Unlike left ventricular (LV) ejection fraction, which provides a precise, reliable, and prognostically valuable measure of systolic function, there is no single analogous measure of LV diastolic function.

Purpose: Rapid evaporative ionization mass spectrometry (REIMS) is an emerging technology for clinical margin detection. Deployment of REIMS depends on construction of reliable deep learning models that can categorize tissue according to its metabolomic signature. Challenges associated with developing these models include the presence of noise during data acquisition and the variance in tissue signatures between patients. In this study, we propose integration of uncertainty estimation in deep models to factor predictive confidence into margin detection in cancer surgery. Methods: iKnife is used to collect 693 spectra of cancer and healthy samples acquired from 91 patients during basal cell carcinoma resection. A Bayesian neural network and two baseline models are trained on these data to perform classification as well as uncertainty estimation. The samples with high estimated uncertainty are then removed, and new models are trained using the clean data. The performance of proposed and baseline models, with different ratios of filtered data, is then compared. Results: The data filtering does not improve the performance of the baseline models as they cannot provide reliable estimations of uncertainty. In comparison, the proposed model demonstrates a statistically significant improvement in average balanced accuracy (75.2%), sensitivity (74.1%) and AUC (82.1%) after removing uncertain training samples. We also demonstrate that if highly uncertain samples are predicted and removed from the test data, sensitivity further improves to 88.2%. Conclusions: This is the first study that applies uncertainty estimation to inform model training and deployment for tissue recognition in cancer surgery. Uncertainty estimation is leveraged in two ways: by factoring a measure of input noise in training the models and by including predictive confidence in reporting the outputs. We empirically show that considering uncertainty for model development can help improve the overall accuracy of a margin detection system using REIMS.

Purpose: Using machine learning, we developed a proprietary ultrasound software called the Spine Level Identification (SLIDE) system, which automatically identifies lumbar landmarks in real time as the operator slides the transducer over the lumber spine. Here, we assessed the agreement between SLIDE and manual palpation and traditional lumbar ultrasound (LUS) for determining the primary target L3–4 interspace. Methods: Upon institutional ethics approval and informed consent, 76 healthy term parturients scheduled for elective Caesarean delivery were recruited. The L3–4 interspace was identified by manual palpation and then by the SLIDE method. The reference standard was located using traditional LUS by an experienced operator. The primary outcome was the L3–4 interspace identification agreement of manual palpation and SLIDE with the reference standard, as percentage agreement and Gwet’s agreement coefficient (AC1). Results: The raw agreement was 70% with Gwet’s agreement coefficient (AC1) = 0.59 (95% confidence interval [CI], 0.41 to 0.77) for manual palpation and 84% with Gwet’s AC1 = 0.82 (95% CI, 0.70 to 0.93) for SLIDE. When the levels differ from the reference, the manual palpation method identified L2–3 more often than L4–5 while the SLIDE method identified equally above or below L3–4. The SLIDE system had greater agreement than palpation in locating L3–4 and all other lumber interspaces after controlling for body mass index (adjusted odds ratio, 2.99; 95% CI, 1.21 to 8.7; P= 0.02). Conclusion: The SLIDE system had higher agreement with traditional ultrasound than manual palpation did in identifying L3–4 and all other lumber interspaces after adjusting for BMI in healthy term obstetric patients. Future studies should examine factors that affect agreement and ways to improve SLIDE for clinical integration. Study Registration: www.ClinicalTrials.gov(NCT02982317); registered 5 December 2016.

Purpose:: Ultrasound-guided biopsy plays a major role in prostate cancer (PCa) detection, yet is limited by a high rate of false negatives and low diagnostic yield of the current systematic, non-targeted approaches. Developing machine learning models for accurately identifying cancerous tissue in ultrasound would help sample tissues from regions with higher cancer likelihood. A plausible approach for this purpose is to use individual ultrasound signals corresponding to a core as inputs and consider the histopathology diagnosis for the entire core as labels. However, this introduces significant amount of label noise to training and degrades the classification performance. Previously, we suggested that histopathology-reported cancer involvement can be a reasonable approximation for the label noise. Methods:: Here, we propose an involvement-based label refinement (iLR) method to correct corrupted labels and improve cancer classification. The difference between predicted and true cancer involvements is used to guide the label refinement process. We further incorporate iLR into state-of-the-art methods for learning with noisy labels and predicting cancer involvement. Results:: We use 258 biopsy cores from 70 patients and demonstrate that our proposed label refinement method improves the performance of multiple noise-tolerant approaches and achieves a balanced accuracy, correlation coefficient, and mean absolute error of 76.7%, 0.68, and 12.4, respectively. Conclusions:: Our key contribution is to leverage a data-centric method to deal with noisy labels using histopathology reports, and improve the performance of prostate cancer diagnosis through a hierarchical training process with label refinement.

Purpose:: Ultrasound is the standard-of-care to guide the systematic biopsy of the prostate. During the biopsy procedure, up to 12 biopsy cores are randomly sampled from six zones within the prostate, where the histopathology of those cores is used to determine the presence and grade of the cancer. Histopathology reports only provide statistical information on the presence of cancer and do not normally contain fine-grain information of cancer distribution within each core. This limitation hinders the development of machine learning models to detect the presence of cancer in ultrasound so that biopsy can be more targeted to highly suspicious prostate regions. Methods:: In this paper, we tackle this challenge in the form of training with noisy labels derived from histopathology. Noisy labels often result in the model overfitting to the training data, hence limiting its generalizability. To avoid overfitting, we focus on the generalization of the features of the model and present an iterative data label refinement algorithm to amend the labels gradually. We simultaneously train two classifiers, with the same structure, and automatically stop the training when we observe any sign of overfitting. Then, we use a confident learning approach to clean the data labels and continue with the training. This process is iteratively applied to the training data and labels until convergence. Results:: We illustrate the performance of the proposed method by classifying prostate cancer using a dataset of ultrasound images from 353 biopsy cores obtained from 90 patients. We achieve area under the curve, sensitivity, specificity, and accuracy of 0.73, 0.80, 0.63, and 0.69, respectively. Conclusion:: Our approach is able to provide clinicians with a visualization of regions that likely contain cancerous tissue to obtain more accurate biopsy samples. The results demonstrate that our proposed method produces superior accuracy compared to the state-of-the-art methods.

Purpose: Systematic prostate biopsy is widely used for cancer diagnosis. The procedure is blind to underlying prostate tissue micro-structure; hence, it can lead to a high rate of false negatives. Development of a machine-learning model that can reliably identify suspicious cancer regions is highly desirable. However, the models proposed to-date do not consider the uncertainty present in their output or the data to benefit clinical decision making for targeting biopsy. Methods: We propose a deep network for improved detection of prostate cancer in systematic biopsy considering both the label and model uncertainty. The architecture of our model is based on U-Net, trained with temporal enhanced ultrasound (TeUS) data. We estimate cancer detection uncertainty using test-time augmentation and test-time dropout. We then use uncertainty metrics to report the cancer probability for regions with high confidence to help the clinical decision making during the biopsy procedure. Results: Experiments for prostate cancer classification includes data from 183 prostate biopsy cores of 41 patients. We achieve an area under the curve, sensitivity, specificity and balanced accuracy of 0.79, 0.78, 0.71 and 0.75, respectively. Conclusion: Our key contribution is to automatically estimate model and label uncertainty towards enabling targeted ultrasound-guided prostate biopsy. We anticipate that such information about uncertainty can decrease the number of unnecessary biopsy with a higher rate of cancer yield.

PurposeA solution for real-time, ultrasound-guided, central neuraxial blockade placement remains elusive. A device that enables single-operator neuraxial placement while simultaneously visualising the spinal anatomy and needle trajectory may improve patient safety. We engineered a novel needle guide, the EpiGuide two dimensional (2D), and compared prepuncture insertion sites as located using the guide versus standard manual palpation.MethodsInterdisciplinary collaboration between engineers and obstetric anaesthesiologists and multiple iterative refinement led to the EpiGuide 2D, a prototype multichannel needle guide for ultrasound transducers. Following ethics committee approval, 22 healthy adult participants were recruited to undergo prepuncture lumbar epidural placement using the guide. The primary outcome was accuracy, defined as the percentage of successful placements of prepuncture needle insertion sites within the manually palpated intervertebral space. The secondary outcome was distance between the prepuncture insertion sites guided by the EpiGuide 2D versus by sites guided by manually palpation.ResultsMean (SD) body mass index of participants was 22.6 (2.1) kg/m2. The success rate of selecting a prepuncture site within the manually palpated intervertebral space using EpiGuide 2D was 95.5% (42 of 44 attempts). Of the failed sites, the mean (SD) distance to the intervertebral space was 1.3 (0.3) mm in the caudal direction. The mean (SD) distance between EpiGuide 2D sites and palpated sites was 3.6 (2.0) mm.ConclusionThe EpiGuide 2D, jointly developed between engineers and anaesthesiologists, was found to be as accurate as manual palpation in placing a prepuncture site within the intervertebral space.

ABSTRACTHeart failure with reduced ejection fraction (HFrEF) is associated with high mortality rates. The identification of patients with HFrEF in urgent and emergent situations can be delayed in the absence of clinicians skilled in acquiring and interpreting echocardiography (echo) images. Although the standard-of-care for measuring ejection fraction (EF) involves segmentation of the left ventricle in echo, experienced interpreters often opt for visual estimation of EF in clinical settings. In this paper, we mimic this process using dual-channel deep neural networks for segmentation-free classification of echo cine loops: low-risk () vs. high-risk (reduced EF, i.e. ). The proposed architecture utilises densely connected networks for extraction of unsupervised spatial features. Temporal embedding is then achieved by aggregating these frame-level feature vectors and feeding them through recurrent blocks. We use 949 and 237 clinical echo exams to train and validate the proposed method, respectively. We demonstrate that direct cine-based detection of patients at high-risk of HFrEF is feasible using densely connected convolutional and recurrent-based architectures.

Purpose: Optimizing patient position and needle puncture site are important factors for successful neuraxial anesthesia. Two paramedian approaches are commonly utilized and we sought to determine whether variations of the seated position would increase the chance of puncture success. Methods: We simulated paramedian needle passes on three-dimensional lumbar spine models registered to volumetric ultrasound data acquired from ten healthy volunteers in three different positions: 1) prone; 2) seated with thoracic and lumbar flexion; and 3) seated as in position 2, but with a 10° dorsal tilt. Simulated paramedian needle passes from the right side performed on validated models were used to determine L2–3 and L3–4 neuraxial target size and success. We selected two paramedian puncture sites according to standard anesthesia textbook descriptions: 10 mm lateral and 10 mm caudal from inferior edge of the superior spinous process as described by Miller, and 10 mm lateral from the superior edge of the inferior spinous process as described by Barash. Results: A significant increase in the area available for dural puncture was found in the L2–3 (61–62 mm2) and L3–4 (76–79 mm2) vertebral levels for all seated positions relative to the prone position (P< 0.001). Similarly, a significant increase in the total number of successful punctures was found in the L2–3 (77–79) and L3–4 (119–120) vertebral levels for all seated positions relative to the prone position (P< 0.001). No differences were found between seated positions. The Barash puncture site achieved a higher number of successful punctures than the Miller puncture site in both the L2–3 (19) and L3–4 (84) vertebral levels (P< 0.001). Conclusion: An added dorsal table tilt did not increase puncture success in the seated position. The landmarks for puncture site described by Barash resulted in significantly more successful punctures compared with those described by Miller in all positions.

Purpose: The detection of clinically significant prostate cancer (PCa) is shown to greatly benefit from MRI–ultrasound fusion biopsy, which involves overlaying pre-biopsy MRI volumes (or targets) with real-time ultrasound images. In previous literature, machine learning models trained on either MRI or ultrasound data have been proposed to improve biopsy guidance and PCa detection. However, quantitative fusion of information from MRI and ultrasound has not been explored in depth in a large study. This paper investigates information fusion approaches between MRI and ultrasound to improve targeting of PCa foci in biopsies. Methods: We build models of fully convolutional networks (FCN) using data from a newly proposed ultrasound modality, temporal enhanced ultrasound (TeUS), and apparent diffusion coefficient (ADC) from 107 patients with 145 biopsy cores. The architecture of our models is based on U-Net and U-Net with attention gates. Models are built using joint training through intermediate and late fusion of the data. We also build models with data from each modality, separately, to use as baseline. The performance is evaluated based on the area under the curve (AUC) for predicting clinically significant PCa. Results: Using our proposed deep learning framework and intermediate fusion, integration of TeUS and ADC outperforms the individual modalities for cancer detection. We achieve an AUC of 0.76 for detection of all PCa foci, and 0.89 for PCa with larger foci. Results indicate a shared representation between multiple modalities outperforms the average unimodal predictions. Conclusion: We demonstrate the significant potential of multimodality integration of information from MRI and TeUS to improve PCa detection, which is essential for accurate targeting of cancer foci during biopsy. By using FCNs as the architecture of choice, we are able to predict the presence of clinically significant PCa in entire imaging planes immediately, without the need for region-based analysis. This reduces the overall computational time and enables future intra-operative deployment of this technology.

Purpose: Ultrasound imaging is routinely used in prostate biopsy, which involves obtaining prostate tissue samples using a systematic, yet, non-targeted approach. This approach is blinded to individual patient intraprostatic pathology, and unfortunately, has a high rate of false negatives. Methods: In this paper, we propose a deep network for improved detection of prostate cancer in systematic biopsy. We address several challenges associated with training such network: (1) Statistical labels: Since biopsy core’s pathology report only represents a statistical distribution of cancer within the core, we use multiple instance learning (MIL) networks to enable learning from ultrasound image regions associated with those data; (2) Limited labels: The number of biopsy cores are limited to at most 12 per patient. As a result, the number of samples available for training a deep network is limited. We alleviate this issue by effectively combining Independent Conditional Variational Auto Encoders (ICVAE) with MIL. We train ICVAE to learn label-invariant features of RF data, which is subsequently used to generate synthetic data for improved training of the MIL network. Results: Our in vivo study includes data from 339 prostate biopsy cores of 70 patients. We achieve an area under the curve, sensitivity, specificity, and balanced accuracy of 0.68, 0.77, 0.55 and 0.66, respectively. Conclusion: The proposed approach is generic and can be applied to several other scenarios where unlabeled data and noisy labels in training samples are present.

Purpose: The emerging market of cardiac handheld ultrasound (US) is on the rise. Despite the advantages in ease of access and the lower cost, a gap in image quality can still be observed between the echocardiography (echo) data captured by point-of-care ultrasound (POCUS) compared to conventional cart-based US, which limits the further adaptation of POCUS. In this work, we aim to present a machine learning solution based on recent advances in adversarial training to investigate the feasibility of translating POCUS echo images to the quality level of high-end cart-based US systems. Methods: We propose a constrained cycle-consistent generative adversarial architecture for unpaired translation of cardiac POCUS to cart-based US data. We impose a structured shape-wise regularization via a critic segmentation network to preserve the underlying shape of the heart during quality translation. The proposed deep transfer model is constrained to the anatomy of the left ventricle (LV) in apical two-chamber (AP2) echo views. Results: A total of 1089 echo studies from 841 patients are used in this study. The AP2 frames are captured by POCUS (Philips Lumify and Clarius) and cart-based (Philips iE33 and Vivid E9) US machines. The dataset of quality translation comprises a total of 441 echo studies from 395 patients. Data from both POCUS and cart-based systems of the same patient were available in 122 cases. The deep-quality transfer model is integrated into a pipeline for an automated cardiac evaluation task, namely segmentation of LV in AP2 view. By transferring the low-quality POCUS data to the cart-based US, a significant average improvement of 30% and 34 mm is obtained in the LV segmentation Dice score and Hausdorff distance metrics, respectively. Conclusion: This paper presents the feasibility of a machine learning solution to transform the image quality of POCUS data to that of high-quality high-end cart-based systems. The experiments show that by leveraging the quality translation through the proposed constrained adversarial training, the accuracy of automatic segmentation with POCUS data could be improved.

Prostate cancer (PCa) is the most frequent noncutaneous cancer in men. Early detection of PCa is essential for clinical decision making, and reducing metastasis and mortality rates. The current approach for PCa diagnosis is histopathologic analysis of core biopsies taken under transrectal ultrasound guidance (TRUS-guided). Both TRUS-guided systematic biopsy and MR-TRUS-guided fusion biopsy have limitations in accurately identifying PCa, intraoperatively. There is a need to augment this process by visualizing highly probable areas of PCa. Temporal enhanced ultrasound (TeUS) has emerged as a promising modality for PCa detection. Prior work focused on supervised classification of PCa verified by gold standard pathology. Pathology labels are noisy, and data from an entire core have a single label even when significantly heterogeneous. Additionally, supervised methods are limited by data from cores with known pathology, and a significant portion of prostate data is discarded without being used. We provide an end-to-end unsupervised solution to map PCa distribution from TeUS data using an innovative representation learning method, deep neural maps. TeUS data are transformed to a topologically arranged hyper-lattice, where similar samples are closer together in the lattice. Therefore, similar regions of malignant and benign tissue in the prostate are clustered together. Our proposed method increases the number of training samples by several orders of magnitude. Data from biopsy cores with known labels are used to associate the clusters with PCa. Cancer probability maps generated using the unsupervised clustering of TeUS data help intuitively visualize the distribution of abnormal tissue for augmenting TRUS-guided biopsies.

Left ventricular ejection fraction (LVEF) is one of the key metrics to assess the heart functionality, and cardiac ultrasound (echo) is a standard imaging modality for EF measurement. There is an emerging interest to exploit the point-of-care ultrasound (POCUS) usability due to low cost and ease of access. In this work, we aim to present a computationally efficient mobile application for accurate LVEF estimation. Our proposed mobile application for LVEF estimation runs in real time on Android mobile devices that have either a wired or wireless connection to a cardiac POCUS device. We propose a pipeline for biplane ejection fraction estimation using apical two-chamber (AP2) and apical four-chamber (AP4) echo views. A computationally efficient multi-task deep fully convolutional network is proposed for simultaneous LV segmentation and landmark detection in these views, which is integrated into the LVEF estimation pipeline. An adversarial critic model is used in the training phase to impose a shape prior on the LV segmentation output. The system is evaluated on a dataset of 427 patients. Each patient has a pair of captured AP2 and AP4 echo studies, resulting in a total of more than 40,000 echo frames. The mobile system reaches a noticeably high average Dice score of 92% for LV segmentation, an average Euclidean distance error of 2.85 pixels for the detection of anatomical landmarks used in LVEF calculation, and a median absolute error of 6.2% for LVEF estimation compared to the expert cardiologist’s annotations and measurements. The proposed system runs in real time on mobile devices. The experiments show the effectiveness of the proposed system for automatic LVEF estimation by demonstrating an adequate correlation with the cardiologist’s examination.

ABSTRACTThis paper presents an automatic pathology (AutoPath) approach to detect prostatic adenocarcinoma based on morphological analysis of high resolution whole mount (WM) histopathology images of the prostate. In the first stage of the cancer detection algorithm, a pre-screening of cancerous regions is performed at low magnification (5×) based on regional features. In the second stage, we propose a novel technique of labelling individual glands as benign or malignant using gland specific features at high magnification (20×). Two new features, Number of Nuclei Layers and Epithelial Layer Density, are proposed to label individual glands. We validate the approach on 70 WM slides, obtained from 30 patients, and achieve average sensitivity of 90%, specificity of 93% and accuracy of 93%. The main advantage of the approach is that detection of individual malignant gland units, irrespective of neighbouring histology and/or the spatial extent of the cancer, allows a finer annotation of cancer. The AutoPath method performs well on slides with low Gleason grades (3 and 4), but is currently limited in its ability to detect cancer in higher Gleason grades.

We have previously proposed temporal enhanced ultrasound (TeUS) as a new paradigm for tissue characterization. TeUS is based on analyzing a sequence of ultrasound data with deep learning and has been demonstrated to be successful for detection of cancer in ultrasound-guided prostate biopsy. Our aim is to enable the dissemination of this technology to the community for large-scale clinical validation. In this paper, we present a unified software framework demonstrating near-real-time analysis of ultrasound data stream using a deep learning solution. The system integrates ultrasound imaging hardware, visualization and a deep learning back-end to build an accessible, flexible and robust platform. A client–server approach is used in order to run computationally expensive algorithms in parallel. We demonstrate the efficacy of the framework using two applications as case studies. First, we show that prostate cancer detection using near-real-time analysis of RF and B-mode TeUS data and deep learning is feasible. Second, we present real-time segmentation of ultrasound prostate data using an integrated deep learning solution. The system is evaluated for cancer detection accuracy on ultrasound data obtained from a large clinical study with 255 biopsy cores from 157 subjects. It is further assessed with an independent dataset with 21 biopsy targets from six subjects. In the first study, we achieve area under the curve, sensitivity, specificity and accuracy of 0.94, 0.77, 0.94 and 0.92, respectively, for the detection of prostate cancer. In the second study, we achieve an AUC of 0.85. Our results suggest that TeUS-guided biopsy can be potentially effective for the detection of prostate cancer.

In the current standard of care, real-time transrectal ultrasound (TRUS) is commonly used for prostate brachytherapy guidance. As TRUS provides limited soft tissue contrast, segmenting the prostate gland in TRUS images is often challenging and subject to inter-observer and intra-observer variability, especially at the base and apex where the gland boundary is hard to define. Magnetic resonance imaging (MRI) has higher soft tissue contrast allowing the prostate to be contoured easily. In this paper, we aim to show that prostate segmentation in TRUS images informed by MRI priors can improve on prostate segmentation that relies only on TRUS images. First, we compare the TRUS-based prostate segmentation used in the treatment of 598 patients with a high-quality MRI prostate atlas and observe inconsistencies at the apex and base. Second, motivated by this finding, we propose an alternative TRUS segmentation technique that is fully automatic and uses MRI priors. The algorithm uses a convolutional neural network to segment the prostate in TRUS images at mid-gland, where the gland boundary can be clearly seen. It then reconstructs the gland boundary at the apex and base with the aid of a statistical shape model built from an MRI atlas of 78 patients. Compared to the clinical TRUS segmentation, our method achieves similar mid-gland segmentation results in the 598-patient database. For the seven patients who had both TRUS and MRI, our method achieved more accurate segmentation of the base and apex with the MRI segmentation used as ground truth. Our results suggest that utilizing MRI priors in TRUS prostate segmentation could potentially improve the performance at base and apex.

Temporal Enhanced Ultrasound (TeUS) has been proposed as a new paradigm for tissue characterization based on a sequence of ultrasound radio frequency (RF) data. We previously used TeUS to successfully address the problem of prostate cancer detection in the fusion biopsies. In this paper, we use TeUS to address the problem of grading prostate cancer in a clinical study of 197 biopsy cores from 132 patients. Our method involves capturing high-level latent features of TeUS with a deep learning approach followed by distribution learning to cluster aggressive cancer in a biopsy core. In this hypothesis-generating study, we utilize deep learning based feature visualization as a means to obtain insight into the physical phenomenon governing the interaction of temporal ultrasound with tissue. Based on the evidence derived from our feature visualization, and the structure of tissue from digital pathology, we build a simulation framework for studying the physical phenomenon underlying TeUS-based tissue characterization. Results from simulation and feature visualization corroborated with the hypothesis that micro-vibrations of tissue microstructure, captured by low-frequency spectral features of TeUS, can be used for detection of prostate cancer.

We present a method for prostate cancer (PCa) detection using temporal enhanced ultrasound (TeUS) data obtained either from radiofrequency (RF) ultrasound signals or B-mode images. For the first time, we demonstrate that by applying domain adaptation and transfer learning methods, a tissue classification model trained on TeUS RF data (source domain) can be deployed for classification using TeUS B-mode data alone (target domain), where both data are obtained on the same ultrasound scanner. This is a critical step for clinical translation of tissue classification techniques that primarily rely on accessing RF data, since this imaging modality is not readily available on all commercial scanners in clinics. Proof of concept is provided for in vivo characterization of PCa using TeUS B-mode data, where different nonlinear processing filters in the pipeline of the RF to B-mode conversion result in a distribution shift between the two domains. Our in vivo study includes data obtained in MRI-guided targeted procedure for prostate biopsy. We achieve comparable area under the curve using TeUS RF and B-mode data for medium to large cancer tumor sizes in biopsy cores (>4 mm). Our result suggests that the proposed adaptation technique is successful in reducing the divergence between TeUS RF and B-mode data.

Percutaneous spinal needle insertion procedures often require proper identification of the vertebral level to effectively and safely deliver analgesic agents. The current clinical method involves “blind” identification of the vertebral level through manual palpation of the spine, which has only 30% reported accuracy. Therefore, there is a need for better anatomical identification prior to needle insertion. A real-time system was developed to identify the vertebral level from a sequence of ultrasound images, following a clinical imaging protocol. The system uses a deep convolutional neural network (CNN) to classify transverse images of the lower spine. Several existing CNN architectures were implemented, utilizing transfer learning, and compared for adequacy in a real-time system. In the system, the CNN output is processed, using a novel state machine, to automatically identify vertebral levels as the transducer moves up the spine. Additionally, a graphical display was developed and integrated within 3D Slicer. Finally, an augmented reality display, projecting the level onto the patient’s back, was also designed. A small feasibility study $$(n=20)$$ (n=20) evaluated performance. The proposed CNN successfully discriminates ultrasound images of the sacrum, intervertebral gaps, and vertebral bones, achieving 88% 20-fold cross-validation accuracy. Seventeen of 20 test ultrasound scans had successful identification of all vertebral levels, processed at real-time speed (40 frames/s). A machine learning system is presented that successfully identifies lumbar vertebral levels. The small study on human subjects demonstrated real-time performance. A projection-based augmented reality display was used to show the vertebral level directly on the subject adjacent to the puncture site.

Epidural and spinal needle insertions, as well as facet joint denervation and injections are widely performed procedures on the lumbar spine for delivering anesthesia and analgesia. Ultrasound (US)-based approaches have gained popularity for accurate needle placement, as they use a non-ionizing, inexpensive and accessible modality for guiding these procedures. However, due to the inherent difficulties in interpreting spinal US, they yet to become the clinical standard-of-care. A novel statistical shape $$+$$ + pose $$+$$ + scale (s $$+$$ + p $$+$$ + s) model of the lumbar spine is jointly registered to preoperative magnetic resonance (MR) and US images. An instance of the model is created for each modality. The shape and scale model parameters are jointly computed, while the pose parameters are estimated separately for each modality. The proposed method is successfully applied to nine pairs of preoperative clinical MR volumes and their corresponding US images. The results are assessed using the target registration error (TRE) metric in both MR and US domains. The s $$+$$ + p $$+$$ + s model in the proposed joint registration framework results in a mean TRE of 2.62 and 4.20 mm for MR and US images, respectively, on different landmarks. The joint framework benefits from the complementary features in both modalities, leading to significantly smaller TREs compared to a model-to-US registration approach. The s $$+$$ + p $$+$$ + s model also outperforms our previous shape $$+$$ + pose model of the lumbar spine, as separating scale from pose allows to better capture pose and guarantees equally-sized vertebrae in both modalities. Furthermore, the simultaneous visualization of the patient-specific models on the MR and US domains makes it possible for clinicians to better evaluate the local registration accuracy.

Volar percutaneous scaphoid fracture fixation is conventionally performed under fluoroscopy-based guidance, where surgeons need to mentally determine a trajectory for the insertion of the screw and its depth based on a series of 2D projection images. In addition to challenges associated with mapping 2D information to a 3D space, the process involves exposure to ionizing radiation. Three-dimensional ultrasound has been suggested as an alternative imaging tool for this procedure; however, it has not yet been integrated into clinical routine since ultrasound only provides a limited view of the scaphoid and its surrounding anatomy. We propose a registration of a statistical wrist shape + scale + pose model to a preoperative CT and intraoperative ultrasound to derive a patient-specific 3D model for guiding scaphoid fracture fixation. The registered model is then used to determine clinically important intervention parameters, including the screw length and the trajectory of screw insertion in the scaphoid bone. Feasibility experiments are performed using 13 cadaver wrists. In 10 out of 13 cases, the trajectory of screw suggested by the registered model meets all clinically important intervention parameters. Overall, an average 94 % of maximum allowable screw length is obtained based on the measurements from gold standard CT. Also, we obtained an average 92 % successful volar accessibility, which indicates that the trajectory is not obstructed by the surrounding trapezium bone. These promising results indicate that determining clinically important screw insertion parameters for scaphoid fracture fixation is feasible using 3D ultrasound imaging. This suggests the potential of this technology in replacing fluoroscopic guidance for this procedure in future applications.

Facet joint injections and epidural needle insertions are widely used for spine anesthesia. Accurate needle placement is important for effective therapy delivery and avoiding complications arising from damage of soft tissue and nerves. Needle guidance is usually performed by fluoroscopy or palpation, resulting in radiation exposure and multiple needle re-insertions. Several ultrasound (US)-based approaches have been proposed but have not found wide acceptance in clinical routine. This is mainly due to difficulties in interpretation of the complex spinal anatomy in US, which leads to clinicians’ lack of confidence in relying only on information derived from US for needle guidance. We introduce a multimodal joint registration technique that takes advantage of easy-to-interpret preprocedure computed topography (CT) scans of the lumbar spine to concurrently register a shape+pose model to the intraprocedure 3D US. Common shape coefficients are assumed between two modalities, while pose coefficients are specific to each modality. The joint method was evaluated on patient data consisting of ten pairs of US and CT scans of the lumbar spine. It was successfully applied in all cases and yielded an RMS shape error of 2.1 mm compared to the CT ground truth. The joint registration technique was compared to a previously proposed method of statistical model to US registration Rasoulian et al. (Information processing in computer-assisted interventions. Springer, Berlin, pp 51–60, 2013). The joint framework improved registration accuracy to US in 7 out of 17 visible vertebrae, belonging to four patients. In the remaining cases, the two methods were equally accurate. The joint registration allows visualization and augmentation of important anatomy in both the US and CT domain and improves the registration accuracy in both modalities. Observing the patient-specific model in the CT domain allows the clinicians to assess the local registration accuracy qualitatively, which is likely to increase their confidence in using the US model for deriving needle guidance decisions.

This paper presents the results of a large study involving fusion prostate biopsies to demonstrate that temporal ultrasound can be used to accurately classify tissue labels identified in multi-parametric magnetic resonance imaging (mp-MRI) as suspicious for cancer. We use deep learning to analyze temporal ultrasound data obtained from 255 cancer foci identified in mp-MRI. Each target is sampled in axial and sagittal planes. A deep belief network is trained to automatically learn the high-level latent features of temporal ultrasound data. A support vector machine classifier is then applied to differentiate cancerous versus benign tissue, verified by histopathology. Data from 32 targets are used for the training, while the remaining 223 targets are used for testing. Our results indicate that the distance between the biopsy target and the prostate boundary, and the agreement between axial and sagittal histopathology of each target impact the classification accuracy. In 84 test cores that are 5 mm or farther to the prostate boundary, and have consistent pathology outcomes in axial and sagittal biopsy planes, we achieve an area under the curve of 0.80. In contrast, all of these targets were labeled as moderately suspicious in mp-MR. Using temporal ultrasound data in a fusion prostate biopsy study, we achieved a high classification accuracy specifically for moderately scored mp-MRI targets. These targets are clinically common and contribute to the high false-positive rates associated with mp-MRI for prostate cancer detection. Temporal ultrasound data combined with mp-MRI have the potential to reduce the number of unnecessary biopsies in fusion biopsy settings.

Spinal needle injections are widely applied to alleviate back pain and for anesthesia. Current treatment is performed either blindly with palpation or using fluoroscopy or computed tomography (CT). Both fluoroscopy and CT guidance expose patients to ionizing radiation. Ultrasound (US) guidance for spinal needle procedures is becoming more prevalent as an alternative. It is challenging to use US as the sole imaging modality for intraoperative guidance of spine needle injections due to the acoustic shadows created by the bony structures of the vertebra that limit visibility of the target areas for injection. We propose registration of CT and the US images to augment anatomical visualization for the clinician during spinal interventions guided by US. The proposed method involves automatic global and multi-vertebrae registration to find the closest alignment between CT and US data. This is performed by maximizing the similarity between the two modalities using voxel intensity information as well as features extracted from the input volumes. In our method, the lumbar spine is first globally aligned between the CT and US data using intensity-based registration followed by point-based registration. To account for possible curvature change of the spine between the CT and US volumes, a multi-vertebrae registration step is also performed. Springs are used to constrain the movement of the individually transformed vertebrae to ensure the optimal alignment is a pose of the lumbar spine that is physically possible. Evaluation of the algorithm is performed on 10 clinical patient datasets. The registration approach was able to align CT and US datasets from initial misalignments of up to 25 mm, with a mean TRE of 1.37 mm. These results suggest that the proposed approach has the potential to offer a sufficiently accurate registration between clinical CT and US data.