Search

In general, deep neural network (DNN) pruning methods fall into two categories: 1) weight-based deterministic constraints and 2) probabilistic frameworks. While each approach has its merits and limitations, there are a set of common practical issues such as trial-and-error to analyze sensitivity and hyper-parameters to prune DNNs, which plague them both. In this work, we propose a new single-shot, fully automated pruning algorithm called slimming neural networks using adaptive connectivity scores (SNACS). Our proposed approach combines a probabilistic pruning framework with constraints on the underlying weight matrices, via a novel connectivity measure, at multiple levels to capitalize on the strengths of both approaches while solving their deficiencies. In SNACS, we propose a fast hash-based estimator of adaptive conditional mutual information (ACMI), that uses a weight-based scaling criterion, to evaluate the connectivity between filters and prune unimportant ones. To automatically determine the limit up to which a layer can be pruned, we propose a set of operating constraints that jointly define the upper pruning percentage limits across all the layers in a deep network. Finally, we define a novel sensitivity criterion for filters that measures the strength of their contributions to the succeeding layer and highlights critical filters that need to be completely protected from pruning. Through our experimental validation, we show that SNACS is faster by over 17× the nearest comparable method and is the state-of-the-art single-shot pruning method across four standard Dataset-DNN pruning benchmarks: CIFAR10-VGG16, CIFAR10-ResNet56, CIFAR10-MobileNetv2, and ILSVRC2012-ResNet50.

Purpose: Articulated hand pose tracking is an under-explored problem that carries the potential for use in an extensive number of applications, especially in the medical domain. With a robust and accurate tracking system on surgical videos, the motion dynamics and movement patterns of the hands can be captured and analyzed for many rich tasks., Methods: In this work, we propose a novel hand pose estimation model, CondPose, which improves detection and tracking accuracy by incorporating a pose prior into its prediction. We show improvements over state-of-the-art methods which provide frame-wise independent predictions, by following a temporally guided approach that effectively leverages past predictions., Results: We collect Surgical Hands, the first dataset that provides multi-instance articulated hand pose annotations for videos. Our dataset provides over 8.1k annotated hand poses from publicly available surgical videos and bounding boxes, pose annotations, and tracking IDs to enable multi-instance tracking. When evaluated on Surgical Hands, we show our method outperforms the state-of-the-art approach using mean Average Precision, to measure pose estimation accuracy, and Multiple Object Tracking Accuracy, to assess pose tracking performance., Conclusion: In comparison to a frame-wise independent strategy, we show greater performance in detecting and tracking hand poses and more substantial impact on localization accuracy. This has positive implications in generating more accurate representations of hands in the scene to be used for targeted downstream tasks., (© 2022. The Author(s).)

Background: Of the 150,000 patients annually undergoing coronary artery bypass grafting, 35% develop complications that increase mortality 5 fold and expenditure by 50%. Differences in patient risk and operative approach explain only 2% of hospital variations in some complications. The intraoperative phase remains understudied as a source of variation, despite its complexity and amenability to improvement., Objective: The objectives of this study are to (1) investigate the relationship between peer assessments of intraoperative technical skills and nontechnical practices with risk-adjusted complication rates and (2) evaluate the feasibility of using computer-based metrics to automate the assessment of important intraoperative technical skills and nontechnical practices., Methods: This multicenter study will use video recording, established peer assessment tools, electronic health record data, registry data, and a high-dimensional computer vision approach to (1) investigate the relationship between peer assessments of surgeon technical skills and variability in risk-adjusted patient adverse events; (2) investigate the relationship between peer assessments of intraoperative team-based nontechnical practices and variability in risk-adjusted patient adverse events; and (3) use quantitative and qualitative methods to explore the feasibility of using objective, data-driven, computer-based assessments to automate the measurement of important intraoperative determinants of risk-adjusted patient adverse events., Results: The project has been funded by the National Heart, Lung and Blood Institute in 2019 (R01HL146619). Preliminary Institutional Review Board review has been completed at the University of Michigan by the Institutional Review Boards of the University of Michigan Medical School., Conclusions: We anticipate that this project will substantially increase our ability to assess determinants of variation in complication rates by specifically studying a surgeon's technical skills and operating room team member nontechnical practices. These findings may provide effective targets for future trials or quality improvement initiatives to enhance the quality and safety of cardiac surgical patient care., International Registered Report Identifier (irrid): PRR1-10.2196/22536., (©Donald Likosky, Steven J Yule, Michael R Mathis, Roger D Dias, Jason J Corso, Min Zhang, Sarah L Krein, Matthew D Caldwell, Nathan Louis, Allison M Janda, Nirav J Shah, Francis D Pagani, Korana Stakich-Alpirez, Milisa M Manojlovich. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 08.01.2021.)

In this work, we explore video frame inpainting, a task that lies at the intersection of general video inpainting, frame interpolation, and video prediction. Although our problem can be addressed by applying methods from other video interpolation or extrapolation tasks, doing so fails to leverage the additional context information that our problem provides. To this end, we devise a method specifically designed for video frame inpainting that is composed of two modules: a bidirectional video prediction module and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, each conditioned on the preceding and following frames respectively, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions by using time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces smoother and more accurate results than state-of-the-art methods for general video inpainting, frame interpolation, and video prediction.

Various perceptual domains have underlying compositional semantics that are rarely captured in current models. We suspect this is because directly learning the compositional structure has evaded these models. Yet, the compositional structure of a given domain can be grounded in a separate domain thereby simplifying its learning. To that end, we propose a new approach to modeling bimodal perceptual domains that explicitly relates distinct projections across each modality and then jointly learns a bimodal sparse representation. The resulting model enables compositionality across these distinct projections and hence can generalize to unobserved percepts spanned by this compositional basis. For example, our model can be trained on red triangles and blue squares; yet, implicitly will also have learned red squares and blue triangles. The structure of the projections and hence the compositional basis is learned automatically; no assumption is made on the ordering of the compositional elements in either modality. Although our modeling paradigm is general, we explicitly focus on a tabletop building-blocks setting. To test our model, we have acquired a new bimodal dataset comprising images and spoken utterances of colored shapes (blocks) in the tabletop setting. Our experiments demonstrate the benefits of explicitly leveraging compositionality in both quantitative and human evaluation studies.

Video understanding of robot-assisted surgery (RAS) videos is an active research area. Modeling the gestures and skill level of surgeons presents an interesting problem. The insights drawn may be applied in effective skill acquisition, objective skill assessment, real-time feedback, and human-robot collaborative surgeries. We propose a solution to the tool detection and localization open problem in RAS video understanding, using a strictly computer vision approach and the recent advances of deep learning. We propose an architecture using multimodal convolutional neural networks for fast detection and localization of tools in RAS videos. To the best of our knowledge, this approach will be the first to incorporate deep neural networks for tool detection and localization in RAS videos. Our architecture applies a region proposal network (RPN) and a multimodal two stream convolutional network for object detection to jointly predict objectness and localization on a fusion of image and temporal motion cues. Our results with an average precision of 91% and a mean computation time of 0.1 s per test frame detection indicate that our study is superior to conventionally used methods for medical imaging while also emphasizing the benefits of using RPN for precision and efficiency. We also introduce a new data set, ATLAS Dione, for RAS video understanding. Our data set provides video data of ten surgeons from Roswell Park Cancer Institute, Buffalo, NY, USA, performing six different surgical tasks on the daVinci Surgical System (dVSS) with annotations of robotic tools per frame.

Semi-supervised clustering seeks to augment traditional clustering methods by incorporating side information provided via human expertise in order to increase the semantic meaningfulness of the resulting clusters. However, most current methods are passive in the sense that the side information is provided beforehand and selected randomly. This may require a large number of constraints, some of which could be redundant, unnecessary, or even detrimental to the clustering results. Thus in order to scale such semi-supervised algorithms to larger problems it is desirable to pursue an active clustering method-i.e., an algorithm that maximizes the effectiveness of the available human labor by only requesting human input where it will have the greatest impact. Here, we propose a novel online framework for active semi-supervised spectral clustering that selects pairwise constraints as clustering proceeds, based on the principle of uncertainty reduction. Using a first-order Taylor expansion, we decompose the expected uncertainty reduction problem into a gradient and a step-scale, computed via an application of matrix perturbation theory and cluster-assignment entropy, respectively. The resulting model is used to estimate the uncertainty reduction potential of each sample in the dataset. We then present the human user with pairwise queries with respect to only the best candidate sample. We evaluate our method using three different image datasets (faces, leaves and dogs), a set of common UCI machine learning datasets and a gene dataset. The results validate our decomposition formulation and show that our method is consistently superior to existing state-of-the-art techniques, as well as being robust to noise and to unknown numbers of clusters.

The rupture of Intracranial Aneurysms is the most severe form of stroke with high rates of mortality and disability. One of its primary treatments is to use stent or Flow Diverter to divert the blood flow away from the IA in a minimal invasive manner. To optimize such treatments, it is desirable to provide an automatic tool for virtual stenting before its actual implantation. In this paper, we propose a novel method, called ball-sweeping, for rapid virtual stenting. Our method sweeps a maximum inscribed sphere through the aneurysmal region of the vessel and directly generates a stent surface touching the vessel wall without needing to iteratively grow a deformable stent surface. Our resulting stent mesh has guaranteed smoothness and variable pore density to achieve an enhanced occlusion performance. Comparing to existing methods, our technique is computationally much more efficient.

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients-manually annotated by up to four raters-and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

Purpose: Detection and segmentation of a brain tumor such as glioblastoma multiforme (GBM) in magnetic resonance (MR) images are often challenging due to its intrinsically heterogeneous signal characteristics. A robust segmentation method for brain tumor MRI scans was developed and tested., Methods: Simple thresholds and statistical methods are unable to adequately segment the various elements of the GBM, such as local contrast enhancement, necrosis, and edema. Most voxel-based methods cannot achieve satisfactory results in larger data sets, and the methods based on generative or discriminative models have intrinsic limitations during application, such as small sample set learning and transfer. A new method was developed to overcome these challenges. Multimodal MR images are segmented into superpixels using algorithms to alleviate the sampling issue and to improve the sample representativeness. Next, features were extracted from the superpixels using multi-level Gabor wavelet filters. Based on the features, a support vector machine (SVM) model and an affinity metric model for tumors were trained to overcome the limitations of previous generative models. Based on the output of the SVM and spatial affinity models, conditional random fields theory was applied to segment the tumor in a maximum a posteriori fashion given the smoothness prior defined by our affinity model. Finally, labeling noise was removed using "structural knowledge" such as the symmetrical and continuous characteristics of the tumor in spatial domain., Results: The system was evaluated with 20 GBM cases and the BraTS challenge data set. Dice coefficients were computed, and the results were highly consistent with those reported by Zikic et al. (MICCAI 2012, Lecture notes in computer science. vol 7512, pp 369-376, 2012)., Conclusion: A brain tumor segmentation method using model-aware affinity demonstrates comparable performance with other state-of-the art algorithms.

Quantifying volume and growth of a brain tumor is a primary prognostic measure and hence has received much attention in the medical imaging community. Most methods have sought a fully automatic segmentation, but the variability in shape and appearance of brain tumor has limited their success and further adoption in the clinic. In reaction, we present a semi-automatic brain tumor segmentation framework for multi-channel magnetic resonance (MR) images. This framework does not require prior model construction and only requires manual labels on one automatically selected slice. All other slices are labeled by an iterative multi-label Markov random field optimization with hard constraints. Structural trajectories-the medical image analog to optical flow and 3D image over-segmentation are used to capture pixel correspondences between consecutive slices for pixel labeling. We show robustness and effectiveness through an evaluation on the 2012 MICCAI BRATS Challenge Dataset; our results indicate superior performance to baselines and demonstrate the utility of the constrained MRF formulation.

Purpose: A CAD system for lumbar disc degeneration and herniation based on clinical MR images can aid diagnostic decision-making provided the method is robust, efficient, and accurate., Material and Methods: A Bayesian-based classifier with a Gibbs distribution was designed and implemented for diagnosing lumbar disc herniation. Each disc is segmented with a gradient vector flow active contour model (GVF-snake) to extract shape features that feed a classifier. The GVF-snake is automatically initialized with an inner boundary of the disc initiated by a point inside the disc. This point is automatically generated by our previous work on lumbar disc labeling. The classifier operates on clinical T2-SPIR weighted sagittal MRI of the lumbar area. The classifier is applied slice-by-slice to tag herniated discs if they are classified as herniated in any of the 2D slices. This technique detects all visible herniated discs regardless of their location (lateral or central). The gold standard for the ground truth was obtained from collaborating radiologists by analyzing the clinical diagnosis report for each case., Results: An average 92.5% herniation diagnosis accuracy was observed in a cross-validation experiment with 65 clinical cases. The random leave-out experiment runs ten rounds; in each round, 35 cases were used for testing and the remaining 30 cases were used for training., Conclusion: An automatic robust disk herniation diagnostic method for clinical lumbar MRI was developed and tested. The method is intended for clinical practice to support reliable decision-making.

Backbone anatomical structure detection and labeling is a necessary step for various analysis tasks of the vertebral column. Appearance, shape and geometry measurements are necessary for abnormality detection locally at each disc and vertebrae (such as herniation) as well as globally for the whole spine (such as spinal scoliosis). We propose a two-level probabilistic model for the localization of discs from clinical magnetic resonance imaging (MRI) data that captures both pixel- and object-level features. Using a Gibbs distribution, we model appearance and spatial information at the pixel level, and at the object level, we model the spatial distribution of the discs and the relative distances between them. We use generalized expectation-maximization for optimization, which achieves efficient convergence of disc labels. Our two-level model allows the assumption of conditional independence at the pixel-level to enhance efficiency while maintaining robustness. We use a dataset that contains 105 MRI clinical normal and abnormal cases for the lumbar area. We thoroughly test our model and achieve encouraging results on normal and abnormal cases.

The use of cone beam computed tomography (CBCT) is growing in the clinical arena due to its ability to provide 3D information during interventions, its high diagnostic quality (sub-millimeter resolution), and its short scanning times (60 s). In many situations, the short scanning time of CBCT is followed by a time-consuming 3D reconstruction. The standard reconstruction algorithm for CBCT data is the filtered backprojection, which for a volume of size 256(3) takes up to 25 min on a standard system. Recent developments in the area of Graphic Processing Units (GPUs) make it possible to have access to high-performance computing solutions at a low cost, allowing their use in many scientific problems. We have implemented an algorithm for 3D reconstruction of CBCT data using the Compute Unified Device Architecture (CUDA) provided by NVIDIA (NVIDIA Corporation, Santa Clara, California), which was executed on a NVIDIA GeForce GTX 280. Our implementation results in improved reconstruction times from minutes, and perhaps hours, to a matter of seconds, while also giving the clinician the ability to view 3D volumetric data at higher resolutions. We evaluated our implementation on ten clinical data sets and one phantom data set to observe if differences occur between CPU and GPU-based reconstructions. By using our approach, the computation time for 256(3) is reduced from 25 min on the CPU to 3.2 s on the GPU. The GPU reconstruction time for 512(3) volumes is 8.5 s., (Copyright 2009 Elsevier Ireland Ltd. All rights reserved.)

Purpose: Detection of abnormal discs from clinical T2-weighted MR Images. This aids the radiologist as well as subsequent CAD methods in focusing only on abnormal discs for further diagnosis. Furthermore, it gives a degree of confidence about the abnormality of the intervertebral discs that helps the radiologist in making his decision., Materials and Methods: We propose a probabilistic classifier for the detection of abnormality of intervertebral discs. We use three features to label abnormal discs that include appearance, location, and context. We model the abnormal disc appearance with a Gaussian model, the location with a 2D Gaussian model, and the context with a Gaussian model for the distance between abnormal discs. We infer on the middle slice of the T2-weighted MRI volume for each case. These MRI scans are specific for the lumbar area. We obtain our gold standard for the ground truth from our collaborating radiologist group by having the clinical diagnosis report for each case., Results: We achieve over 91% abnormality detection accuracy in a cross-validation experiment with 80 clinical cases. The experiment runs ten rounds; in each round, we randomly leave 30 cases out for testing and we use the other 50 cases for training., Conclusion: We achieve high accuracy for detection of abnormal discs using our proposed model that incorporates disc appearance, location, and context. We show the extendability of our proposed model to subsequent diagnosis tasks specific to each intervertebral disc abnormality such as desiccation and herniation.

To predict the three dimensional structure of proteins, many computational methods sample the conformational space, generating a large number of candidate structures. Subsequently, such methods rank the generated structures using a variety of model quality assessment programs in order to obtain a small set of structures that are most likely to resemble the unknown experimentally determined structure. Model quality assessment programs suffer from two main limitations: (i) the rank-one structure is not always the best predicted structure; in other words, the best predicted structure could be ranked as the 10th structure (ii) no single assessment method can correctly rank the predicted structures for all target proteins. However, because often at least some of the methods achieve a good ranking, a model quality assessment method that is based on a consensus of a number of model quality assessment methods is likely to perform better. We have devised the STPdata algorithm, a consensus method based on five model quality assessment programs. We have applied it to build an on-line "custom-trained" hierarchy of general linear models to select and rank the best predicted structures. By "custom-trained", we mean for each target protein the STPdata algorithm trains a unique model on data related to the input target protein. To evaluate our method we participated in CASP8 as human predictors. In CASP8, the STPdata algorithm has trained 128 hierarchical models for each of the 128 target proteins. Based on the official results of CASP8 our method outperformed the best server by 6% and won the fourth position among human predictors. Our CASP results are purely based on computational methods without any human intervention.

Repeatable, quantitative assessment of intervertebral disc pathology requires accurate localization and labeling of the lumbar region discs. To that end, we propose a two-level probabilistic model for such disc localization and labeling. Our model integrates both pixel-level information, such as appearance, and object-level information, such as relative location. Utilizing both levels of information adds robustness to the ambiguous disc intensity signature and high structure variation. Yet, we are able to do efficient (and convergent) localization and labeling with generalized expectation-maximization. We present accurate results on 20 normal cases (96%) and a promising extension to a pathology case.

If imaging is to serve as a valid biomarker in the assessment of the response of cancer to therapies, a reproducible and predictive radiologic metric is required. A biomarker is an indicator of a biological property that can be used to measure the progress of disease. While current size-based, quantitative techniques provide numerical representations of tumors, they are not necessarily indicative of disease progression for advanced cancers. In this paper, we present an end-to-end process to explore the use of other image-based features especially statistical textural features for cancer change detection. We exploit the earth mover's distance metric for measuring the change in the tumor burden over a period, between the time the baseline scans were taken, and the time the therapy response scans were taken. The time-to-progression (TTP) of the disease is our known patient outcome. We analyze the correlations between TTP and our change measurements and discover that the local texture energy feature is most predictive of disease progression, more so than the tumor burden size on which current quantitative measures are made.

We propose a novel algorithm called graph-shifts for performing image segmentation and labeling. This algorithm makes use of a dynamic hierarchical representation of the image. This representation allows each iteration of the algorithm to make both small and large changes in the segmentation, similar to PDE and split-and-merge methods, respectively. In particular, at each iteration we are able to rapidly compute and select the optimal change to be performed. We apply graph-shifts to the task of segmenting sub-cortical brain structures. First we formalize this task as energy function minimization where the energy terms are learned from a training set of labeled images. Then we apply the graphshifts algorithm. We show that the labeling results are comparable in quantitative accuracy to other approaches but are obtained considerably faster: by orders of magnitude (roughly one minute). We also quantitatively demonstrate robustness to initialization and avoidance of local minima in which conventional boundary PDE methods fall.

We propose an extended graph-shifts algorithm for image segmentation and labeling. This algorithm performs energy minimization by manipulating a dynamic hierarchical representation of the image. It consists of a set of moves occurring at different levels of the hierarchy where the types of move, and the level of the hierarchy, are chosen automatically so as to maximally decrease the energy. Extended graph-shifts can be applied to a broad range of problems in medical imaging. In this paper, we apply extended graph-shifts to the detection of pathological brain structures: (i) segmentation of brain tumors, and (ii) detection of multiple sclerosis lesions. The energy terms in these tasks are learned from training data by statistical learning algorithms. We demonstrate accurate results, precision and recall in the order of 93%, and also show that the algorithm is computationally efficient, segmenting a full 3D volume in about one minute.

We present a new method for automatic segmentation of heterogeneous image data, which is very common in medical image analysis. The main contribution of the paper is a mathematical formulation for incorporating soft model assignments into the calculation of affinities, which are traditionally model free. We integrate the resulting model-aware affinities into the multilevel segmentation by weighted aggregation algorithm. We apply the technique to the task of detecting and segmenting brain tumor and edema in multimodal MR volumes. Our results indicate the benefit of incorporating model-aware affinities into the segmentation process for the difficult case of brain tumor.