Search

Medical instrument detection is essential for computer-assisted interventions, since it facilitates clinicians to find instruments efficiently with a better interpretation, thereby improving clinical outcomes. This article reviews image-based medical instrument detection methods for ultrasound-guided (US-guided) operations. Literature is selected based on an exhaustive search in different sources, including Google Scholar, PubMed, and Scopus. We first discuss the key clinical applications of medical instrument detection in the US, including delivering regional anesthesia, biopsy taking, prostate brachytherapy, and catheterization. Then, we present a comprehensive review of instrument detection methodologies, including non-machine-learning and machine-learning methods. The conventional non-machine-learning methods were extensively studied before the era of machine learning methods. The principal issues and potential research directions for future studies are summarized for the computer-assisted intervention community. In conclusion, although promising results have been obtained by the current (non-) machine learning methods for different clinical applications, thorough clinical validations are still required.

Computer-Aided Diagnosis (CADx) systems for characterization of Narrow-Band Imaging (NBI) videos of suspected lesions in Barrett’s Esophagus (BE) can assist endoscopists during endoscopic surveillance. The real clinical value and application of such CADx systems lies in real-time analysis of endoscopic videos inside the endoscopy suite, placing demands on robustness in decision making and insightful classification matching with the clinical opinions. In this paper, we propose a lightweight int8-based quantized neural network architecture supplemented with an efficient stability function on the output for real-time classification of NBI videos. The proposed int8-architecture has low-memory footprint (4.8 MB), enabling operation on a range of edge devices and even existing endoscopy equipment. Moreover, the stability function ensures robust inclusion of temporal information from the video to provide a continuously stable video classification. The algorithm is trained, validated and tested with a total of 3,799 images and 284 videos of in total 598 patients, collected from 7 international centers. Several stability functions are experimented with, some of them being clinically inspired by weighing low-confidence predictions. For the detection of early BE neoplasia, the proposed algorithm achieves a performance of 92.8% accuracy, 95.7% sensitivity, and 91.4% specificity, while only 5.6% of the videos are without a final video classification. This work shows a robust, lightweight and effective deep learning-based CADx system for accurate automated real-time endoscopic video analysis, suited for embedding in endoscopy clinical practice.

Learning anticipation in Multi-Agent Reinforcement Learning (MARL) is a reasoning paradigm where agents anticipate the learning steps of other agents to improve cooperation among themselves. As MARL uses gradient-based optimization, learning anticipation requires using Higher-Order Gradients (HOG), with so-called HOG methods. Existing HOG methods are based on policy parameter anticipation, i.e., agents anticipate the changes in policy parameters of other agents. Currently, however, these existing HOG methods have only been applied to differentiable games or games with small state spaces. In this work, we demonstrate that in the case of non-differentiable games with large state spaces, existing HOG methods do not perform well and are inefficient due to their inherent limitations related to policy parameter anticipation and multiple sampling stages. To overcome these problems, we propose Off-Policy Action Anticipation (OffPA2), a novel framework that approaches learning anticipation through action anticipation, i.e., agents anticipate the changes in actions of other agents, via off-policy sampling. We theoretically analyze our proposed OffPA2 and employ it to develop multiple HOG methods that are applicable to non-differentiable games with large state spaces. We conduct a large set of experiments and illustrate that our proposed HOG methods outperform the existing ones regarding efficiency and performance.

Purpose: Mask-immobilized stereotactic radiosurgery (SRS) using a gating window is an emerging technology. However, the amount of intracranial tumor motion that can be tolerated during treatment while satisfying clinical dosimetric goals is unknown. The purpose of this study was to quantify the sensitivity of target dose to tumor motion. Methods and Materials: In clinical SRS plans, where a nose marker was tracked as surrogate for target motion, translational and rotational target movements were simulated using nose-marker displacements of ±0.5 mm, ±1.0 mm, or ±1.5 mm. The effect on minimum dose to 99% of the target (D99) and percent target coverage by prescription dose was quantified using mixed-effect modeling with variables: displacement, target volume, and location. Results: The effect on dose metrics is statistically larger for translational displacements compared with rotational displacements, and the effect of pitch rotations is statistically larger compared with yaw rotations. The mixed-effect model for translations showed that displacement and target volume are statistically significant variables, for rotation the variable target distance to rotation axis is additionally significant. For mean target volume (12.6 cc) and translational nose-marker displacements of 0.5 mm, 1.0 mm, and 1.5 mm, D99 decreased by 2.2%, 7.1%, and 13.0%, and coverage by 0.4%, 1.8%, and 4.4%, respectively. For mean target volume, mean distance midpoint-target to pitch axis (7.6cm), and rotational nose-marker displacement of 0.5 mm, 1.0 mm, and 1.5 mm, D99 decreased by 1.0%, 3.6%, and 6.9%, and coverage by 0.2%, 0.8%, and 1.9%, respectively. For rotational yaw axis displacement, mean distance midpoint-target axis (4.2cm), D99 decreased by 0.3%, 1.2%, and 2.5%, and coverage by 0.1%, 0.2%, and 0.5%, respectively. Conclusions: Simulated target displacements showed that sensitivity of tumor dose to motion depends on both target volume and target location. Suggesting that patient- and target-specific thresholds may be implemented for optimizing the balance between dosimetric plan accuracy and treatment prolongation caused by out-of-tolerance motion.

Learning anticipation is a reasoning paradigm in multi-agent reinforcement learning, where agents, during learning, consider the anticipated learning of other agents. There has been substantial research into the role of learning anticipation in improving cooperation among self-interested agents in general-sum games. Two primary examples are Learning with Opponent-Learning Awareness (LOLA), which anticipates and shapes the opponent's learning process to ensure cooperation among self-interested agents in various games such as iterated prisoner's dilemma, and Look-Ahead (LA), which uses learning anticipation to guarantee convergence in games with cyclic behaviors. So far, the effectiveness of applying learning anticipation to fully-cooperative games has not been explored. In this study, we aim to research the influence of learning anticipation on coordination among common-interested agents. We first illustrate that both LOLA and LA, when applied to fully-cooperative games, degrade coordination among agents, causing worst-case outcomes. Subsequently, to overcome this miscoordination behavior, we propose Hierarchical Learning Anticipation (HLA), where agents anticipate the learning of other agents in a hierarchical fashion. Specifically, HLA assigns agents to several hierarchy levels to properly regulate their reasonings. Our theoretical and empirical findings confirm that HLA can significantly improve coordination among common-interested agents in fully-cooperative normal-form games. With HLA, to the best of our knowledge, we are the first to unlock the benefits of learning anticipation for fully-cooperative games., Comment: AAMAS 2023 Workshop on Optimization and Learning in Multi-Agent Systems

Understanding procedural actions is important, as it can be used to automatically analyze the execution of a procedure and provide assistance to users by warning for potential mistakes or forgotten steps. However, current approaches require a rigid, step-by-step execution order, laborious and impractical datasets. Furthermore, they are unreliable to variations in viewpoint, or measure the performance of actions rather than the actual completion of actions. To address these limitations and stimulate research in this field, this work proposes the novel task of procedure state recognition (PSR) together with a set of evaluation metrics.

Introduction: Anastomotic leakage after gastrointestinal surgery has a high impact on patient's quality of life and its origin is associated with inadequate perfusion. Imaging photoplethysmography (iPPG) is a noninvasive imaging technique that measures blood-volume changes in the microvascular tissue bed and detects changes in tissue perfusion. Materials and methods: Intraoperative iPPG imaging was performed in 29 patients undergoing an open segment resection of the small intestine or colon. During each surgery, imaging was performed on fully perfused (true positives) and ischemic intestines (true negatives) and the anastomosis (unknowns). Imaging consisted of a 30-s video from which perfusion maps were extracted, providing detailed information about blood flow within the intestine microvasculature. To detect the predictive capabilities of iPPG, true positive and true negative perfusion conditions were used to develop two different perfusion classification methods. Results: iPPG-derived perfusion parameters were highly correlated with perfusion—perfused or ischemic—in intestinal tissues. A perfusion confidence map distinguished perfused and ischemic intestinal tissues with 96% sensitivity and 86% specificity. Anastomosis images were scored as adequately perfused in 86% of cases and 14% inconclusive. The cubic-Support Vector Machine achieved 90.9% accuracy and an area under the curve of 96%. No anastomosis-related postoperative complications were encountered in this study. Conclusions: This study shows that noninvasive intraoperative iPPG is suitable for the objective assessment of small intestine and colon anastomotic perfusion. In addition, two perfusion classification methods were developed, providing the first step in an intestinal perfusion prediction model.

Finding potential driving paths on unstructured roads is a challenging problem for autonomous driving and robotics applications. Although the rise of autonomous driving has resulted in massive public datasets, most of these datasets focus on urban environments and feature almost exclusively paved roads. To circumvent the problem of limited public datasets of unpaved roads, we combine seven public vehicle-mounted-camera datasets with a very small private dataset and train a neural network to achieve accurate road segmentation on almost any type of road. This trained network vastly outperforms networks trained on individual datasets when validated on our unpaved road datasets, with only a minor performance reduction on the highly challenging public WildDash dataset, which is mostly urban. Finally, we develop an algorithm to robustly transform these road segmentations to road centerlines, used to automatically control a vehicle-mounted PTZ camera.

The field of Out-of-Distribution (OOD) detection aims to separate OOD data from in-distribution (ID) data in order to make safe predictions. With the increasing application of Convolutional Neural Networks (CNNs) in sensitive environments such as autonomous driving and security, this field is bound to become indispensable in the future. Although the OOD detection field has made some progress in recent years, a fundamental understanding of the underlying phenomena enabling the separation of datasets remains lacking. We find that the OOD detection relies heavily on the covariate shift of the data and not so much on the semantic shift, i.e. a CNN does not carry explicit semantic information and relies solely on differences in features. Although these features can be affected by the underlying semantics, this relation does not seem strong enough to rely on. Conversely, we found that since the CNN training setup determines what features are learned, that it is an important factor for the OOD performance. We found that variations in the model training can lead to an increase or decrease in the OOD detection performance. Through this insight, we obtain an increase in OOD detection performance on the common OOD detection benchmarks by changing the training procedure and using the simple Maximum Softmax Probability (MSP) model introduced by (Hendrycks and Gimpel, 2016). We hope to inspire others to look more closely into the fundamental principles underlying the separation of two datasets. The code for reproducing our results can be found at https://github.com/SimonMariani/OOD- detection.

For autonomous vehicles and mobile robots to safely operate in the real world, i.e., the wild, scene understanding models should perform well in the many different scenarios that can be encountered. In reality, these scenarios are not all represented in the model’s training data, leading to poor performance. To tackle this, current training strategies attempt to either exploit additional unlabeled data with unsupervised domain adaptation (UDA), or to reduce overfitting using the limited available labeled data with domain generalization (DG). However, it is not clear from current literature which of these methods allows for better generalization to unseen data from the wild. Therefore, in this work, we present an evaluation framework in which the generalization capabilities of state-of-the-art UDA and DG methods can be compared fairly. From this evaluation, we find that UDA methods, which leverage unlabeled data, outperform DG methods in terms of generalization, and can deliver similar performance on unseen data as fully-supervised training methods that require all data to be labeled. We show that semantic segmentation performance can be increased up to 30% for a priori unknown data without using any extra labeled data.

Unified panoptic segmentation methods are achieving state-of-the-art results on several datasets. To achieve these results on high-resolution datasets, these methods apply crop-based training. In this work, we find that, although crop-based training is advantageous in general, it also has a harmful side-effect. Specifically, it limits the ability of unified networks to discriminate between large object instances, causing them to make predictions that are confused between multiple instances. To solve this, we propose Intra-Batch Supervision (IBS), which improves a network's ability to discriminate between instances by introducing additional supervision using multiple images from the same batch. We show that, with our IBS, we successfully address the confusion problem and consistently improve the performance of unified networks. For the high-resolution Cityscapes and Mapillary Vistas datasets, we achieve improvements of up to +2.5 on the Panoptic Quality for thing classes, and even more considerable gains of up to +5.8 on both the pixel accuracy and pixel precision, which we identify as better metrics to capture the confusion problem., WACV 2023. Project page and code: https://ddegeus.github.io/intra-batch-supervision/

Learning to predict the trajectories of pedestrians is essential for improving safety and efficiency of mobile robots. The prediction is challenging since the robot needs to operate in multiple environments in which the motion patterns of pedestrians are different between environments. Existing pedestrian trajectory prediction models heavily rely on the availability of representative data samples during training. In the presence of additional training data from a new environment, these models must be retrained on all datasets to avoid catastrophic forgetting of the knowledge obtained from the already supported environments. In this paper, we address this catastrophic forgetting problem in the context of learning to predict the trajectories of pedestrians. We propose a pseudo-rehearsal approach based on a novel Generative Replay (GR) model, referred to as Social-GR . The proposed method is consistent with crowd motion patterns and is free of any explicit reference to past experiences. To demonstrate the problem of catastrophic forgetting and evaluate our solution, we develop the Continual Trajectory Prediction Benchmark , which consists of four tasks, each representing a real-world pedestrian trajectory dataset from a different environment. By conducting several experiments, we show that our proposed Social-GR approach significantly outperforms other continual learning methods that depend on explicit experience replay, including the state-of-the-art conditional-GR model. We further illustrate the robustness of our proposed approach to mitigating catastrophic forgetting by switching the order of environments and employing a more complex prediction model.

Brain-inspired event-based processors have attracted considerable attention for edge deployment because of their ability to efficiently process Convolutional Neural Networks (CNNs) by exploiting sparsity. On such processors, one critical feature is that the speed and energy consumption of CNN inference are approximately proportional to the number of non-zero values in the activation maps. Thus, to achieve top performance, an efficient training algorithm is required to largely suppress the activations in CNNs. We propose a novel training method, called Adaptive-Regularization Training Schedule (ARTS), which dramatically decreases the non-zero activations in a model by adaptively altering the regularization coefficient through training. We evaluate our method across an extensive range of computer vision applications, including image classification, object recognition, depth estimation, and semantic segmentation. The results show that our technique can achieve 1.41 × to 6.00 × more activation suppression on top of ReLU activation across various networks and applications, and outperforms the state-of-the-art methods in terms of training time, activation suppression gains, and accuracy. A case study for a commercially-available event-based processor, Neuronflow, shows that the activation suppression achieved by ARTS effectively reduces CNN inference latency by up to 8.4 × and energy consumption by up to 14.1 ×.

Medical instrument segmentation in 3D ultrasound is essential for image-guided intervention. However, to train a successful deep neural network for instrument segmentation, a large number of labeled images are required, which is expensive and time-consuming to obtain. In this article, we propose a semi-supervised learning (SSL) framework for instrument segmentation in 3D US, which requires much less annotation effort than the existing methods. To achieve the SSL learning, a Dual-UNet is proposed to segment the instrument. The Dual-UNet leverages unlabeled data using a novel hybrid loss function, consisting of uncertainty and contextual constraints. Specifically, the uncertainty constraints leverage the uncertainty estimation of the predictions of the UNet, and therefore improve the unlabeled information for SSL training. In addition, contextual constraints exploit the contextual information of the training images, which are used as the complementary information for voxel-wise uncertainty estimation. Extensive experiments on multiple ex-vivo and in-vivo datasets show that our proposed method achieves Dice score of about 68.6%-69.1% and the inference time of about 1 sec. per volume. These results are better than the state-of-the-art SSL methods and the inference time is comparable to the supervised approaches., Comment: Accepted by IEEE JBHI

Barrett’s Esophagus (BE) is a well-known precursor for Esophageal Adenocarcinoma (EAC). Endoscopic detection and diagnosisof early BE neoplasia is performed in two steps: primary detection of asuspected lesion in overview and a targeted and detailed inspection of thespecific area using Narrow-Band Imaging (NBI). Despite the improvedvisualization of tissue by NBI and clinical classification systems, endoscopists have difficulties with correct characterization of the imagery.Computer-aided Diagnosis (CADx) may assist endoscopists in the classification of abnormalities in NBI imagery. We propose an endoscopydriven pre-trained deep learning-based CADx, for the characterization of NBI imagery of BE. We evaluate the performance of the algorithm on images as well as on videos, for which we use several post-hoc and real-time video analysis methods. The proposed real-time methods outperform the post-hoc methods on average by 1.2% and 2.3% for accuracy and specificity, respectively. The obtained results show promising methods towards real-time endoscopic video analysis and identifies steps forfurther development.

An intelligent transportation system (ITS) is one of the core elements of smart cities, enhancing public safety and relieving traffic congestion. Detection and classification of critical vehicles, such as police cars and ambulances, passing through roadways form crucial use cases for ITS. This paper proposes a solution for detecting and classifying safety-critical vehicles on urban roadways using deep learning models. At present, a large-scale dataset for critical vehicles is not publicly available. The appearance scarcity of emergency vehicles and different coloring standards in various countries are significant challenges. To cope with the mentioned drawbacks and to address the unique requirements of our smart city project, we first generate a large-scale critical vehicle dataset, combining images retrieved from various sources with the support of the YOLO vehicle detection model. The classes of the generated dataset are: fire truck, police car, ambulance, military police car, dang erous truck, and standard vehicle. Second, we compare the performance of the Vision in Transformer (ViT) network against the traditional convolutional neural networks (CNNs) for the task of critical vehicle classification. Experimental results on our dataset reveal that the ViT-based solution reaches an average accuracy and recall of 99.39% and 99.34%, respectively.

Background Optical diagnosis of colorectal polyps remains challenging. Image-enhancement techniques such as narrow-band imaging and blue-light imaging (BLI) can improve optical diagnosis. We developed and prospectively validated a computer-aided diagnosis system (CADx) using high-definition white-light (HDWL) and BLI images, and compared the system with the optical diagnosis of expert and novice endoscopists. Methods CADx characterized colorectal polyps by exploiting artificial neural networks. Six experts and 13 novices optically diagnosed 60 colorectal polyps based on intuition. After 4 weeks, the same set of images was permuted and optically diagnosed using the BLI Adenoma Serrated International Classification (BASIC). Results CADx had a diagnostic accuracy of 88.3 % using HDWL images and 86.7 % using BLI images. The overall diagnostic accuracy combining HDWL and BLI (multimodal imaging) was 95.0 %, which was significantly higher than that of experts (81.7 %, P = 0.03) and novices (66.7 %, P Conclusion CADx had a significantly higher diagnostic accuracy than experts and novices for the optical diagnosis of colorectal polyps. Multimodal imaging, incorporating both HDWL and BLI, improved the diagnostic accuracy of CADx. BASIC did not increase the diagnostic accuracy of endoscopists compared with intuitive optical diagnosis.

Breast cancer is the most common invasive cancer with the highest cancer occurrence in females. Handheld ultrasound is one of the most efficient ways to identify and diagnose the breast cancer. The area and the shape information of a lesion is very helpful for clinicians to make diagnostic decisions. In this study we propose a new deep-learning scheme, semi-pixel-wise cycle generative adversarial net (SPCGAN) for segmenting the lesion in 2D ultrasound. The method takes the advantage of a fully convolutional neural network (FCN) and a generative adversarial net to segment a lesion by using prior knowledge. We compared the proposed method to a fully connected neural network and the level set segmentation method on a test dataset consisting of 32 malignant lesions and 109 benign lesions. Our proposed method achieved a Dice similarity coefficient (DSC) of 0.92 while FCN and the level set achieved 0.90 and 0.79 respectively. Particularly, for malignant lesions, our method increases the DSC (0.90) of the fully connected neural network to 0.93 significantly (p < 0.001). The results show that our SPCGAN can obtain robust segmentation results. The framework of SPCGAN is particularly effective when sufficient training samples are not available compared to FCN. Our proposed method may be used to relieve the radiologists' burden for annotation.

In this paper, we define a novel inverse reinforcement learning (IRL) problem where the demonstrations are multi-intention, i.e., collected from multi-intention experts, unlabeled, i.e., without intention labels, and partially overlapping, i.e., shared between multiple intentions. In the presence of overlapping demonstrations, current IRL methods, developed to handle multi-intention and unlabeled demonstrations, cannot successfully learn the underlying reward functions. To solve this limitation, we propose a novel clustering-based approach to disentangle the observed demonstrations and experimentally validate its advantages. Traditional clustering-based approaches to multi-intention IRL, which are developed on the basis of model-based Reinforcement Learning (RL), formulate the problem using parametric density estimation. However, in high-dimensional environments and unknown system dynamics, i.e., model-free RL, the solution of parametric density estimation is only tractable up to the density normalization constant. To solve this, we formulate the problem as a mixture of logistic regressions to directly handle the unnormalized density. To research the challenges faced by overlapping demonstrations, we introduce the concepts of shared pair, which is a state-action pair that is shared in more than one intention, and separability, which resembles how well the multiple intentions can be separated in the joint state-action space. We provide theoretical analyses under the global optimality condition and the existence of shared pairs. Furthermore, we conduct extensive experiments on four simulated robotics tasks, extended to accept different intentions with specific levels of separability, and a synthetic driver task developed to directly control the separability. We evaluate the existing baselines on our defined problem and demonstrate, theoretically and experimentally, the advantages of our clustering-based solution, especially when the separability of the demonstrations decreases.

During minimally-invasive surgery, endoscopes and other surgical tools enter the body of the patient through small openings, allowing the operations to be performed with less post-operative pain and faster recovery for the patient, as well as less wound complications. Although great advantages are achieved from the patient’s point-of-view, several difficulties have still to be handled by the surgeons. First, the limited field of view of the endoscope makes target-area localizations and related identifications complex. Second, surgeons should still match mentally the medical images for the surgical planning with the current patient anatomy. These difficulties can be addressed by using computer vision technologies for guiding surgical procedures. More specifically, this thesis aims at the following three points, the first two for neurosurgery, and the third for perfusion assessment during surgery. The challenges for this dissertation are partitioned in two primary aspects. The first point aims at the fusion of medical images on the endoscopic view, in order to develop an augmented reality system. The second point is based on hyperspectral imaging (HSI), which is compared with diffuse reflectance spectroscopy (DRS), to improve brain tissue classification for neurosurgery. The third point exploits the PPG imaging (iPPG) technique and its potential use is evaluated for peripheral arterial disease (PAD) and organ perfusion assessment during surgery. To address the first point, a new neurosurgical application is implemented on the already existing Philips Augmented Reality (AR) surgical navigation system, designed for spinal surgery. This navigation system incorporates an optical tracking system (OTS) with four video cameras embedded in the flat detector of the motorized C-arm. A hand-eye camera calibration algorithm for the fusion of medical images on the endoscopic view is implemented and integrated into the AR system. This technology is validated for endo-nasal surgery, a neurosurgical procedure for the removal of tumors located at the skull base, such as pituitary tumors. Intra-operative Cone Beam Computed Tomography (CBCT) images are fused with the view of the surgical field obtained by the endoscope camera. The accuracy of CBCT image co-registration is tested, using a custom-made grid with incorporated 3D spheres. The system achieves a sub-millimeter accuracy of image overlay of 0.55 mm, measured as mean target registration error (TRE), with a standard deviation of 0.24 mm. Afterwards, an anatomically realistic head phantom is developed, with materials chosen to achieve both X-ray attenuation and mechanical properties, similar to the real tissue. Using the phantom, a proof of concept of the skull-base surgical simulation is provided, then the accuracy and efficacy of the AR system are evaluated for the insertion of biopsy needles, which is a common neurosurgical procedure that requires high precision. Several 2-mm spherical biopsy targets are inserted inside the brain of the brain phantom. The obtained mean accuracy of the biopsy needle insertions (n=30) is 0.8±0.43 mm, with a mean device insertion time of 155±43 seconds. These experiments demonstrate that a high accuracy is obtained during neurosurgery with the proposed methods and one phantom. For the second point, a near-infrared (NIR) hyperspectral imaging (HSI) sensor is mounted on an endoscope, to explore contactless brain-tissue classification and HSI is compared with diffuse reflectance spectroscopy (DRS), which is an alternative optical technique that requires a probe in contact with the tissue. The classification is performed on ex-vivo porcine brain tissue, which is analyzed and classified in white and gray matter. The HSI reaches a sensitivity of 95% and specificity of 93%, whereas DRS reaches sensitivity and specificity of 96%. The results show that the spectral signature of the tissue in the NIR range contains sufficient information to discriminate brain tissue in white/ gray matter. Further investigation on ex-vivo tumor sample data is required prior to the clinical validation of HSI. The third part concentrates on exploring PPG imaging for extracting perfusion information on tissue. By using an off-the-shelf camera and a light source, the dynamic changes in blood volume are remotely detected beneath the skin and a map is derived correlated to the blood perfusion. After evaluating PPG imaging for local and temporal perfusion-change detections, it is employed for Peripheral Arterial Diseases (PAD) assessment. Reduced blood flow is simulated on 21 volunteers and iPPG is compared with ultrasound and Laser Speckle Contrast Analysis. These experiments show that iPPG can detect reduced perfusion levels and correlates well with the other measurement systems. Finally, this technology is deployed for organ perfusion assessment during intestine surgery. The experiments demonstrate that PPG imaging can be successfully used for extracting perfusion maps from the organ surface, even for detecting perturbations and perfusion changes during several stages of the surgery. The results of this dissertation contribute with novel techniques and approaches to add value to endoscopic and navigation technology systems, as well as to tissue classification and perfusion monitoring, during minimally-invasive surgery. The three main explored research points and their proposed techniques, namely image fusion with the endoscopic view, tissue classification via HSI and PPG imaging for perfusion assessment, can be potentially implemented and combined into a single endoscopic platform, resulting into a new multi-modal endoscopic system. Moreover, the thesis shows that the proposed techniques and algorithms increase the quality of the decision-making process and have the potential to improve the patient surgical outcome.

Computational dentistry uses computerized methods and mathematical models for dental image analysis. One of the fundamental problems in computational dentistry is accurate tooth instance segmentation in high-resolution mesh data of intra-oral scans (IOS). This paper presents a new computational model based on deep neural networks, called Mask-MCNet, for end-to-end learning of tooth instance segmentation in 3D point cloud data of IOS. The proposed Mask-MCNet localizes each tooth instance by predicting its 3D bounding box and simultaneously segments the points that belong to each individual tooth instance. The proposed model processes the input raw 3D point cloud in its original spatial resolution without employing a voxelization or down-sampling technique. Such a characteristic preserves the finely detailed context in data like fine curvatures in the border between adjacent teeth and leads to a highly accurate segmentation as required for clinical practice (e.g. orthodontic planning). The experiments show that the Mask-MCNet outperforms state-of-the-art models by achieving 98% Intersection over Union (IoU) score on tooth instance segmentation which is very close to human expert performance.

Background: Detecting discomfort in infants is an important topic for their well-being and development. In this paper, we present an automatic and continuous video-based system for monitoring and detecting discomfort in infants. Methods: The proposed system employs a novel and efficient 3D convolutional neural network (CNN), which achieves an end-to-end solution without the conventional face detection and tracking steps. In the scheme of this study, we thoroughly investigate the video characteristics (e.g., intensity images and motion images) and CNN architectures (e.g., 2D and 3D) for infant discomfort detection. The realized improvements of the 3D-CNN are based on capturing both the motion and the facial expression information of the infants. Results: The performance of the system is assessed using videos recorded from 24 hospitalized infants by visualizing receiver operating characteristic (ROC) curves and measuring the values of area under the ROC curve (AUC). Additional performance metrics (labeling accuracy) are also calculated. Experimental results show that the proposed system achieves an AUC of 0.99, while the overall labeling accuracy is 0.98. Conclusions: These results confirms the robustness by using the 3D-CNN for infant discomfort monitoring and capturing both motion and facial expressions simultaneously.

Introduction In an earlier study by King’s College London (KCL), a framework for the automatic segmentation of vestibular schwannomas (VS) from MRI scans was proposed. They demonstrated that artificial intelligence is capable of annotating and calculating VS tumor volumes, which can benefit tumor progression monitoring. In this study, their algorithm is validated using data from another institution. The obtained results are subsequently compared to a multi-institutional model, in which the data from both centers is combined. Methods In addition to the original KCL dataset of 375 contrast-enhanced T1-weighted (ceT1) MRI scans, a total of 1,115 ceT1 scans of individual VS patients from the ETZ hospital in Tilburg, the Netherlands, were used. All tumors were manually annotated by a neurosurgeon at time of treatment. Employing the original 2.5D convolutional neural network, two different models were subsequently trained. First, one model is trained solely on 176 scans from the KCL dataset, using an additional 20 and 46 scans for hyper-parameter tuning and internal validation, respectively. This single-institution model is externally validated on the entire ETZ dataset. The second model employed 242 scans from the KCL dataset and all of the ETZ data from before 2015 (733 scans), with the remainder of the ETZ data equally distributed between a tuning and validation set. The remaining 133 scans from the KCL dataset were also used for validation. Results The single-institutional model achieved internal and external validation mean Dice scores of 92.0±5.1% and 64.5±32.4%, respectively. The external validation set included 175 scans where the model failed to recognize any VS, resulting in a Dice score of zero for these scans, which skews the results. The second model yielded validation mean dice scores of 92.5±5.1% for the ETZ scans and 89.1±9.6% for the KCL scans. During validation, most of the low-scoring segmentations are of tumors that are either very small or cystic. Conclusions The significant difference in performance between internal and external validation of the first model, thereby surmising a poor generalization, can be explained by variations in spatial resolution and image acquisition parameters between the two datasets. The second model shows performances approaching inter-observer variability of human annotators (cf. 93.8±3.1%). The results show that creating a robust and well-generalizing model from single-institution data is challenging. However, the multi-institutional results empower the earlier demonstrated capabilities of the framework for the automatic segmentation of VS.

Temporal localization of driving actions plays a crucial role in advanced driver-assistance systems and naturalistic driving studies. However, this is a challenging task due to strict requirements for robustness, reliability and accurate localization. In this work, we focus on improving the overall performance by efficiently utilizing video action recognition networks and adapting these to the problem of action localization. To this end, we first develop a density-guided label smoothing technique based on label probability distributions to facilitate better learning from boundary videosegments that typically include multiple labels. Second, we design a post-processing step to efficiently fuse information from video-segments and multiple camera views into scene-level predictions, which facilitates elimination of false positives. Our methodology yields a competitive performance on the A2 test set of the naturalistic driving action recognition track of the 2022 NVIDIA AI City Challenge with an F1 score of 0.271.

Encoding-decoding (ED) CNNs have demonstrated state-of-the-art performance for noise reduction over the past years. This has triggered the pursuit of better understanding the inner workings of such architectures, which has led to the theory of deep convolutional framelets (TDCF), revealing important links between signal processing and CNNs. Specifically, the TDCF demonstrates that ReLU CNNs induce low-rankness, since these models often do not satisfy the necessary redundancy to achieve perfect reconstruction (PR). In contrast, this paper explores CNNs that do meet the PR conditions. We demonstrate that in these type of CNNs soft shrinkage and PR can be assumed. Furthermore, based on our explorations we propose the learned wavelet-frame shrinkage network, or LWFSN and its residual counterpart, the rLWFSN. The ED path of the (r)LWFSN complies with the PR conditions, while the shrinkage stage is based on the linear expansion of thresholds proposed Blu and Luisier. In addition, the LWFSN has only a fraction of the training parameters (

Logs describing the execution of procedural steps during manufacturing and maintenance tasks are important for quality control and configuration management. Such logs are currently hand-written or typed during a procedure, which requires engineers to frequently step away from their work and results in difficulties for searching and optimizing logs. In this paper, we propose to automatically generate standardized, searchable logs, by visually perceiving and monitoring the progress of the procedure in real-time, and comparing this to the expected procedure. Unlike related work, we propose an approach which does not restrict the engineers to rigid, sequential sequences and instead allows them to execute procedures in a variety of different sequences where possible. The proposed framework is experimentally validated on the task of (dis)assembling a Duplo block model and operates properly when occlusions are absent.

To support the ongoing size reduction in integrated circuits, the need for accurate depth measurements of on-chip structures becomes increasingly important. Unfortunately, present metrology tools do not offer a practical solution. In the semiconductor industry, critical dimension scanning electron microscopes (CD-SEMs) are predominantly used for 2D imaging at a local scale. The main objective of this work is to investigate whether sufficient 3D information is present in a single SEM image for accurate surface reconstruction of the device topology. In this work, we present a method that is able to produce depth maps from synthetic and experimental SEM images. We demonstrate that the proposed neural network architecture, together with a tailored training procedure, leads to accurate depth predictions. The training procedure includes a weakly supervised domain adaptation step, which is further referred to as pixel-wise fine-tuning. This step employs scatterometry data to address the ground-truth scarcity problem. We have tested this method first on a synthetic contact hole dataset, where a mean relative error smaller than 6.2% is achieved at realistic noise levels. Additionally, it is shown that this method is well suited for other important semiconductor metrics, such as top critical dimension (CD), bottom CD and sidewall angle. To the extent of our knowledge, we are the first to achieve accurate depth estimation results on real experimental data, by combining data from SEM and scatterometry measurements. An experiment on a dense line space dataset yields a mean relative error smaller than 1%.

Correction to: Machine Learning https://doi.org/10.1007/s10994-022-06273-x There are two mistakes in the published article: 1. One of the references in the manuscript is incorrect. Here is the incorrect reference: “Bighashdel, A., Meletis, P., Jancura, P., & Dubbelman, G. (2021). In Proceeding of Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 206–221).” and here is the corrected reference: “Bighashdel, A., Meletis, P., Jancura, P., & Dubbelman, G. (2021). Deep Adaptive Multi-Intention Inverse Reinforcement Learning. In Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 206– 221).” 2. There is a missing character “i” in equation 3 of the published manuscript. Here is the incorrect equation:(Formula Presented.) and here is the corrected equation: (Formula Presented) The character “i” is missing in the term “ (a|s) ”. The corrected term is “ (a|s, i)”. The original article has been corrected.

Introduction: Fully automated artificial intelligence (AI) frameworks have recently reached outstanding performance for automatic segmentation of vestibular schwannomas (VS) from MRI. However, they often generalize poorly when the training data is acquired with scanners or imaging sequences that are different from those of the testing data. This problem strongly reduces the applicability of AI frameworks in real world radiosurgery settings. To increase the robustness of AI tools, various technical approaches have been proposed. However, these techniques have been validated either on private datasets or on small publicly available datasets. To tackle these limitations, the authors organized the CrossModality Domain Adaptation (crossMoDA) challenge in conjunction with the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021). Methods: The challenge’s goal was to segment two key brain structures involved in the follow-up and treatment planning of VS: the tumor and the cochlea. While contrast-enhanced T1 (ceT1) scans are commonly used for VS segmentation, recent work has demonstrated that high-resolution T2 (hrT2) imaging could be a reliable, safer, and lower-cost alternative to ceT1. For these reasons, the authors proposed an unsupervised cross-modality challenge to benchmark the generalization capability of techniques developed on images acquired with one sequence (ceT1) and tested on images acquired with another one (hrT2). Specifically, participants had access to a training set of unpaired annotated ceT1 (N=105) and non-annotated hrT2 (N=105). The automated segmentation tools developed by the challenge participants were then tested on a private hrT2 evaluation set (N=137). Images were collected on consecutive patients with a single sporadic VS treated with Gamma Knife stereotactic radiosurgery. Results: A total of 341 teams registered for the challenge, allowing them to download the data. 55 teams from 16 countries submitted predictions to the validation leaderboard. Among them, 16 teams from 9 countries submitted their algorithm for the evaluation phase. The level of performance reached by the topperforming teams is strikingly high (best median Dice score - VS: 88.4%; Cochleas: 85.7%) and close to a state-of-the-art model developed using hrT2 scans and their corresponding annotations (median Dice score - VS: 92.5%; Cochleas: 87.7%). Conclusions: The authors organized the first international benchmark assessing the robustness of AI frameworks for stereotactic surgery planning of VS. The excellent results obtained by the top-performing teams suggest that AI tools can be robust to different imaging sequences. The next challenge edition will assess the robustness of AI tools for Koos grade classification.

Introduction Prognostic factors of tumor control after Gamma Knife radiosurgery (GKRS) for vestibular schwannomas (VS) remain largely unknown. Four years ago, we reported that the growth rate of VS before GKRS is indicative of the probability that radiosurgery achieves tumor control. These findings may have important implications for treatment strategies and may lead to advise for either microsurgery or higher marginal doses for fast-growing tumors. The objective of this study is to validate the previously obtained results in the correlation between the pretreatment growth rate and tumor control after GKRS, using an updated and significantly larger database. Methods Patients treated between 2002 and 2015 were identified, who had a pretreatment scan available of at least 6 months prior to treatment and had at least 2 years of follow-up after GKRS. Tumor volumes before, at, and after treatment were assessed. GKRS was performed in a uniform way with a marginal dose between 11 and 13 Gy. Treatment failure was defined as radiological progression beyond 2 years after GKRS. Volume doubling times (VDTs) before treatment were correlated with the observed tumor control rates and volumetric responses after treatment. Kaplan-Meier and Cox regression analyses were employed to investigate the effect of the VDT on the treatment response. Results A total of 402 patients met the inclusion criteria. The median follow-up time is 85 months. In this cohort, 50 patients showed a radiological failure. The resulting 5- and 10-year tumor control rates are 92.7% and 83.2%, respectively. The calculated VDTs vary between 3 and 344 months, with a median VDT of 15 months. Splitting the cohort into two sub-cohorts using the median VDT, results in 5- and 10-year control rates of 87.1% and 74.3% for the faster growing tumors, and 97.8% and 91.8% for the slower growing tumors, respectively (log-rank, p

For safe and comfortable navigation of autonomous vehicles, it is crucial to know the pedestrian's intention of crossing the street. Generally, human drivers are aware of the traffic objects (e.g., crosswalks and traffic lights) in the environment while driving; likewise, these objects would play a crucial role for autonomous vehicles. In this research, we propose a novel pedestrian intention estimation method that not only takes into account the influence of traffic objects but also learns their contribution levels on the intention of the pedestrian. Our proposed method, referred to as Scene SpatioTemporal Graph Convolutional Network (Scene-STGCN), takes benefits from the strength of Graph Convolutional Networks and efficiently encodes the relationships between the pedestrian and the scene objects both spatially and temporally. We conduct several experiments on the Pedestrian Intention Estimation (PIE) dataset and illustrate the importance of scene objects and their contribution levels in the task of pedestrian intention estimation. Furthermore, we perform statistical analysis on the relevance of different traffic objects in the PIE dataset and carry out an ablation study on the effect of various information sources in the scene. Finally, we demonstrate the significance of the proposed Scene-STGCN through experimental comparisons with several baselines. The results indicate that our proposed Scene-STGCN outperforms the current state-of-the-art method by 0.03 in terms of ROC-AUC metric.

Computer-Aided Diagnosis (CADx) systems for in-vivo characterization of Colorectal Polyps (CRPs) which are precursor lesions of Colorectal Cancer (CRC), can assist clinicians with diagnosis and better informed decisionmaking during colonoscopy procedures. Current deep learning-based state-of-the-art solutions achieve a high classification performance, but lack measures to increase the reliability of such systems. In this paper, the reliability of a Convolutional Neural Network (CNN) for characterization of CRPs is specifically addressed by confidence calibration. Well-calibrated models produce classification-confidence scores that reflect the actual correctness likelihood of the model, thereby supporting reliable predictions by trustworthy and informative confidence scores. Two recently proposed trainable calibration methods are explored for CRP classification to calibrate the confidence of the proposed CNN. We show that the confidence-calibration error can be decreased by 33.86% (-0.01648 ± 0.01085), 48.33% (-0.04415 ± 0.01731), 50.57% (-0.11423 ± 0.00680), 61.68% (-0.01553 ± 0.00204) and 48.27% (-0.22074 ± 0.08652) for the Expected Calibration Error (ECE), Average Calibration Error (ACE), Maximum Calibration Error (MCE), Over-Confidence Error (OE) and Cumulative Calibration Error (CUMU), respectively. Moreover, the absolute difference between the average entropy and the expected entropy was considerably reduced by 32.00% (-0.04374 ± 0.01238) on average. Furthermore, even a slightly improved classification performance is observed, compared to the uncalibrated equivalent. The obtained results show that the proposed model for CRP classification with confidence calibration produces better calibrated predictions without sacrificing classification performance. This work shows promising points of engagement towards obtaining reliable and well-calibrated CADx systems for in-vivo polyp characterization, to assist clinicians during colonoscopy procedures.

Deep learning-based pose estimation algorithms can successfully estimate the pose of objects in an image, especially in the field of color images. 6D Object pose estimation based on deep learning models for X-ray images often use custom architectures that employ extensive CAD models and simulated data for training purposes. Recent RGB-based methods opt to solve pose estimation problems using small datasets, making them more attractive for the X-ray domain where medical data is scarcely available. We refine an existing RGB-based model (SingleShotPose) to estimate the 6D pose of a marked cube from grayscale X-ray images by creating a generic solution trained on only real X-ray data and adjusted for X-ray acquisition geometry. The model regresses 2D control points and calculates the pose through 2D/3D correspondences using Perspective-n-Point(PnP), allowing a single trained model to be used across all supporting cone-beam-based X-ray geometries. Since modern X-ray systems continuously adjust acquisition parameters during a procedure, it is essential for such a pose estimation network to consider these parameters in order to be deployed successfully and find a real use case. With a 5-cm/5-degree accuracy of 93% and an average 3D rotation error of 2.2 degrees, the results of the proposed approach are comparable with state-of-the-art alternatives, while requiring significantly less real training examples and being applicable in real-time applications., Published at SPIE Medical Imaging 2022

Background Estimates on miss rates for upper gastrointestinal neoplasia (UGIN) rely on registry data or old studies. Quality assurance programs for upper GI endoscopy are not fully established owing to the lack of infrastructure to measure endoscopists’ competence. We aimed to assess endoscopists’ accuracy for the recognition of UGIN exploiting the framework of artificial intelligence (AI) validation studies. Methods Literature searches of databases (PubMed/MEDLINE, EMBASE, Scopus) up to August 2020 were performed to identify articles evaluating the accuracy of individual endoscopists for the recognition of UGIN within studies validating AI against a histologically verified expert-annotated ground-truth. The main outcomes were endoscopists’ pooled sensitivity, specificity, positive and negative predictive value (PPV/NPV), and area under the curve (AUC) for all UGIN, for esophageal squamous cell neoplasia (ESCN), Barrett esophagus-related neoplasia (BERN), and gastric adenocarcinoma (GAC). Results Seven studies (2 ESCN, 3 BERN, 1 GAC, 1 UGIN overall) with 122 endoscopists were included. The pooled endoscopists’ sensitivity and specificity for UGIN were 82 % (95 % confidence interval [CI] 80 %–84 %) and 79 % (95 %CI 76 %–81 %), respectively. Endoscopists’ accuracy was higher for GAC detection (AUC 0.95 [95 %CI 0.93–0.98]) than for ESCN (AUC 0.90 [95 %CI 0.88–0.92]) and BERN detection (AUC 0.86 [95 %CI 0.84–0.88]). Sensitivity was higher for Eastern vs. Western endoscopists (87 % [95 %CI 84 %–89 %] vs. 75 % [95 %CI 72 %–78 %]), and for expert vs. non-expert endoscopists (85 % [95 %CI 83 %–87 %] vs. 71 % [95 %CI 67 %–75 %]). Conclusion We show suboptimal accuracy of endoscopists for the recognition of UGIN even within a framework that included a higher prevalence and disease awareness. Future AI validation studies represent a framework to assess endoscopist competence.

BACKGROUND AND AIMS: The endoscopic evaluation of narrow-band imaging (NBI) zoom imagery in Barrett's esophagus (BE) is associated with suboptimal diagnostic accuracy and poor interobserver agreement. Computer-aided diagnosis (CAD) systems may assist endoscopists in the characterization of Barrett's mucosa. Our aim was to demonstrate the feasibility of a deep-learning CAD system for tissue characterization of NBI zoom imagery in BE. METHODS: The CAD system was first trained using 494,364 endoscopic images of general endoscopic imagery. Next, 690 neoplastic BE and 557 nondysplastic BE (NDBE) white-light endoscopy overview images were used for refinement training. Subsequently, a third dataset of 112 neoplastic and 71 NDBE NBI zoom images with histologic correlation was used for training and internal validation. Finally, the CAD system was further trained and validated with a fourth, histologically confirmed dataset of 59 neoplastic and 98 NDBE NBI zoom videos. Performance was evaluated using fourfold cross-validation. The primary outcome was the diagnostic performance of the CAD system for classification of neoplasia in NBI zoom videos. RESULTS: The CAD system demonstrated accuracy, sensitivity, and specificity for detection of BE neoplasia using NBI zoom images of 84%, 88%, and 78%, respectively. In total, 30,021 individual video frames were analyzed by the CAD system. Accuracy, sensitivity, and specificity of the video-based CAD system were 83% (95% confidence interval [CI], 78%-89%), 85% (95% CI, 76%-94%), and 83% (95% CI, 76%-90%), respectively. The mean assessment speed was 38 frames per second. CONCLUSION: We have demonstrated promising diagnostic accuracy of predicting the presence/absence of Barrett's neoplasia on histologically confirmed unaltered NBI zoom videos with fast corresponding assessment time. ispartof: GASTROINTESTINAL ENDOSCOPY vol:93 issue:1 pages:89-98 ispartof: location:United States status: published

This paper presents a deep Inverse Reinforcement Learning (IRL) framework that can learn an a priori unknown number of nonlinear reward functions from unlabeled experts’ demonstrations. For this purpose, we employ the tools from Dirichlet processes and propose an adaptive approach to simultaneously account for both complex and unknown number of reward functions. Using the conditional maximum entropy principle, we model the experts’ multi-intention behaviors as a mixture of latent intention distributions and derive two algorithms to estimate the parameters of the deep reward network along with the number of experts’ intentions from unlabeled demonstrations. The proposed algorithms are evaluated on three benchmarks, two of which have been specifically extended in this study for multi-intention IRL, and compared with well-known baselines. We demonstrate through several experiments the advantages of our algorithms over the existing approaches and the benefits of online inferring, rather than fixing beforehand, the number of expert’s intentions.

There has been a vast increase in GI literature focused on the use of machine learning in endoscopy. The relative novelty of this field poses a challenge for reviewers and readers of GI journals. To appreciate scientific quality and novelty of machine learning studies, understanding of the technical basis and commonly used techniques is required. Clinicians often lack this technical background, while machine learning experts may be unfamiliar with clinical relevance and implications for daily practice. Therefore, there is an increasing need for a multidisciplinary, international evaluation on how to perform high-quality machine learning research in endoscopy. This review aims to provide guidance for readers and reviewers of peer-reviewed GI journals to allow critical appraisal of the most relevant quality requirements of machine learning studies. The paper provides an overview of common trends and their potential pitfalls and proposes comprehensive quality requirements in six overarching themes: terminology, data, algorithm description, experimental setup, interpretation of results and machine learning in clinical practice.

Infant monitoring is very important in daily life, since infants lack the ability to verbalize and communicate with their caregivers. However, a full-time monitoring provided by caregivers or practitioners is very expensive and labor intensive. To solve the practical difficulties for particular nighttime monitoring and further reduce the labor cost, in this article, we propose a novel 24-hour infant monitoring system, enabling continuous monitoring for infant discomfort detection using both Red-Green-Blue (RGB) and Near Infrared-light (NIR) videos. The proposed algorithm is robust to arbitrary head rotations, occlusions and face profiles. For this purpose, a Faster R-CNN architecture is first pre-trained with the ImageNet dataset, and then fine-tuned with a specific training dataset of different infant expressions. Our proposed method obtains an Average Precision of more than 87% for classifying discomfort expression. Infant monitoring during the nighttime using NIR videos is proposed for the first time, and the presented system enables reflux disease analysis and continuous remote home monitoring in a more relaxed environment, which is largely preferred by pediatricians and parents.

Aims Artificial intelligence (AI) has great potential in gastrointestinal endoscopy. Aim was to evaluate real-time diagnostic performances of our Artificial Intelligence for ColoRectal Polyps (AI4CRP) computer-aided diagnosis system for optical diagnosis of diminutive colorectal polyps (CRPs) and compare it with CAD EYE and an expert endoscopist.Methods AI4CRP was developed using convolutional neural networks and previously trained and tested. In this prospective real-time pilot study, AI4CRP was compared with CAD EYE© (Fujifilm, Tokyo, Japan) and one expert endoscopist unaware of AI-output. Blue light imaging was used for characterization and histopathology as gold standard. CRPs were characterized as hyperplastic (hyperplastic polyps) or neoplastic (adenomas, sessile serrated lesions[SSLs]) by AI4CRP and the endoscopist, and as hyperplastic (hyperplastic polyps, SSLs) or neoplastic (adenomas) by CAD EYE. CAD EYE’s inconclusive diagnoses were excluded. Enabling self-critical AI4CRP, post-hoc analysis excluded low confidence scores.Results Real-time testing included 30 patients with 51 CRPs (32 adenomas, 6 SSLs, 12 hyperplastic polyps). AI4CRP had a diagnostic accuracy of 80.4%, sensitivity of 82.1%, and specificity of 75.0%. For self-critical AI4CRP (n=37) the diagnostic accuracy was 89.2%, sensitivity 89.7%, and specificity 87.5%. CAD EYE (n=49) had a diagnostic accuracy of 83.7%, sensitivity of 74.2%, and specificity of 100.0%. For the expert endoscopist the diagnostic accuracy was 88.2%, sensitivity 94.9%, and specificity 66.7%.

Accurate and efficient catheter segmentation in 3D ultrasound (US) is essential for cardiac intervention. Currently, the state-of-the-art segmentation algorithms are based on convolutional neural networks (CNNs), which achieved remarkable performances in a standard Cartesian volumetric data. Nevertheless, these approaches suffer the challenges of low efficiency and GPU unfriendly image size. Therefore, such difficulties and expensive hardware requirements become a bottleneck to build accurate and efficient segmentation models for real clinical application. In this paper, we propose a novel Frustum ultrasound based catheter segmentation method. Specifically, Frustum ultrasound is a polar coordinate based image, which includes same information of standard Cartesian image but has much smaller size, which overcomes the bottleneck of efficiency than conventional Cartesian images. Nevertheless, the irregular and deformed Frustum images lead to more efforts for accurate voxel-level annotation. To address this limitation, a weakly supervised learning framework is proposed, which only needs 3D bounding box annotations overlaying the region-of-interest to training the CNNs. Although the bounding box annotation includes noise and inaccurate annotation to mislead to model, it is addressed by the proposed pseudo label generated scheme. The labels of training voxels are generated by incorporating class activation maps with line filtering, which is iteratively updated during the training. Our experimental results show the proposed method achieved the state-of-the-art performance with an efficiency of 0.25 second per volume. More crucially, the Frustum image segmentation provides a much faster and cheaper solution for segmentation in 3D US image, which meet the demands of clinical applications.

The most important prognostic factor for the survival of advanced-stage epithelial ovarian cancer (EOC) is the completeness of cytoreductive surgery (CRS). Therefore, an intraoperative technique to detect microscopic tumors would be of great value. The aim of this pilot study is to assess the feasibility of near-infrared hyperspectral imaging (HSI) for EOC detection in ex vivo tissue samples. Images were collected during CRS in 11 patients in the wavelength range of 665–975 nm, and processed by calibration, normalization, and noise filtering. A linear support vector machine (SVM) was employed to classify healthy and tumorous tissue (defined as >50% tumor cells). Classifier performance was evaluated using leave-one-out cross-validation. Images of 26 tissue samples from 10 patients were included, containing 26,446 data points that were matched to histopathology. Tumorous tissue could be classified with an area under the curve of 0.83, a sensitivity of 0.81, a specificity of 0.70, and Matthew’s correlation coefficient of 0.41. This study paves the way to in vivo and intraoperative use of HSI during CRS. Hyperspectral imaging can scan a whole tissue surface in a fast and non-contact way. Our pilot study demonstrates that HSI and SVM learning can be used to discriminate EOC from surrounding tissue.

Since the introduction of the GDPR and CCPA privacy legislation, both public and private facial image datasets are increasingly scrutinized. Several datasets have been taken offline completely and some have been anonymized. However, it is unclear how anonymization impacts face detection performance. To our knowledge, this paper presents the first empirical study on the effect of image anonymization on supervised training of face detectors. We compare conventional face anonymiz-ers with three state-of-the-art Generative Adversarial Network-based (GAN) methods, by training an off-the-shelf face detector on anonymized data. Our experiments investigate the suitability of anonymization methods for maintaining face detector performance, the effect of detectors overtraining on anonymization artefacts, dataset size for training an anonymizer, and the effect of training time of anonymization GANs. A final experiment investigates the correlation between common GAN evaluation metrics and the performance of a trained face detector. Although all tested anonymization methods lower the performance of trained face detectors, faces anonymized using GANs cause far smaller performance degradation than conventional methods. As the most important finding, the best-performing GAN, DeepPrivacy, removes identifiable faces for a face detector trained on anonymized data, resulting in a modest decrease from 91.0 to 88.3 mAP. In the last few years, there have been rapid improvements in realism of GAN-generated faces. We expect that further progression in GAN research will allow the use of Deep Fake technology for privacy-preserving Safe Fakes, without any performance degradation for training face detectors.