Search

Medical instrument segmentation in 3D ultrasound is essential for image-guided intervention. However, to train a successful deep neural network for instrument segmentation, a large number of labeled images are required, which is expensive and time-consuming to obtain. In this article, we propose a semi-supervised learning (SSL) framework for instrument segmentation in 3D US, which requires much less annotation effort than the existing methods. To achieve the SSL learning, a Dual-UNet is proposed to segment the instrument. The Dual-UNet leverages unlabeled data using a novel hybrid loss function, consisting of uncertainty and contextual constraints. Specifically, the uncertainty constraints leverage the uncertainty estimation of the predictions of the UNet, and therefore improve the unlabeled information for SSL training. In addition, contextual constraints exploit the contextual information of the training images, which are used as the complementary information for voxel-wise uncertainty estimation. Extensive experiments on multiple ex-vivo and in-vivo datasets show that our proposed method achieves Dice score of about 68.6%-69.1% and the inference time of about 1 sec. per volume. These results are better than the state-of-the-art SSL methods and the inference time is comparable to the supervised approaches., Comment: Accepted by IEEE JBHI

Breast cancer is the most common invasive cancer with the highest cancer occurrence in females. Handheld ultrasound is one of the most efficient ways to identify and diagnose the breast cancer. The area and the shape information of a lesion is very helpful for clinicians to make diagnostic decisions. In this study we propose a new deep-learning scheme, semi-pixel-wise cycle generative adversarial net (SPCGAN) for segmenting the lesion in 2D ultrasound. The method takes the advantage of a fully convolutional neural network (FCN) and a generative adversarial net to segment a lesion by using prior knowledge. We compared the proposed method to a fully connected neural network and the level set segmentation method on a test dataset consisting of 32 malignant lesions and 109 benign lesions. Our proposed method achieved a Dice similarity coefficient (DSC) of 0.92 while FCN and the level set achieved 0.90 and 0.79 respectively. Particularly, for malignant lesions, our method increases the DSC (0.90) of the fully connected neural network to 0.93 significantly (p < 0.001). The results show that our SPCGAN can obtain robust segmentation results. The framework of SPCGAN is particularly effective when sufficient training samples are not available compared to FCN. Our proposed method may be used to relieve the radiologists' burden for annotation.

Computational dentistry uses computerized methods and mathematical models for dental image analysis. One of the fundamental problems in computational dentistry is accurate tooth instance segmentation in high-resolution mesh data of intra-oral scans (IOS). This paper presents a new computational model based on deep neural networks, called Mask-MCNet, for end-to-end learning of tooth instance segmentation in 3D point cloud data of IOS. The proposed Mask-MCNet localizes each tooth instance by predicting its 3D bounding box and simultaneously segments the points that belong to each individual tooth instance. The proposed model processes the input raw 3D point cloud in its original spatial resolution without employing a voxelization or down-sampling technique. Such a characteristic preserves the finely detailed context in data like fine curvatures in the border between adjacent teeth and leads to a highly accurate segmentation as required for clinical practice (e.g. orthodontic planning). The experiments show that the Mask-MCNet outperforms state-of-the-art models by achieving 98% Intersection over Union (IoU) score on tooth instance segmentation which is very close to human expert performance.

Background: Detecting discomfort in infants is an important topic for their well-being and development. In this paper, we present an automatic and continuous video-based system for monitoring and detecting discomfort in infants. Methods: The proposed system employs a novel and efficient 3D convolutional neural network (CNN), which achieves an end-to-end solution without the conventional face detection and tracking steps. In the scheme of this study, we thoroughly investigate the video characteristics (e.g., intensity images and motion images) and CNN architectures (e.g., 2D and 3D) for infant discomfort detection. The realized improvements of the 3D-CNN are based on capturing both the motion and the facial expression information of the infants. Results: The performance of the system is assessed using videos recorded from 24 hospitalized infants by visualizing receiver operating characteristic (ROC) curves and measuring the values of area under the ROC curve (AUC). Additional performance metrics (labeling accuracy) are also calculated. Experimental results show that the proposed system achieves an AUC of 0.99, while the overall labeling accuracy is 0.98. Conclusions: These results confirms the robustness by using the 3D-CNN for infant discomfort monitoring and capturing both motion and facial expressions simultaneously.

This paper presents a deep Inverse Reinforcement Learning (IRL) framework that can learn an a priori unknown number of nonlinear reward functions from unlabeled experts’ demonstrations. For this purpose, we employ the tools from Dirichlet processes and propose an adaptive approach to simultaneously account for both complex and unknown number of reward functions. Using the conditional maximum entropy principle, we model the experts’ multi-intention behaviors as a mixture of latent intention distributions and derive two algorithms to estimate the parameters of the deep reward network along with the number of experts’ intentions from unlabeled demonstrations. The proposed algorithms are evaluated on three benchmarks, two of which have been specifically extended in this study for multi-intention IRL, and compared with well-known baselines. We demonstrate through several experiments the advantages of our algorithms over the existing approaches and the benefits of online inferring, rather than fixing beforehand, the number of expert’s intentions.

There has been a vast increase in GI literature focused on the use of machine learning in endoscopy. The relative novelty of this field poses a challenge for reviewers and readers of GI journals. To appreciate scientific quality and novelty of machine learning studies, understanding of the technical basis and commonly used techniques is required. Clinicians often lack this technical background, while machine learning experts may be unfamiliar with clinical relevance and implications for daily practice. Therefore, there is an increasing need for a multidisciplinary, international evaluation on how to perform high-quality machine learning research in endoscopy. This review aims to provide guidance for readers and reviewers of peer-reviewed GI journals to allow critical appraisal of the most relevant quality requirements of machine learning studies. The paper provides an overview of common trends and their potential pitfalls and proposes comprehensive quality requirements in six overarching themes: terminology, data, algorithm description, experimental setup, interpretation of results and machine learning in clinical practice.

Infant monitoring is very important in daily life, since infants lack the ability to verbalize and communicate with their caregivers. However, a full-time monitoring provided by caregivers or practitioners is very expensive and labor intensive. To solve the practical difficulties for particular nighttime monitoring and further reduce the labor cost, in this article, we propose a novel 24-hour infant monitoring system, enabling continuous monitoring for infant discomfort detection using both Red-Green-Blue (RGB) and Near Infrared-light (NIR) videos. The proposed algorithm is robust to arbitrary head rotations, occlusions and face profiles. For this purpose, a Faster R-CNN architecture is first pre-trained with the ImageNet dataset, and then fine-tuned with a specific training dataset of different infant expressions. Our proposed method obtains an Average Precision of more than 87% for classifying discomfort expression. Infant monitoring during the nighttime using NIR videos is proposed for the first time, and the presented system enables reflux disease analysis and continuous remote home monitoring in a more relaxed environment, which is largely preferred by pediatricians and parents.

Early Barrett’s neoplasia are often missed due to subtle visual features and inexperience of the non-expert endoscopist with such lesions. While promising results have been reported on the automated detection of this type of early cancer in still endoscopic images, video-based detection using the temporal domain is still open. The temporally stable nature of video data in endoscopic examinations enables to develop a framework that can diagnose the imaged tissue class over time, thereby yielding a more robust and improved model for spatial predictions. We show that the introduction of Recurrent Neural Network nodes offers a more stable and accurate model for tissue classification, compared to classification on individual images. We have developed a customized Resnet18 feature extractor with four types of classifiers: Fully Connected (FC), Fully Connected with an averaging filter (FC Avg (n = 5)), Long Short Term Memory (LSTM) and a Gated Recurrent Unit (GRU). Experimental results are based on 82 pullback videos of the esophagus with 46 high-grade dysplasia patients. Our results demonstrate that the LSTM classifier outperforms the FC, FC Avg (n = 5) and GRU classifier with an average accuracy of 85.9% compared to 82.2%, 83.0% and 85.6%, respectively. The benefit of our novel implementation for endoscopic tissue classification is the inclusion of spatio-temporal information for improved and robust decision making, and it is the first step towards full temporal learning of esophageal cancer detection in endoscopic video.

This paper extends the 'go-with-the-flow' method to explore enclosed environments, like oil reservoirs, pipe lines that transport liquids, and industrial tanks for processing chemicals, where sensing nodes cannot establish communication with the external world. Nonetheless, large quantities of highly miniaturized, thus power-constrained sensor nodes are injected into these environment and flow through them along with the medium, monitoring their environment but also reconstructing their time-varying position from mutual communication, but without any communication to external base stations or beacons. The relative trajectories of nodes yield essential insights of the fluid flow in the otherwise inaccessible environment. We present a functional implementation of a ranging protocol accommodating size and energy constraints. Our simulation chain models node movement from different types of flow dynamics. It comprehensively assesses not only the performance of the communication and ranging protocols, but also of the reconstruction algorithm. Our assessments cover a wide range of different environments and flow profiles, including highly dynamic ones.

The lane-level localization of vehicles with low-cost sensors is a challenging task. In situations in which Global Navigation Satellite Systems (GNSSs) suffer from weak observation geometry or from the influence of reflected signals, the fusion of heterogeneous information presents a suitable approach for improving the localization accuracy. We propose a solution based on a monocular front-facing camera, a low-cost inertial measurement unit (IMU), and a single-frequency GNSS receiver. The sensor data fusion is implemented as a tightly coupled Kalman filter that corrects the IMU-based trajectory with GNSS observations while employing European Geostationary Overlay Service correction data. Further, we consider vision-based complementary data that serve as an additional source of information. In contrast to other approaches, the camera is not used to infer the motion of the vehicle, but rather for directly correcting the localization results under the usage of map information. More specifically, the so-called camera-to-map alignment is done by comparing virtual 3D views (candidates) created from projected map data with lane geometry features that are extracted from the camera image. One strength of the proposed solution is its compatibility with state-of-the-art map data, which are publicly available from different sources. We validate the approach on real-world data recorded in The Netherlands and show that it presents a promising and cost-efficient means to support future advanced driver assistance systems.

Person re-identification (re-ID) is a significant problem of computer vision with increasing scientific attention. To date, numerous studies have been conducted to improve the accuracy and robustness of person re-ID to meet the practical demands. However, most of the previous efforts concentrated on solving the closed-set variant of the problem, where a query is assumed to always have a correct match within the set of known people (the gallery set). However, this assumption is usually not valid for the industrial re-ID use cases. In this study, we focus on the open-set person re-ID problem, where, in addition to the similarity ranking, the solution is expected to detect the presence or absence of a given query identity within the gallery set. To determine good practices and to assess the practicality of the person re-ID in industrial applications, first, we convert popular closed-set person re-ID datasets into the open-set scenario. Second, we compare performance of eight state-of-the-art closed-set person re-ID methods under the open-set conditions. Third, we experimentally determine the efficiency of using different loss function combinations for the open-set problem. Finally, we investigate the impact of a statistics-driven gallery refinement approach on the open-set person re-ID performance in the low false-acceptance rate (FAR) region, while simultaneously reducing the computational demands of retrieval. Results show an average detection and identification rate increase of 8.38% and 3.39% on the DukeMTMC-reID and Market1501 datasets, respectively, for a FAR of 1%.

The availability of large-scale remote sensing (RS) data facilitates a wide range of applications, such as disaster management and urban planning. An approach for such problems is image retrieval, where, given a query image, the goal is to find the most relevant match from a database. Most RS literature has been focused on single-label retrieval, where we assume an image has a single label. The primary challenge in single-label RS retrieval is that performance in most datasets is saturated, and it has become difficult to compare the performance of different methods. In this work, we extend the major multilabel classification datasets to the multilabel retrieval problem. We also define protocols, provide evaluation metrics, and study the impact of commonly used loss functions and reranking methods for multilabel retrieval. To this end, a novel multilabel loss function and a reranking technique are proposed, which circumvent the challenges present in conventional single-label image retrieval. The developed loss function considers both class and feature similarity. The proposed reranking technique achieves high performance with computation cost that is well-suited for fast online retrieval.

In this work, we introduce the new scene understanding task of Part-aware Panoptic Segmentation (PPS), which aims to understand a scene at multiple levels of abstraction, and unifies the tasks of scene parsing and part parsing. For this novel task, we provide consistent annotations on two commonly used datasets: Cityscapes and Pascal VOC. Moreover, we present a single metric to evaluate PPS, called Part-aware Panoptic Quality (PartPQ). For this new task, using the metric and annotations, we set multiple baselines by merging results of existing state-of-the-art methods for panoptic segmentation and part segmentation. Finally, we conduct several experiments that evaluate the importance of the different levels of abstraction in this single task., Comment: CVPR 2021. Code and data: https://github.com/tue-mps/panoptic_parts

Radiation exposure in CT imaging leads to increased patient risk. This motivates the pursuit of reduced-dose scanning protocols, in which noise reduction processing is indispensable to warrant clinically acceptable image quality. Convolutional Neural Networks (CNNs) have received significant attention as an alternative for conventional noise reduction and are able to achieve state-of-the art results. However, the internal signal processing in such networks is often unknown, leading to sub-optimal network architectures. The need for better signal preservation and more transparency motivates the use of Wavelet Shrinkage Networks (WSNs), in which the Encoding-Decoding (ED) path is the fixed wavelet frame known as Overcomplete Haar Wavelet Transform (OHWT) and the noise reduction stage is data-driven. In this work, we considerably extend the WSN framework by focusing on three main improvements. First, we simplify the computation of the OHWT that can be easily reproduced. Second, we update the architecture of the shrinkage stage by further incorporating knowledge of conventional wavelet shrinkage methods. Finally, we extensively test its performance and generalization, by comparing it with the RED and FBPConvNet CNNs. Our results show that the proposed architecture achieves similar performance to the reference in terms of MSSIM (0.667, 0.662 and 0.657 for DHSN2, FBPConvNet and RED, respectively) and achieves excellent quality when visualizing patches of clinically important structures. Furthermore, we demonstrate the enhanced generalization and further advantages of the signal flow, by showing two additional potential applications, in which the new DHSN2 is used as regularizer: (1) iterative reconstruction and (2) ground-truth free training of the proposed noise reduction architecture. The presented results prove that the tight integration of signal processing and deep learning leads to simpler models with improved generalization.

Colorectal polyps (CRP) are precursor lesions of colorectal cancer (CRC). Correct identification of CRPs during in-vivo colonoscopy is supported by the endoscopist's expertise and medical classification models. A recent developed classification model is the Blue light imaging Adenoma Serrated International Classification (BASIC) which describes the differences between non-neoplastic and neoplastic lesions acquired with blue light imaging (BLI). Computer-aided detection (CADe) and diagnosis (CADx) systems are efficient at visually assisting with medical decisions but fall short at translating decisions into relevant clinical information. The communication between machine and medical expert is of crucial importance to improve diagnosis of CRP during in-vivo procedures. In this work, the combination of a polyp image classification model and a language model is proposed to develop a CADx system that automatically generates text comparable to the human language employed by endoscopists. The developed system generates equivalent sentences as the human-reference and describes CRP images acquired with white light (WL), blue light imaging (BLI) and linked color imaging (LCI). An image feature encoder and a BERT module are employed to build the AI model and an external test set is used to evaluate the results and compute the linguistic metrics. The experimental results show the construction of complete sentences with an established metric scores of BLEU-1 = 0.67, ROUGE-L = 0.83 and METEOR = 0.50. The developed CADx system for automatic CRP image captioning facilitates future advances towards automatic reporting and may help reduce time-consuming histology assessment.

This study presents a scalable automated video surveillance framework that (1) automatically detects the occurrences of abnormal behavior patterns by both pedestrians and vehicles, and (2) directs the focus of the security personnel to the relevant camera view, thereby providing global situational awareness. Powered by deep learning, our methodology can detect both vision and location-based abnormalities, including the events of vandalism, violence, loitering, scouting, and speeding. The proposed framework requires a low initial investment cost and features both real-time detection of various abnormal behaviors and post-crime analysis in scalable form, by enabling wide-area multi-camera networks with person/object re-identification. By combining multiple functionalities in an efficient framework, the proposed system opens up new possibilities for surveillance.

Retrieving specific vehicle tracks by Natural Language (NL)-based descriptions is a convenient way to monitor vehicle movement patterns and traffic-related events. NL-based image retrieval has several applications in smart cities, traffic control, etc. In this work, we propose TIED, a text-to-image encoder-decoder model for the simultaneous extraction of visual and textual information for vehicle track retrieval. The model consists of an encoder network that enforces the two modalities into a common latent space and a decoder network that performs an inverse mapping to the text descriptions. The method exploits visual semantic attributes of a target vehicle along with a cycle-consistency loss. The proposed method employs both intra-modal and inter-modal relationships to improve retrieval performance. Our system yields competitive performance achieving the 7th position in the Natural Language-Based Vehicle Retrieval public track of the 2021 NVIDIA AI City Challenge. We demonstrate that the proposed TIED model obtains six times higher Mean Reciprocal Rank (MRR) than the baseline, achieving an MRR of 15.48. The code and models will be made publicly available.

A major problem in applying machine learning for the medical domain is the scarcity of labeled data, which results in the demand for methods that enable high-quality models trained with little to no labels. Self-supervised learning methods present a plausible solution to this problem, enabling the use of large sets of unlabeled data for model pretraining. In this study, multiple of these methods and training strategies are employed on a large dataset of endoscopic images from the gastrointestinal tract (GastroNet). The suitability of these methods is assessed for an intra-domain downstream classification task on a small endoscopic dataset, involving neoplasia in Barrett’s esophagus. The classification performances are compared against pretraining on ImageNet and training from scratch. This yields promising results for domain-specific self-supervised methods, where super-resolution outperforms pretraining on ImageNet with a mean classification accuracy of 83.8% (cf. 79.2%). This implies that the large amounts of unlabeled data in hospitals could be employed in combination with self-supervised learning methods to improve models for downstream tasks.

This paper presents a camera-based vessel-speed enforcement system based on two cameras. The proposed system detects and tracks vessels per camera view and employs a re-identification (re-ID) function for linking vessels between the two cameras based on multiple bounding-box images per vessel. Newly detected vessels in one camera (query) are compared to the gallery set of all vessels detected by the other camera. To train and evaluate the proposed detection and re-ID system, a new Vessel-reID dataset is introduced. This extensive dataset has captured a total of 2474 different vessels covered in multiple images, resulting in a total of 136,888 vessel bounding-box images. Multiple CNN detector architectures are evaluated in-depth. The SSD512 detector performs best with respect to its speed (85.0% Recall@95Precision at 20.1 frames per second). For the re-ID of vessels, a large portion of the total trajectory can be covered by the successful detections of the SSD model. The re-ID experiments start with a baseline single-image evaluation obtaining a score of 55.9% Rank-1 (49.7% mAP) for the existing TriNet network, while the available MGN model obtains 68.9% Rank-1 (62.6% mAP). The performance significantly increases with 5.6% Rank-1 (5.7% mAP) for MGN by applying matching with multiple images from a single vessel. When emphasizing more fine details by selecting only the largest bounding-box images, another 2.0% Rank-1 (1.4% mAP) is added. Application-specific optimizations such as travel-time selection and applying a cross-camera matching constraint further enhance the results, leading to a final 88.9% Rank-1 and 83.5% mAP performance.

Current prognostic risk scores for transcatheter aortic valve implantation (TAVI) do not benefit yet from modern machine learning techniques, which can improve risk stratification of one-year mortality of patients before TAVI. Despite the advancement of machine learning in healthcare, data sharing regulations are very strict and typically prevent exchanging patient data, without the involvement of ethical committees. A very robust validation approach, including 1300 and 631 patients per center, was performed to validate a machine learning model of one center at the other external center with their data, in a mutual fashion. This was achieved without any data exchange but solely by exchanging the models and the data processing pipelines. A dedicated exchange protocol was designed to evaluate and quantify the model’s robustness on the data of the external center. Models developed with the larger dataset offered similar or higher prediction accuracy on the external validation. Logistic regression, random forest and CatBoost lead to areas under curve of the ROC of 0.65, 0.67 and 0.65 for the internal validation and of 0.62, 0.66, 0.68 for the external validation, respectively. We propose a scalable exchange protocol which can be further extended on other TAVI centers, but more generally to any other clinical scenario, that could benefit from this validation approach.

In medical imaging, a proper gold-standard ground truth as, e.g., annotated segmentations by assessors or experts is lacking or only scarcely available and suffers from large intervariability in those segmentations. Most state-of-the-art segmentation models do not take inter-observer variability into account and are fully deterministic in nature. In this work, we propose hypersphere encoder–decoder networks in combination with dynamic leaky ReLUs, as a new method to explicitly incorporate inter-observer variability into a segmentation model. With this model, we can then generate multiple proposals based on the inter-observer agreement. As a result, the output segmentations of the proposed model can be tuned to typical margins inherent to the ambiguity in the data. For experimental validation, we provide a proof of concept on a toy data set as well as show improved segmentation results on two medical data sets. The proposed method has several advantages over current state-of-the-art segmentation models such as interpretability in the uncertainty of segmentation borders. Experiments with a medical localization problem show that it offers improved biopsy localizations, which are on average 12% closer to the optimal biopsy location.

Recent deep object detection methods neglect the intrinsic properties of the license plate, which limits the detection performance in unconstrained scenes. In this paper, we propose a two-stage deep learning-based method to locate license plates in unconstrained scenes, especially for special license plates such as fouling, occlusion, and so on. A deep network consisting of convolutional neural network (CNN) and recurrent neural network is designed. In the first stage, fine-scale proposals are detected according to the characteristics of the license plate characters, and CNN is used to extract the local features of characters. A vertical anchor mechanism is designed to jointly predict the position and confidence of each fix-width character. Furthermore, the sequential contexts of characters are modeled with the bi-directional long short-term memory, which greatly improves the locating rate of license plates in complex scenes. In the second stage, the whole license plate is obtained by connecting the fine-scale proposals. The experimental results show that the proposed method not only locates license plates of different countries accurately but also be robust to scenes of illumination variation, noise distortion, and blurry effects. The average precision reaches 97.11% on multi-country license plates, and the precision and recall reaches 99.10% and 98.68%, respectively, on Chinese license plate images.

Image captioning, which aims to automatically generate text description of given images, has received much attention from researchers. Most existing approaches adopt a recurrent neural network (RNN) as a decoder to generate captions conditioned on the input image information. However, traditional RNNs deal with the sequence in a recurrent way, squeezing the information of all previous words into hidden cells and updating the context information by fusing the hidden states with the current word information. This may miss the rich knowledge too far in the past. In this paper, we propose a memory-enhanced captioning model for image captioning. We firstly introduce an external memory to store the past knowledge, i.e., all the information of generated words. When predicting the next word, the decoder can retrieve knowledge information about the past by means of a selective reading mechanism. Furthermore, to better explore the knowledge stored in the memory, we introduce several variants that consider different types of past knowledge. To verify the effectiveness of the proposed model, we conduct extensive experiments and comparisons on the well-known image captioning dataset MS COCO. Compared with the state-of-the-art captioning models, the proposed memory-enhanced captioning model shows a significant improvement in terms of the performance (improving 3.5% in terms of CIDEr). The proposed memory-enhanced captioning model, as demonstrated in the experiments, is more effective and superior to the state-of-the-art methods.

Mass spectrometry imaging (MS1) reveals the localization of a broad scale of compounds ranging from metabolites to proteins in biological tissues. This makes MS1. an attractive tool in biomedical research for studying diseases. Computer-aided diagnosis (CAD) systems facilitate the analysis of the molecular profile in tumor tissues to provide a distinctive fingerprint for finding biomarkers. In this paper, the performance of recurrent neural networks (RNNs) is studied on MSI data to exploit their learning capabilities for finding irregular patterns and dependencies in sequential data. In order to design a better CAD model for tumor detection/classification, several configurations of Long Short-Time Memory (LSTM) are examined. The proposed model consists of a 2-layer bidirectional LSTM, each containing 100 LSTM units. The proposed RNN model outperforms the state-of-the-art CNN model by 1.87% and 1.45% higher accuracy in mass spectra classification on lung and bladder cancer datasets with a sixfold faster training time.

Automated detection of cancer metastases in lymph nodes has the potential to improve the assessment of prognosis for patients. To enable fair comparison between the algorithms for this purpose, we set up the CAMELYON17 challenge in conjunction with the IEEE International Symposium on Biomedical Imaging 2017 Conference in Melbourne. Over 300 participants registered on the challenge website, of which 23 teams submitted a total of 37 algorithms before the initial deadline. Participants were provided with 899 whole-slide images (WSIs) for developing their algorithms. The developed algorithms were evaluated based on the test set encompassing 100 patients and 500 WSIs. The evaluation metric used was a quadratic weighted Cohen's kappa. We discuss the algorithmic details of the 10 best pre-conference and two post-conference submissions. All these participants used convolutional neural networks in combination with pre- and postprocessing steps. Algorithms differed mostly in neural network architecture, training strategy, and pre- and postprocessing methodology. Overall, the kappa metric ranged from 0.89 to -0.13 across all submissions. The best results were obtained with pre-trained architectures such as ResNet. Confusion matrix analysis revealed that all participants struggled with reliably identifying isolated tumor cells, the smallest type of metastasis, with detection rates below 40%. Qualitative inspection of the results of the top participants showed categories of false positives, such as nerves or contamination, which could be targeted for further optimization. Last, we show that simple combinations of the top algorithms result in higher kappa metric values than any algorithm individually, with 0.93 for the best combination.

Maritime vessel re-identification (re-ID) is a computer vision task of vessel identity matching across disjoint camera views. Prominent applications of vessel re-ID exist in the fields of surveillance and maritime traffic flow analysis. However, the field suffers from the absence of a large-scale dataset that enables training of deep learning models. In this study, we present a new dataset that includes 4614 images of 729 vessels along with 5-bin orientation and 8-class vessel-type annotations to promote further research. A second contribution of this study is the baseline re-ID analysis of our new dataset. Performances of 10 recent deep learning architectures are quantitatively compared to reveal the best practices. Lastly, we propose a novel multi-branch deep learning architecture, Maritime Vessel Re-ID network (MVR-net), to address the challenging problem of vessel re-ID. Evaluation of our approach on the new dataset yields 74.5% mAP and 77.9% Rank-1 score, providing a performance increase of 5.7% mAP and 5.0% Rank-1 over the best-performing baseline. MVR-net also outperforms the PRN (a pioneering vehicle re-ID network), by 2.9% and 4.3% higher mAP and Rank-1, respectively.

In this work, we propose and evaluate a pose-graph optimization-based real-time multi-sensor fusion framework for vehicle positioning using low-cost automotive-grade sensors. Pose-graphs can model multiple absolute and relative vehicle positioning sensor measurements and can be optimized using nonlinear techniques. We model pose-graphs using measurements from a precise stereo camera-based visual odometry system, a robust odometry system using the in-vehicle velocity and yaw-rate sensor, and an automotive-grade GNSS receiver. Our evaluation is based on a dataset with 180 km of vehicle trajectories recorded in highway, urban, and rural areas, accompanied by postprocessed Real-Time Kinematic GNSS as ground truth. We compare the architecture’s performance with (i) vehicle odometry and GNSS fusion and (ii) stereo visual odometry, vehicle odometry, and GNSS fusion, for offline and real-time optimization strategies. The results exhibit a 20.86% reduction in the localization error’s standard deviation and a significant reduction in outliers when compared with automotive-grade GNSS receivers.

This paper presents a vision-based collision-warning system for ADAS in intelligent vehicles, with a focus on urban scenarios. In most current systems, collision warnings are based on radar, or on monocular vision using pattern recognition. Since detecting collisions is a core functionality of intelligent vehicles, redundancy is essential, so that we explore the use of stereo vision. First, our approach is generic and class-agnostic, since it can detect general obstacles that are on a colliding path with the ego-vehicle without relying on semantic information. The framework estimates disparity and flow from a stereo video stream and calculates stixels. Then, the second contribution is the use of the new asteroids concept as a consecutive step. This step samples particles based on a probabilistic uncertainty analysis of the measurement process to model potential collisions. Third, this is all enclosed in a Bayesian histogram filter around a newly introduced time-to-collision versus angle-of-impact state space. The evaluation shows that the system correctly avoids any false warnings on the real-world KITTI dataset, detects all collisions in a newly simulated dataset when the obstacle is higher than 0.4m, and performs excellent on our new qualitative real-world data with near-collisions, both in daytime and nighttime conditions.

Barrett's Esophagus is a precursor of esophageal adenocarcinoma, one of the most lethal forms of cancer. Volumetric laser endomicroscopy (VLE) is a relatively new technology used for early detection of abnormal cells in BE by imaging the inner tissue layers of the esophagus. Computer-Aided Detection (CAD) shows great promise in analyzing the VLE frames due to the advances in deep learning. However, a full VLE scan produces 1,200 scans of 4,096 x 2,048 pixels, making automated pre-processing for the tissue of interest extraction necessary. This paper explores an object detection for tissue detection in VLE scans. We show that this can be achieved in real time with very low inference time, using single-stage object detection like YOLO. Our best performing model achieves a value of 98.23% for the mean average precision of bounding boxes correctly predicting the tissue of interest. Additionally, we have found that the tiny YOLO with Partial Residual Networks architecture further reduces the inference speed with a factor of 10, while only sacrificing less than 1% of accuracy. This proposed method does not only segment the tissue of interest in real time without any latency, but it can also achieve this efficiently using limited GPU resources, rendering it attractive for embedded applications. Our paper is the first to introduce object detection as a new approach for VLE-data tissue segmentation and paves the way for real-time VLE-based detection of early cancer in BE.

MR-guided high-intensity focused ultrasound (MR-HIFU) is a non-invasive therapeutic technology which has demonstrated clinical potential for tissue ablation. The application of this therapeutic approach facilitated to be a promising option to achieve faster pain palliation in patients with bone metastasis. However, its clinical adoption is still hampered by a lack of workflow integration. Currently, to ensure sufficient positioning, MR images have to be repeatedly acquired in between patient re-positioning tasks, leading to a time-consuming preparation phase of at least 30 minutes, involving extra costs and time to the available treatment time. Augmented reality (AR) is a promising technology that enables the fusion of medical images, such as MRI, with the view of an external camera. AR represents a valid tool for a faster localization and visualization of the lesion during positioning. The aim of this work is the implementation of a novel AR setup for accelerating the patient positioning during MRHIFU treatments by enabling adequate target positioning outside the MRI scanner. A marker-based approach was investigated for fusing the MR data with video data for providing an augmented view. Initial experiments on four volunteers show that MR images were overlaid on the camera views with an average re-projection error of 3.13 mm, which matches the clinical requirements for this specific application. It can be concluded that the implemented system is suitable for MR-HIFU procedures and supports its clinical adoption by improving the patient positioning, thereby offering potential for faster treatment time.

Autonomous vehicles are dependent on High Definition (HD) maps. The process of generating and updating these maps is slow, expensive, and not scalable for the whole world. Crowdsourcing vehicle sensor data to generate and update maps is a solution to the problem. In this paper, we propose and evaluate an end-to-end pose-graph optimization-based mapping framework using crowdsourced vehicle data. The in-vehicle data acquisition framework and the cloud-based mapping framework that fuses data from a consumer-grade Global Navigation Satellite System (GNSS) receiver, an odometry sensor, and a stereo camera is described in detail. We focus on using stereo image pairs for loop-closure detection to combine crowdsourced data from different sessions that are affected by GNSS biases. We evaluate our framework on a data-set of more than 180 km recorded around the Eindhoven area. After the map generation process, the results exhibit a 56.23% improvement in maximum offset error and a 24.39% improvement in precision around the loop-closure area.

A Dual-Energy CT (DECT) with a spectral detector greatly extends the capabilities of CT by incorporating energy-dependent information of the X-ray attenuation. In order to fully exploit DECT capabilities, it is required to perform a process known as spectral decomposition. However, this process is sensitive to noise, suffers from reduced photon count per layer in DECT scans and generates anti-correlated noise in the estimated material specific images. In order to overcome these problems, the Anti-Correlated Rudin, Osher and Fatemi (AC-ROF) model is applied for noise reduction, exploiting the relationship between the material-specific images. However, this model deteriorates the structural information with intense noise. In this paper we propose to extend this method by integrating it into an iterative reconstruction procedure to improve the noise reduction performance. The resulting algorithm is called Iterative Reconstruction AC-ROF, or IR-AC-ROF. We have tested AC-ROF and IR-AC-ROF algorithms with realistic brain simulation phantoms and show encouraging results indicating that the resulting material-specific images of IR-AC-ROF can generate better mono-energetic images with improved brain structure visibility. This demonstrates the benefit of including the noise reduction constraints within the reconstruction procedure, rather than using them in a post-processing step.

Background Minimally invasive spine surgery is dependent on accurate navigation. Computer-assisted navigation is increasingly used in minimally invasive surgery (MIS), but current solutions require the use of reference markers in the surgical field for both patient and instruments tracking. Purpose To improve reliability and facilitate clinical workflow, this study proposes a new marker-free tracking framework based on skin feature recognition. Methods Maximally Stable Extremal Regions (MSER) and Speeded Up Robust Feature (SURF) algorithms are applied for skin feature detection. The proposed tracking framework is based on a multi-camera setup for obtaining multi-view acquisitions of the surgical area. Features can then be accurately detected using MSER and SURF and afterward localized by triangulation. The triangulation error is used for assessing the localization quality in 3D. Results The framework was tested on a cadaver dataset and in eight clinical cases. The detected features for the entire patient datasets were found to have an overall triangulation error of 0.207 mm for MSER and 0.204 mm for SURF. The localization accuracy was compared to a system with conventional markers, serving as a ground truth. An average accuracy of 0.627 and 0.622 mm was achieved for MSER and SURF, respectively. Conclusions This study demonstrates that skin feature localization for patient tracking in a surgical setting is feasible. The technology shows promising results in terms of detected features and localization accuracy. In the future, the framework may be further improved by exploiting extended feature processing using modern optical imaging techniques for clinical applications where patient tracking is crucial.

Computer-aided detection (CAD) approaches have shown promising results for early esophageal cancer detection using Volumetric Laser Endoscopy (VLE) imagery. However, the relatively slow and computationally costly tissue segmentation employed in these approaches hamper their clinical applicability. In this paper, we propose to reframe the 2D tissue segmentation problem into a 1D tissue boundary detection problem. Instead of using an encoder-decoder architecture, we propose to follow the tissue boundary using a Recurrent Neural Network (RNN), exploiting the spatio-temporal relations within VLE frames. We demonstrate a near state-of-the-art performance using 18 times less floating point operations, enabling real-time execution in clinical practice.

Quantifying uncertainty in medical image segmentation applications is essential, as it is often connected to vital decision-making. Compelling attempts have been made in quantifying the uncertainty in image segmentation architectures, e.g. to learn a density segmentation model conditioned on the input image. Typical work in this field restricts these learnt densities to be strictly Gaussian. In this paper, we propose to use a more flexible approach by introducing Normalizing Flows (NFs), which enables the learnt densities to be more complex and facilitate more accurate modeling for uncertainty. We prove this hypothesis by adopting the Probabilistic U-Net and augmenting the posterior density with an NF, allowing it to be more expressive. Our qualitative as well as quantitative (GED and IoU) evaluations on the multi-annotated and single-annotated LIDC-IDRI and Kvasir-SEG segmentation datasets, respectively, show a clear improvement. This is mostly apparent in the quantification of aleatoric uncertainty and the increased predictive performance of up to 14%. This result strongly indicates that a more flexible density model should be seriously considered in architectures that attempt to capture segmentation ambiguity through density modeling. The benefit of this improved modeling will increase human confidence in annotation and segmentation, and enable eager adoption of the technology in practice.

Neurosurgical training is performed on human cadavers and simulation models, such as VR platforms, which have several drawbacks. Head phantoms could solve most of the issues related to these trainings. The aim of this study was to design a realistic and CT-compatible head phantom, with a specific focus on endo-nasal skull-base surgery and brain biopsy. A head phantom was created by segmenting an image dataset from a cadaver. The skull, which includes a complete structure of the nasal cavity and detailed skull-base anatomy, is 3D printed using PLA with calcium, while the brain is produced using a PVA mixture. The radiodensity and mechanical properties of the phantom were tested and adjusted in material choice to mimic real-life conditions. Surgeons find the skull, the structures at the skull-base and the brain realistically reproduced. The head phantom can be employed for neurosurgical education, training and surgical planning, and can be successfully used for simulating surgeries.

Predicting the future paths of pedestrians is shown to be crucial in various research domains, including intelligent vehicles and surveillance. Although numerous studies are conducted, the task has remained challenging, requiring a fundamental understanding of the complex human-human interactions. While current interaction-based frameworks are developed to benefit from information sharing between pedestrians, such frameworks are often ill-suited for monitoring the decisions-making process of pedestrians. In this paper, the problem of human-human interaction has been tackled by combining tools from multi-agent decision theory. Employing the game theory solution concept of Nash equilibrium, a novel equilibrium-based modification is proposed. With the aim of generalization, the proposed modification is applied to two different interaction-based frameworks and verified on publicly available datasets. Through these experiments, the advantages of the proposed modification are demonstrated.

The primary treatment for malignant brain tumors is surgical resection. While gross total resection improves the prognosis, a supratotal resection may result in neurological deficits. On the other hand, accurate intraoperative identification of the tumor boundaries may be very difficult, resulting in subtotal resections. Histological examination of biopsies can be used repeatedly to help achieve gross total resection but this is not practically feasible due to the turn-around time of the tissue analysis. Therefore, intraoperative techniques to recognize tissue types are investigated to expedite the clinical workflow for tumor resection and improve outcome by aiding in the identification and removal of the malignant lesion. Hyperspectral imaging (HSI) is an optical imaging technique with the power of extracting additional information from the imaged tissue. Because HSI images cannot be visually assessed by human observers, we instead exploit artificial intelligence techniques and leverage a Convolutional Neural Network (CNN) to investigate the potential of HSI in twelve in vivo specimens. The proposed framework consists of a 3D&ndash, 2D hybrid CNN-based approach to create a joint extraction of spectral and spatial information from hyperspectral images. A comparison study was conducted exploiting a 2D CNN, a 1D DNN and two conventional classification methods (SVM, and the SVM classifier combined with the 3D&ndash, 2D hybrid CNN) to validate the proposed network. An overall accuracy of 80% was found when tumor, healthy tissue and blood vessels were classified, clearly outperforming the state-of-the-art approaches. These results can serve as a basis for brain tumor classification using HSI, and may open future avenues for image-guided neurosurgical applications.

We present a novel training strategy that allows convolutional encoder-decoder networks, to complete partially observed data by means of hallucination. As input, it takes data from a partially observed domain, for which no complete ground truth is available, and data from an unpaired prior knowledge domain and trains the network in an end-to-end manner. This strategy is demonstrated for the task of completing 2-D road layouts as well as 3-D vehicle shapes. In contrast to alternative approaches, our strategy is compatible with networks that use skip connections, to improve detail in the completed output, while not requiring adversarial supervision. To demonstrate its benefits, our training strategy is benchmarked against two state-of-the-art baselines, one using a two-step auto-encoder training strategy and one using an adversarial strategy. Our novel strategy achieves an improvement up to +12% F-measure on the Cityscapes dataset. The learned network intrinsically generalizes better than the baselines on unseen datasets, which is demonstrated by an improvement up to +24% F-measure on the unseen KITTI dataset. Moreover, our approach outperforms the baselines using the same backbone network on the 3-D shape completion benchmark by reducing the Hamming distance with 15%.

Convolutional Neural Networks (CNNs) are reshaping signal processing and computer vision by providing data-driven solutions for inverse problems such as noise reduction. However, their relationship with established signal processing methods is sometimes unclear and its development not fully exploiting the existing knowledge. In this paper, rather than improving existing CNNs with wavelet transformations as explored earlier, we improve the wavelet shrinkage approach to noise-reduction with a data-driven solution. The resulting CNN has clear encoding, decoding and processing paths. As application, we perform noise reduction in Dual-Energy Cone- Beam CT. The obtained results were compared to a UNet-like architecture, which reveal better noise-free images without aliasing artifacts. This indicates that that our architecture is able to preserve well the information contained in the images because the architecture exploits explicitly the underlying signal representation.

In a cooperative automated driving scenario like platooning, the ego vehicle needs reliable and accurate perception capabilities to autonomously follow the lead vehicle. This paper presents the architecture design and development of an on-board stereo vision system for cooperative automated vehicles. The input to the proposed system is stereo image pairs. It uses three deep neural networks to detect and classify objects, lane markings, and free space boundary simultaneously in front of the ego vehicle. The rectified left and right image frames of the stereo camera are used to compute a disparity map to estimate the detected object’s depth and radial distance. It also estimates the object’s relative velocity, azimuth, and elevation angle with respect to the ego vehicle. It sends the perceived information to the vehicle control system and displays the perceived information in a meaningful way on the human-machine interface. The system runs on both PC (x86_64 architecture) with Nvidia GPU, and the Nvidia Drive PX 2 (aarch64 architecture) automotive-grade compute platform. It is deployed and evaluated on Renault Twizy cooperative automated driving research platform. The presented results show that the stereo vision system works in real-time and is useful for cooperative automated vehicles.

Biological neurons are known to have sparse and asynchronous communications using spikes. Despite our incomplete understanding of processing strategies of the brain, its low energy consumption in fulfilling delicate tasks suggests the existence of energy efficient mechanisms. Inspired by these key factors, we introduce SpArNet, a bio-inspired quantization scheme to convert a pre-trained convolutional neural network to a spiking neural network, with the aim of minimizing the computational load for execution on neuromorphic processors. The proposed scheme has significant advantages over the reference CNN in a reduced number of synaptic operations, and can be used for frequent executions of inference tasks. The computational load of SpArNet is adjusted to the spatio-temporal dynamics of the the input data. We have tested the converted network on two applications (autonomous steering and hand gesture recognition), demonstrating a significant reduction on the number of required synaptic operations.

Patients suffering from Barrett's Esophagus (BE) are at an increased risk of developing esophageal adenocarcinoma and early detection is crucial for a good prognosis. To aid the endoscopists with the early detection for this preliminary stage of esophageal cancer, this work concentrates on the development and extensive evaluation of a state-of-the-art computer-aided classification and localization algorithm for dysplastic lesions in BE. To this end, we have employed a large-scale endoscopic data set, consisting of 494,355 images, in combination with a novel semi-supervised learning algorithm to pretrain several instances of the proposed neural network architecture. Next, several Barrett-specific data sets that are increasingly closer to the target domain with significantly more data compared to other related work, were used in a multi-stage transfer learning strategy. Additionally, the algorithm was evaluated on two prospectively gathered external test sets and compared against 53 medical professionals. Finally, the model was also evaluated in a live setting without interfering with the current biopsy protocol. Results from the performed experiments show that the proposed model improves on the state-of-the-art on all measured metrics. More specifically, compared to the best performing state-of-the-art model, the specificity is improved by more than 20% points while simultaneously preserving high sensitivity and reducing the false positive rate substantially. Our algorithm yields similar scores on the localization metrics, where the intersection of all experts is correctly indicated in approximately 92% of the cases. Furthermore, the live pilot study shows great performance in a clinical setting with a patient level accuracy, sensitivity, and specificity of 90%. Finally, the proposed algorithm outperforms each individual medical expert by at least 5% and the average assessor by more than 10% over all assessor groups with respect to accuracy.

Transcatheter aortic valve implantation (TAVI) is the routine treatment worldwide for aortic valve stenosis in low-to high-risk patients. Assessing patient risk is essential to identify the most suitable candidates that could benefit from the procedure. Despite the broad use of statistical predictors in patient selection, current machine learning predictors have only been validated on retrospective data collected in single centers. Further, external validation is needed to assess the improvement in accuracy, which is offered by machine learning and deep learning techniques. In this study, we propose a finetuning approach for deep learning models by performing an inter-center cross-validation and finetuning technique, in order to improve the cross-validation accuracy results. We aimed to overcome data exchange and policy-related issues of two medical centers with a dedicated protocol, exploiting the exchange of deep learning models, data processing and validation steps which does not require any patient data sharing. The finetuning is based on the other center's data for further training of the initial model. After finetuning the model, we obtain an average AUC improvement of 13% and 7% with respect to the initial models. This research demonstrates that the predicting capabilities of deep learning models can be extended to and cross-validated with other centers, independent of limitations in data-sharing policies. Moreover, the study shows that finetuning can be exploited to considerably improve the accuracy of the prediction models.

Surgical navigation systems are increasingly used for complex spine procedures to avoid neurovascular injuries and minimize the risk for reoperations. Accurate patient tracking is one of the prerequisites for optimal motion compensation and navigation. Most current optical tracking systems use dynamic reference frames (DRFs) attached to the spine, for patient movement tracking. However, the spine itself is subject to intrinsic movements which can impact the accuracy of the navigation system. In this study, we aimed to detect the actual patient spine features in different image views captured by optical cameras, in an augmented reality surgical navigation (ARSN) system. Using optical images from open spinal surgery cases, acquired by two gray-scale cameras, spinal landmarks were identified and matched in different camera views. A computer vision framework was created for preprocessing of the spine images, detecting and matching local invariant image regions. We compared four feature detection algorithms, Speeded Up Robust Feature (SURF), Maximal Stable Extremal Region (MSER), Features from Accelerated Segment Test (FAST), and Oriented FAST and Rotated BRIEF (ORB) to elucidate the best approach. The framework was validated in 23 patients and the 3D triangulation error of the matched features was &lt, 0 . 5 mm. Thus, the findings indicate that spine feature detection can be used for accurate tracking in navigated surgery.

Vehicle re-identification plays a crucial role in the management of transportation infrastructure and traffic flow. However, this is a challenging task due to the large view-point variations in appearance, environmental and instance-related factors. Modern systems deploy CNNs to produce unique representations from the images of each vehicle instance. Most work focuses on leveraging new losses and network architectures to improve the descriptiveness of these representations. In contrast, our work concentrates on re-ranking and embedding expansion techniques. We propose an efficient approach for combining the outputs of multiple models at various scales while exploiting tracklet and neighbor information, called dual embedding expansion (DEx). Additionally, a comparative study of several common image retrieval techniques is presented in the context of vehicle re-ID. Our system yields competitive performance in the 2020 NVIDIA AI City Challenge with promising results. We demonstrate that DEx when combined with other re-ranking techniques, can produce an even larger gain without any additional attribute labels or manual supervision.

In spinal surgery, surgical navigation is an essential tool for safe intervention, including the placement of pedicle screws without injury to nerves and blood vessels. Commercially available systems typically rely on the tracking of a dynamic reference frame attached to the spine of the patient. However, the reference frame can be dislodged or obscured during the surgical procedure, resulting in loss of navigation. Hyperspectral imaging (HSI) captures a large number of spectral information bands across the electromagnetic spectrum, providing image information unseen by the human eye. We aim to exploit HSI to detect skin features in a novel methodology to track patient position in navigated spinal surgery. In our approach, we adopt two local feature detection methods, namely a conventional handcrafted local feature and a deep learning-based feature detection method, which are compared to estimate the feature displacement between different frames due to motion. To demonstrate the ability of the system in tracking skin features, we acquire hyperspectral images of the skin of 17 healthy volunteers. Deep-learned skin features are detected and localized with an average error of only 0.25 mm, outperforming the handcrafted local features with respect to the ground truth based on the use of optical markers.

Endoscopic diagnosis of early neoplasia in Barrett&rsquo, s Esophagus is generally a two-step process of primary detection in overview, followed by detailed inspection of any visible abnormalities using Narrow Band Imaging (NBI). However, endoscopists struggle with evaluating NBI-zoom imagery of subtle abnormalities. In this work, we propose the first results of a deep learning system for the characterization of NBI-zoom imagery of Barrett&rsquo, s Esophagus with an accuracy, sensitivity, and specificity of 83.6%, 83.1%, and 84.0%, respectively. We also show that endoscopy-driven pretraining outperforms two models, one without pretraining as well as a model with ImageNet initialization. The final model outperforms absence of pretraining by approximately 10% and the performance is 2% higher in terms of accuracy compared to ImageNet pretraining. Furthermore, the practical deployment of our model is not hampered by ImageNet licensing, thereby paving the way for clinical application.

Dual-Energy CT offers significant advantages over traditional CT imaging because it offers energy-based awareness of the image content and facilitates material discrimination in the projection domain. The Dual-Energy CT concept has intrinsic redundancy that can be used for improving image quality, by jointly exploiting the high- and low-energy projections. In this paper we focus on noise reduction. This work presents the novel noise-reduction algorithm Dual Energy Shifted Wavelet Denoising (DESWD), which renders high-quality Dual-Energy CBCT projections out of noisy ones. To do so, we first apply a Generalized Anscombe Transform, enabling us to use denoising methods proposed for Gaussian noise statistics. Second, we use a 3D transformation to denoise all the projections at once. Finally we exploit the inter-channel redundancy of the projections to create sparsity in the signal for better denoising with a channel-decorrelation step. Our simulation experiments show that DESWD performs better than a state-of-the-art denoising method (BM4D) in limited photon-count imaging, while BM4D achieves excellent results for less noisy conditions.

In this work, we present an end-to-end network for fast panoptic segmentation. This network, called Fast Panoptic Segmentation Network (FPSNet), does not require computationally costly instance mask predictions or rule-based merging operations. This is achieved by casting the panoptic task into a custom dense pixel-wise classification task, which assigns a class label or an instance id to each pixel. We evaluate FPSNet on the Cityscapes and Pascal VOC datasets, and find that FPSNet is faster than existing panoptic segmentation methods, while achieving better or similar panoptic segmentation performance. On the Cityscapes validation set, we achieve a Panoptic Quality score of 55.1%, at prediction times of 114 milliseconds for images with a resolution of 1024 × 2048 pixels. For lower resolutions of the Cityscapes dataset and for the Pascal VOC dataset, FPSNet achieves prediction times as low as 45 and 28 milliseconds, respectively.