Search

High dynamic range (HDR) imaging is a technique that allows an extensive dynamic range of exposures, which is important in image processing, computer graphics, and computer vision. In recent years, there has been a significant advancement in HDR imaging using deep learning (DL). This study conducts a comprehensive and insightful survey and analysis of recent developments in deep HDR imaging methodologies. We hierarchically and structurally group existing deep HDR imaging methods into five categories based on (1) number/domain of input exposures, (2) number of learning tasks, (3) novel sensor data, (4) novel learning strategies, and (5) applications. Importantly, we provide a constructive discussion on each category regarding its potential and challenges. Moreover, we review some crucial aspects of deep HDR imaging, such as datasets and evaluation metrics. Finally, we highlight some open problems and point out future research directions., Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), main and suppl. material

Event cameras sense brightness changes in each pixel and yield asynchronous event streams instead of producing intensity images. They have distinct advantages over conventional cameras, such as a high dynamic range (HDR) and no motion blur. To take advantage of event cameras with existing image-based algorithms, a few methods have been proposed to reconstruct images from event streams. However, the output images have a low resolution (LR) and are unrealistic. Low-quality outputs stem from broader applications of event cameras, where high-quality and high-resolution (HR) images are needed. In this work, we consider the problem of reconstructing and super-resolving images from LR events when no ground truth (GT) HR images and degradation models are available. We propose a novel end-to-end joint framework for single image reconstruction and super-resolution from LR event data. Our method is primarily unsupervised to handle the absence of real inputs from GT and deploys adversarial learning. To train our framework, we constructed an open dataset, including simulated events and real-world images. The use of the dataset boosts the network performance, and the network architectures and various loss functions in each phase help improve the quality of the resulting image. Various experiments showed that our method surpasses the state-of-the-art LR image reconstruction methods for real-world and synthetic datasets. The experiments for super-resolution (SR) image reconstruction also substantiate the effectiveness of the proposed method. We further extended our method to more challenging problems of HDR, sharp image reconstruction, and color events. In addition, we demonstrate that the reconstruction and super-resolution results serve as intermediate representations of events for high-level tasks, such as semantic segmentation, object recognition, and detection. We further examined how events affect the outputs of the three phases and analyze our method's efficacy through an ablation study.

Omni-directional images are becoming more prevalent for understanding the scene of all directions around a camera, as they provide a much wider field-of-view (FoV) compared to conventional images. In this work, we present a novel approach to represent omni-directional images and suggest how to apply CNNs on the proposed image representation. The proposed image representation method utilizes a spherical polyhedron to reduce distortion introduced inevitably when sampling pixels on a non-Euclidean spherical surface around the camera center. To apply convolution operation on our representation of images, we stack the neighboring pixels on top of each pixel and multiply with trainable parameters. This approach enables us to apply the same CNN architectures used in conventional Euclidean 2D images on our proposed method in a straightforward manner. Compared to the previous work, we additionally compare different designs of kernels that can be applied to our proposed method. We also show that our method outperforms in monocular depth estimation task compared to other state-of-the-art representation methods of omni-directional images. In addition, we propose a novel method to fit bounding ellipses of arbitrary orientation using object detection networks and apply it to an omni-directional real-world human detection dataset.

Event-based cameras sense changes in light intensity, and asynchronously outputs event data. Compared with conventional cameras, event-based cameras have the advantages of low latency, high dynamic range, and low power consumption. Event cameras have been utilized for various vision tasks, such as depth estimation and object detection under severe illumination changes and dynamic motion. Fisheye or omnidirectional cameras, on the other hand, have much wider field-of-view (FoV) allowing a more compact system for omnidirectional vision in comparison with multiple normal-FoV camera setups. In this work, we propose a new multi-view stereo method, called EOMVS, to reconstruct a 3D scene with a wide view using the event data captured by omnidirectional event cameras. The proposed method has the advantages of both event and fisheye cameras. To validate our EOMVS method, we conducted experiments using both synthetic and real-world datasets, and evaluated the performance both qualitatively and quantitatively. The experiments show that the presented approach can accurately obtain wide-view 3D information.

Most existing deep learning-based depth completion methods are only suitable for high ( e . g . 64-scanline) resolution LiDAR measurements, and they usually fail to predict a reliable dense depth map with low resolution (4, 8, or 16-scanline) LiDAR. However, it is of great interest to reduce the number of LiDAR channels in many aspects (cost, weight of a device, power consumption). In this letter, we propose a new depth completion framework with various LiDAR scanline resolutions, which performs as well as methods built for 64-scanline resolution LiDAR inputs. For this, we define a consistency loss between the predictions from LiDAR measurements of different scanline resolutions. ( i . e ., 4, 8, 16, 32-scanline LiDAR measurements) Also, we design a fusion module to integrate features from different modalities. Experiments show our proposed method outperforms the current state-of-the-art depth completion methods for input LiDAR measurements of low scanline resolution and performs comparably to the methods(models) for input LiDAR measurements of 64-scanline resolution on the KITTI benchmark dataset.

Owing to the recent advances in the field of deep-learning-based approaches, state-of-the-art performance has been achieved for optical flow estimation. However, non-deep-learning-based improvement in the optical flow estimation performance is still required because many platforms, such as small UGVs and drones, involve constraints that make mounting GPUs difficult. Thus, in this study, we do not apply deep learning methods; rather, we improve the accuracy of the optical flow pipeline by optimizing only its fundamental parameters. The optical flow estimation performance is influenced by the number of coarse-to-fine estimation pyramids, the filter applied to each pyramid level, the window size, and the graduated nonconvexity (GNC) step number. No significant differences from the previous research were achieved by optimizing the parameters heuristically because they have already been optimized. Therefore, we decided to change the filter applied in each pyramid, which is the most important factor in determining the optical flow estimation performance. As a result of verification, the optical flow with the median filter did not show good optical flow estimation performance due to oversmoothing of the image boundary, and the optical flow with the weighted median filter proposed to overcome this drawback could not well address the complexity of the equation and deal with the large computation cost. The proposed Hampel filter shows better performance by minimizing the loss of the original image in the image smoothing process and reduced computational complexity compared to the weighted median filter. The principle of the Hampel filter is similar to that of the median filter, except that if the reference pixel is statistically close to the pixel median in the window, it works by preserving the original pixel value. This is usually used for filtering outliers in a 1D signal but has never been applied for 2D image filters. This study expanded the application of the existing Hampel filter to include 2D images. Compared to the conventional median filter, this filter demonstrates relatively good performance, as it can considerably reduce the loss in the image data and improve the excessive edge smoothing effect such that it can better identify motion. The performance-related improvements achieved by this research were verified using the KITTI Vision Benchmark Suite, and the simulation results demonstrate that the Hampel filter can replace the existing filter and that it has strengths related to multi-object detection, background recognition, and the identification of moving objects at the edges of images.

Recent advances in deep neural networks (DNNs) have facilitated high-end applications, including holistic scene understanding (HSU), in which many tasks run in parallel with the same visual input. Following this trend, various methods have been proposed to use DNNs to perform multiple vision tasks. However, these methods are task-specific and less effective when considering multiple HSU tasks. End-to-end demonstrations of adversarial examples, which generate one-to-many heterogeneous adversarial examples in parallel from the same input, are scarce. Additionally, one-to-many mapping of adversarial examples for HSU usually requires joint representation learning and flexible constraints on magnitude, which can render the prevalent attack methods ineffective. In this paper, we propose PSAT-GAN, an end-to-end framework that follows the pipeline of HSU. It is based on a mixture of generative models and an adversarial classifier that employs partial weight sharing to learn a one-to-many mapping of adversarial examples in parallel, each of which is effective for its corresponding task in HSU attacks. PSAT-GAN is further enhanced by applying novel adversarial and soft-constraint losses to generate effective perturbations and avoid studying transferability. Experimental results indicate that our method is efficient in generating both universal and image-dependent adversarial examples to fool HSU tasks under either targeted or non-targeted settings.

Most of existing deep learning-based depth and optical flow estimation methods require the supervision of a lot of ground truth data, and hardly generalize to video frames, resulting in temporal inconsistency. In this letter, we propose a joint framework that estimates disparity and optical flow of stereo videos and generalizes across various video frames by considering the spatiotemporal relation between the disparity and flow without supervision. To improve both accuracy and consistency, we propose a loop consistency loss which enforces the spatiotemporal consistency of the estimated disparity and optical flow. Furthermore, we introduce a video-based training scheme using the c-LSTM to reinforce the temporal consistency. Extensive experiments show our proposed methods not only estimate disparity and optical flow accurately but also further improve spatiotemporal consistency. Our framework outperforms the state-of-the-art unsupervised depth and optical flow estimation models on the KITTI benchmark dataset.

We present a robust approach for computing disparity maps with a supervised learning-based confidence prediction. This approach takes into consideration following features. First, we analyze the characteristics of various confidence measures in the random forest framework to select effective confidence measures depending on the characteristics of the training data and matching strategies, such as similarity measures and parameters. We then train a random forest using the selected confidence measures to improve the efficiency of confidence prediction and to build a better prediction model. Second, we present a confidence-based matching cost modulation scheme, based on predicted confidence values, to improve the robustness and accuracy of the (semi-) global stereo matching algorithms. Finally, we apply the proposed modulation scheme to popularly used algorithms to make them robust against unexpected difficulties that could occur in an uncontrolled environment using challenging outdoor datasets. The proposed confidence measure selection and cost modulation schemes are experimentally verified from various perspectives using the KITTI and Middlebury datasets.

This paper addresses the multi-attributed graph matching problem, which considers multiple attributes jointly while preserving the characteristics of each attribute for graph matching. Since most of conventional graph matching algorithms integrate multiple attributes to construct a single unified attribute in an oversimplified manner, the information from multiple attributes is often not completely utilized. In order to solve this problem, we propose a novel multi-layer graph structure that can preserve the characteristics of each attribute in separated layers, and also propose a multi-attributed graph matching algorithm based on random walk centrality with the proposed multi-layer graph structure. We compare the proposed algorithm with other state-of-the-art graph matching algorithms based on a single-layer structure using synthetic and real data sets and demonstrate the superior performance of the proposed multi-layer graph structure and the multi-attributed graph matching algorithm.

Road surface profile (RSP) estimation is an important task to find the imperfections of a road surface, thereby improving ride quality. The RSP estimation has been recently studied using stereo vision owing to its affordable price. However, the existing methods provide noisy and temporally unstable results for real-world driving scenes because of noisy range measurements, noisy pitch angle estimates between the camera and the road surface, and interference from obstacles. This paper proposes a novel method for temporally consistent and robust RSP estimation to overcome these problems. The proposed method consists of three steps: free space estimation, digital elevation map (DEM) estimation, and RSP estimation. We first estimate a drivable area, i.e., free space. For robust and fast free space estimation, we propose an optimization-based non-parametric road surface modeling method and an integral disparity histogram-based free space estimation method. Then, the DEM of the road surface is estimated using range measurements on the free space to avoid obstacle interference. The DEM is temporally updated every frame using the moving average filter and the DEM reference grid update scheme. Owing to these strategies, the proposed method reduces the elevation estimation noise and the pitch angle error, and therefore provides the temporally consistent RSP. We demonstrate the superiority of the proposed method experimentally using stereo image sequences captured in real-world driving scenes.

A robust algorithm is proposed for tracking a target object in dynamic conditions including motion blurs, illumination changes, pose variations, and occlusions. To cope with these challenging factors, multiple trackers based on different feature representations are integrated within a probabilistic framework. Each view of the proposed multiview (multi-channel) feature learning algorithm is concerned with one particular feature representation of a target object from which a tracker is developed with different levels of reliability. With the multiple trackers, the proposed algorithm exploits tracker interaction and selection for robust tracking performance. In the tracker interaction, a transition probability matrix is used to estimate dependencies between trackers. Multiple trackers communicate with each other by sharing information of sample distributions. The tracker selection process determines the most reliable tracker with the highest probability. To account for object appearance changes, the transition probability matrix and tracker probability are updated in a recursive Bayesian framework by reflecting the tracker reliability measured by a robust tracker likelihood function that learns to account for both transient and stable appearance changes. Experimental results on benchmark datasets demonstrate that the proposed interacting multiview algorithm performs robustly and favorably against state-of-the-art methods in terms of several quantitative metrics.

Recent achievement of the learning-based classification leads to the noticeable performance improvement in automatic polyp detection. Here, building large good datasets is very crucial for learning a reliable detector. However, it is practically challenging due to the diversity of polyp types, expensive inspection, and labor-intensive labeling tasks. For this reason, the polyp datasets usually tend to be imbalanced, i.e., the number of non-polyp samples is much larger than that of polyp samples, and learning with those imbalanced datasets results in a detector biased toward a non-polyp class. In this paper, we propose a data sampling-based boosting framework to learn an unbiased polyp detector from the imbalanced datasets. In our learning scheme, we learn multiple weak classifiers with the datasets rebalanced by up/down sampling, and generate a polyp detector by combining them. In addition, for enhancing discriminability between polyps and non-polyps that have similar appearances, we propose an effective feature learning method using partial least square analysis, and use it for learning compact and discriminative features. Experimental results using challenging datasets show obvious performance improvement over other detectors. We further prove effectiveness and usefulness of the proposed methods with extensive evaluation.

In this paper, we consider a multiobject tracking problem in complex scenes. Unlike batch tracking systems using detections of the entire sequence, we propose a novel online multiobject tracking system in order to build tracks sequentially using online provided detections. To track objects robustly even under frequent occlusions, the proposed system consists of three main parts: 1) visual tracking with a novel data association with a track existence probability by associating online detections with the corresponding tracks under partial occlusions; 2) track management to associate terminated tracks for linking tracks fragmented by long-term occlusions; and 3) online model learning to generate discriminative appearance models for successful associations in other two parts. Experimental results using challenging public data sets show the obvious performance improvement of the proposed system, compared with other state-of-the-art tracking systems. Furthermore, extensive performance analysis of the three main parts demonstrates effects and usefulness of the each component for multiobject tracking.

Depth map up-sampling methods have achieved remarkable improvement by exploiting sensor fusion techniques where they assume that the depth map discontinuities and image edges coincide, and the depth values of the temporal neighbors are stable during time variation. However, inherent noise of depth data acquired by active range sensors often violates these assumptions, and results in undesirable error propagation. To alleviate the error propagation, this letter presents a new adaptive supporting method of spatially-temporally neighboring samples. On the basis of a spatial-temporal Markov random field model, the weight coefficients of the smoothness terms are adaptively computed according to the reliability of neighboring samples. The experiments show that the proposed method outperforms the previous works in terms of quantitative and qualitative criteria.

We present a new window-based method for correspondence search using varying support-weights. We adjust the support-weights of the pixels in a given support window based on color similarity and geometric proximity to reduce the image ambiguity. Our method outperforms other local methods on standard stereo benchmarks.