Search

As interests in air quality monitoring related to environmental pollution and industrial safety increase, demands for gas sensors are rapidly increasing. Among various gas sensor types, the semiconductor metal oxide (SMO)-type sensor has advantages of high sensitivity, low cost, mass production, and small size but suffers from poor selectivity. To solve this problem, electronic nose (e-nose) systems using a gas sensor array and pattern recognition are widely used. However, as the number of sensors in the e-nose system increases, total power consumption also increases. In this study, an ultra-low-power e-nose system was developed using ultraviolet (UV) micro-LED (μLED) gas sensors and a convolutional neural network (CNN). A monolithic photoactivated gas sensor was developed by depositing a nanocolumnar In

High dynamic range (HDR) imaging is a technique that allows an extensive dynamic range of exposures, which is important in image processing, computer graphics, and computer vision. In recent years, there has been a significant advancement in HDR imaging using deep learning (DL). This study conducts a comprehensive and insightful survey and analysis of recent developments in deep HDR imaging methodologies. We hierarchically and structurally group existing deep HDR imaging methods into five categories based on (1) number/domain of input exposures, (2) number of learning tasks, (3) novel sensor data, (4) novel learning strategies, and (5) applications. Importantly, we provide a constructive discussion on each category regarding its potential and challenges. Moreover, we review some crucial aspects of deep HDR imaging, such as datasets and evaluation metrics. Finally, we highlight some open problems and point out future research directions., Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), main and suppl. material

Event cameras sense brightness changes in each pixel and yield asynchronous event streams instead of producing intensity images. They have distinct advantages over conventional cameras, such as a high dynamic range (HDR) and no motion blur. To take advantage of event cameras with existing image-based algorithms, a few methods have been proposed to reconstruct images from event streams. However, the output images have a low resolution (LR) and are unrealistic. Low-quality outputs stem from broader applications of event cameras, where high-quality and high-resolution (HR) images are needed. In this work, we consider the problem of reconstructing and super-resolving images from LR events when no ground truth (GT) HR images and degradation models are available. We propose a novel end-to-end joint framework for single image reconstruction and super-resolution from LR event data. Our method is primarily unsupervised to handle the absence of real inputs from GT and deploys adversarial learning. To train our framework, we constructed an open dataset, including simulated events and real-world images. The use of the dataset boosts the network performance, and the network architectures and various loss functions in each phase help improve the quality of the resulting image. Various experiments showed that our method surpasses the state-of-the-art LR image reconstruction methods for real-world and synthetic datasets. The experiments for super-resolution (SR) image reconstruction also substantiate the effectiveness of the proposed method. We further extended our method to more challenging problems of HDR, sharp image reconstruction, and color events. In addition, we demonstrate that the reconstruction and super-resolution results serve as intermediate representations of events for high-level tasks, such as semantic segmentation, object recognition, and detection. We further examined how events affect the outputs of the three phases and analyze our method's efficacy through an ablation study.

Event cameras asynchronously output the polarity values of pixel-level log intensity alterations. They are robust against motion blur and can be adopted in challenging light conditions. Owing to these advantages, event cameras have been employed in various vision tasks such as depth estimation, visual odometry, and object detection. In particular, event cameras are effective in stereo depth estimation to find correspondence points between two cameras under challenging illumination conditions and/or fast motion. However, because event cameras provide spatially sparse event stream data, it is difficult to obtain a dense disparity map. Although it is possible to estimate disparity from event data at the edge of a structure where intensity changes are likely to occur, estimating the disparity in a region where event occurs rarely is challenging. In this study, we propose a deep network that combines the features of an image with the features of an event to generate a dense disparity map. The proposed network uses images to obtain spatially dense features that are lacking in events. In addition, we propose a spatial multi-scale correlation between two fused feature maps for an accurate disparity map. To validate our method, we conducted experiments using synthetic and real-world datasets.

Electronic nose (e-nose) technology for selectively identifying a target gas through chemoresistive sensors has gained much attention for various applications, such as smart factory and personal health monitoring. To overcome the cross-reactivity problem of chemoresistive sensors to various gas species, herein, we propose a novel sensing strategy based on a single micro-LED (μLED)-embedded photoactivated (μLP) gas sensor, utilizing the time-variant illumination for identifying the species and concentrations of various target gases. A fast-changing pseudorandom voltage input is applied to the μLED to generate forced transient sensor responses. A deep neural network is employed to analyze the obtained complex transient signals for gas detection and concentration estimation. The proposed sensor system achieves high classification (~96.99%) and quantification (mean absolute percentage error ~ 31.99%) accuracies for various toxic gases (methanol, ethanol, acetone, and nitrogen dioxide) with a single gas sensor consuming 0.53 mW. The proposed method may significantly improve the efficiency of e-nose technology in terms of cost, space, and power consumption.

Omni-directional images are becoming more prevalent for understanding the scene of all directions around a camera, as they provide a much wider field-of-view (FoV) compared to conventional images. In this work, we present a novel approach to represent omni-directional images and suggest how to apply CNNs on the proposed image representation. The proposed image representation method utilizes a spherical polyhedron to reduce distortion introduced inevitably when sampling pixels on a non-Euclidean spherical surface around the camera center. To apply convolution operation on our representation of images, we stack the neighboring pixels on top of each pixel and multiply with trainable parameters. This approach enables us to apply the same CNN architectures used in conventional Euclidean 2D images on our proposed method in a straightforward manner. Compared to the previous work, we additionally compare different designs of kernels that can be applied to our proposed method. We also show that our method outperforms in monocular depth estimation task compared to other state-of-the-art representation methods of omni-directional images. In addition, we propose a novel method to fit bounding ellipses of arbitrary orientation using object detection networks and apply it to an omni-directional real-world human detection dataset.

Event-based cameras sense changes in light intensity, and asynchronously outputs event data. Compared with conventional cameras, event-based cameras have the advantages of low latency, high dynamic range, and low power consumption. Event cameras have been utilized for various vision tasks, such as depth estimation and object detection under severe illumination changes and dynamic motion. Fisheye or omnidirectional cameras, on the other hand, have much wider field-of-view (FoV) allowing a more compact system for omnidirectional vision in comparison with multiple normal-FoV camera setups. In this work, we propose a new multi-view stereo method, called EOMVS, to reconstruct a 3D scene with a wide view using the event data captured by omnidirectional event cameras. The proposed method has the advantages of both event and fisheye cameras. To validate our EOMVS method, we conducted experiments using both synthetic and real-world datasets, and evaluated the performance both qualitatively and quantitatively. The experiments show that the presented approach can accurately obtain wide-view 3D information.

Most existing deep learning-based depth completion methods are only suitable for high ( e . g . 64-scanline) resolution LiDAR measurements, and they usually fail to predict a reliable dense depth map with low resolution (4, 8, or 16-scanline) LiDAR. However, it is of great interest to reduce the number of LiDAR channels in many aspects (cost, weight of a device, power consumption). In this letter, we propose a new depth completion framework with various LiDAR scanline resolutions, which performs as well as methods built for 64-scanline resolution LiDAR inputs. For this, we define a consistency loss between the predictions from LiDAR measurements of different scanline resolutions. ( i . e ., 4, 8, 16, 32-scanline LiDAR measurements) Also, we design a fusion module to integrate features from different modalities. Experiments show our proposed method outperforms the current state-of-the-art depth completion methods for input LiDAR measurements of low scanline resolution and performs comparably to the methods(models) for input LiDAR measurements of 64-scanline resolution on the KITTI benchmark dataset.

Owing to the recent advances in the field of deep-learning-based approaches, state-of-the-art performance has been achieved for optical flow estimation. However, non-deep-learning-based improvement in the optical flow estimation performance is still required because many platforms, such as small UGVs and drones, involve constraints that make mounting GPUs difficult. Thus, in this study, we do not apply deep learning methods; rather, we improve the accuracy of the optical flow pipeline by optimizing only its fundamental parameters. The optical flow estimation performance is influenced by the number of coarse-to-fine estimation pyramids, the filter applied to each pyramid level, the window size, and the graduated nonconvexity (GNC) step number. No significant differences from the previous research were achieved by optimizing the parameters heuristically because they have already been optimized. Therefore, we decided to change the filter applied in each pyramid, which is the most important factor in determining the optical flow estimation performance. As a result of verification, the optical flow with the median filter did not show good optical flow estimation performance due to oversmoothing of the image boundary, and the optical flow with the weighted median filter proposed to overcome this drawback could not well address the complexity of the equation and deal with the large computation cost. The proposed Hampel filter shows better performance by minimizing the loss of the original image in the image smoothing process and reduced computational complexity compared to the weighted median filter. The principle of the Hampel filter is similar to that of the median filter, except that if the reference pixel is statistically close to the pixel median in the window, it works by preserving the original pixel value. This is usually used for filtering outliers in a 1D signal but has never been applied for 2D image filters. This study expanded the application of the existing Hampel filter to include 2D images. Compared to the conventional median filter, this filter demonstrates relatively good performance, as it can considerably reduce the loss in the image data and improve the excessive edge smoothing effect such that it can better identify motion. The performance-related improvements achieved by this research were verified using the KITTI Vision Benchmark Suite, and the simulation results demonstrate that the Hampel filter can replace the existing filter and that it has strengths related to multi-object detection, background recognition, and the identification of moving objects at the edges of images.

Event cameras have numerous advantages over traditional cameras, such as low latency, high temporal resolution, and high dynamic range (HDR). We initially investigate the potential of creating intensity images/videos from an adjustable portion of the event data stream via event-based conditional generative adversarial networks (cGANs). Using the proposed framework, we further show the versatility of our method in directly handling similar supervised tasks, such as optical flow and depth prediction. Stacks of space-time coordinates of events are used as the inputs while the proposed framework is trained to predict either the intensity images, optical flows, or depth outputs according to the target task. We further demonstrate the unique capability of our approach in generating HDR images even under extreme illumination conditions, creating non-blurred images under rapid motion, and generating very high frame rate videos up to the temporal resolution of event cameras. The proposed framework is evaluated using a publicly available real-world dataset and a synthetic dataset we prepared by utilizing an event camera simulator.

Recent advances in deep neural networks (DNNs) have facilitated high-end applications, including holistic scene understanding (HSU), in which many tasks run in parallel with the same visual input. Following this trend, various methods have been proposed to use DNNs to perform multiple vision tasks. However, these methods are task-specific and less effective when considering multiple HSU tasks. End-to-end demonstrations of adversarial examples, which generate one-to-many heterogeneous adversarial examples in parallel from the same input, are scarce. Additionally, one-to-many mapping of adversarial examples for HSU usually requires joint representation learning and flexible constraints on magnitude, which can render the prevalent attack methods ineffective. In this paper, we propose PSAT-GAN, an end-to-end framework that follows the pipeline of HSU. It is based on a mixture of generative models and an adversarial classifier that employs partial weight sharing to learn a one-to-many mapping of adversarial examples in parallel, each of which is effective for its corresponding task in HSU attacks. PSAT-GAN is further enhanced by applying novel adversarial and soft-constraint losses to generate effective perturbations and avoid studying transferability. Experimental results indicate that our method is efficient in generating both universal and image-dependent adversarial examples to fool HSU tasks under either targeted or non-targeted settings.

Most of existing deep learning-based depth and optical flow estimation methods require the supervision of a lot of ground truth data, and hardly generalize to video frames, resulting in temporal inconsistency. In this letter, we propose a joint framework that estimates disparity and optical flow of stereo videos and generalizes across various video frames by considering the spatiotemporal relation between the disparity and flow without supervision. To improve both accuracy and consistency, we propose a loop consistency loss which enforces the spatiotemporal consistency of the estimated disparity and optical flow. Furthermore, we introduce a video-based training scheme using the c-LSTM to reinforce the temporal consistency. Extensive experiments show our proposed methods not only estimate disparity and optical flow accurately but also further improve spatiotemporal consistency. Our framework outperforms the state-of-the-art unsupervised depth and optical flow estimation models on the KITTI benchmark dataset.

An event camera reports per-pixel intensity differences as an asynchronous stream of events with low latency, high dynamic range (HDR), and low power consumption. This stream of sparse/dense events limits the direct use of well-known computer vision applications for event cameras. Further applications of event streams to vision tasks that are sensitive to image quality issues, such as spatial resolution and blur, e.g., object detection, would benefit from a higher resolution of image reconstruction. Moreover, despite the recent advances in spatial resolution in event camera hardware, the majority of commercially available event cameras still have relatively low spatial resolutions when compared to conventional cameras. We propose an end-to-end recurrent network to reconstruct high-resolution, HDR, and temporally consistent grayscale or color frames directly from the event stream, and extend it to generate temporally consistent videos. We evaluate our algorithm on real-world and simulated sequences and verify that it reconstructs fine details of the scene, outperforming previous methods in quantitative quality measures. We further investigate how to (1) incorporate active pixel sensor frames (produced by an event camera) and events together in a complementary setting and (2) reconstruct images iteratively to create an even higher quality and resolution in the images.

A widely used approach for estimating camera orientation is to use the points at infinity, i.e., the vanishing points (VPs). Enforcement of the orthogonal constraint between the VPs, known as the Manhattan world constraint, enables an estimation of the drift-free camera orientation to be achieved. However, in practical applications, this approach is neither effective (because of noisy parallel line segments) nor performable in non-Manhattan world scenes. To overcome these limitations, we propose a novel method that jointly estimates the VPs and camera orientation based on sequential Bayesian filtering. The proposed method does not require the Manhattan world assumption, and can perform a highly accurate estimation of camera orientation. In order to enhance the robustness of the joint estimation, we propose a keyframe-based feature management technique that removes false positives from parallel line clusters and detects new parallel line sets using geometric properties such as the orthogonality and rotational dependence for a VP, a line, and the camera rotation. In addition, we propose a 3-line camera rotation estimation method that does not require the Manhattan world assumption. The 3-line method is applied to the RANSAC-based outlier rejection technique to eliminate outlier measurements; therefore, the proposed method achieves accurate and robust estimation of the camera orientation and VPs in general scenes with non-orthogonal parallel lines. We demonstrate the superiority of the proposed method by conducting an extensive evaluation using synthetic and real datasets and by comparison with other state-of-the-art methods.

In this paper, we propose a novel distance-based camera network topology inference method for efficient person re-identification. To this end, we first calibrate each camera and estimate relative scales between cameras. Using the calibration results of multiple cameras, we calculate the speed of each person and infer the distance between cameras to generate distance-based camera network topology. The proposed distance-based topology can be applied adaptively to each person according to its speed and handle diverse transition time of people between non-overlapping cameras. To validate the proposed method, we tested the proposed method using an open person re-identification dataset and compared to state-of-the-art methods. The experimental results show that the proposed method is effective for person re-identification in the large-scale camera network with various people transition time., Comment: 10 pages, 11 figures

We present a robust approach for computing disparity maps with a supervised learning-based confidence prediction. This approach takes into consideration following features. First, we analyze the characteristics of various confidence measures in the random forest framework to select effective confidence measures depending on the characteristics of the training data and matching strategies, such as similarity measures and parameters. We then train a random forest using the selected confidence measures to improve the efficiency of confidence prediction and to build a better prediction model. Second, we present a confidence-based matching cost modulation scheme, based on predicted confidence values, to improve the robustness and accuracy of the (semi-) global stereo matching algorithms. Finally, we apply the proposed modulation scheme to popularly used algorithms to make them robust against unexpected difficulties that could occur in an uncontrolled environment using challenging outdoor datasets. The proposed confidence measure selection and cost modulation schemes are experimentally verified from various perspectives using the KITTI and Middlebury datasets.

Visual-inertial odometry (VIO) is the process of estimating ego-motion using a camera and the inertial measurement unit (IMU). It shows outstanding performance in estimating the ego-motion of a vehicle at the absolute scale thanks to the gyroscope and the accelerometer. However, VIO has some difficulties in estimating the translation in large-scale outdoor environments because the feature points along the motion direction and distant feature points in the images can cause degenerate situations. To resolve these difficulties, the authors propose to infer the confidence measures of the feature points and appropriately incorporate them into the Kalman filter-based VIO. The confidence is computed from the motion direction and the displacements of the tracked feature points under authors' urban canyon prior. It is applied in situations where the camera is moving forward towards the measurement noise covariance of the Kalman filter for ego-motion estimation. Experimental results on the public KITTI dataset show that VIO outperforms monocular and stereo-visual odometries, and the proposed VIO with confidence analysis achieves a translation error of 1.82% and a rotation error of 0.0018 deg/m.

We propose an approach to compute dense disparity maps that takes the characteristics of man-made environments into account. The key contribution is to generate a piecewise planar disparity map while preventing the oversimplification problem in non-planar regions. To achieve this, we decompose the stereo matching problem into three sequential subproblems: initial disparity map estimation, plane hypotheses generation, and global optimization with plane hypotheses. After finding an initial disparity map, we find local and global plane hypotheses from the disparity map through segmentation-based local plane fitting, agglomerative hierarchical clustering, and energy-based multi-model fitting techniques. We then estimate a disparity map that is a mixture of over-parameterized and scalar disparity values while identifying unreliable pixels in an energy minimization framework; disparity values in planar regions are parameterized as a plane, disparity values in non-planar regions are represented as scalar, and unreliable pixels are marked as outliers. As a post-processing step, we perturb assigned plane parameters as well as scalar disparity values. We experimentally verify the proposed method using publicly available benchmarks and various stereo matching algorithms.

Vehicle Re-identification (Re-ID) aims to retrieve all instances of query vehicle images present in an image pool. However viewpoint, illumination, and occlusion variations along with subtle differences between two unique images pose a significant challenge towards achieving an effective system. In this paper, we emphasize upon enhancing the performance of visual feature based ReID system by improving feature embedding quality and propose (1) an attention-guided hierarchical feature extractor (HFE) that leverages the structure of a backbone CNN to extract coarse and fine-grained features and (2) to train the proposed network within a hard negative adversarial framework that generates samples exhibiting extreme variations, encouraging the network to extract important distinguishing features across varying scales. To demonstrate the effectiveness of the proposed framework we use VERI-Wild, VRIC and Veri-776 datasets that exhibit extreme intra-class and minute inter-class differences and achieve state-of-the-art (SoTA) performance. Codes related to this paper are publicly available at https://github.com/PS06/VReID.

Event cameras sense per-pixel intensity changes and produce asynchronous event streams with high dynamic range and less motion blur, showing advantages over conventional cameras. A hurdle of training event-based models is the lack of large qualitative labeled data. Prior works learning end-tasks mostly rely on labeled or pseudo-labeled datasets obtained from the active pixel sensor (APS) frames; however, such datasets' quality is far from rivaling those based on the canonical images. In this paper, we propose a novel approach, called \textbf{EvDistill}, to learn a student network on the unlabeled and unpaired event data (target modality) via knowledge distillation (KD) from a teacher network trained with large-scale, labeled image data (source modality). To enable KD across the unpaired modalities, we first propose a bidirectional modality reconstruction (BMR) module to bridge both modalities and simultaneously exploit them to distill knowledge via the crafted pairs, causing no extra computation in the inference. The BMR is improved by the end-tasks and KD losses in an end-to-end manner. Second, we leverage the structural similarities of both modalities and adapt the knowledge by matching their distributions. Moreover, as most prior feature KD methods are uni-modality and less applicable to our problem, we propose to leverage an affinity graph KD loss to boost the distillation. Our extensive experiments on semantic segmentation and object recognition demonstrate that EvDistill achieves significantly better results than the prior works and KD with only events and APS frames., Comment: CVPR 2021 (updated references in this version)

A neuromorphic module of an electronic nose (E-nose) is demonstrated by hybridizing a chemoresistive gas sensor made of a semiconductor metal oxide (SMO) and a single transistor neuron (1T-neuron) made of a metal-oxide-semiconductor field-effect transistor (MOSFET). By mimicking a biological olfactory neuron, it simultaneously detects a gas and encoded spike signals for in-sensor neuromorphic functioning. It identifies an odor source by analyzing the complicated mixed signals using a spiking neural network (SNN). The proposed E-nose does not require conversion circuits, which are essential for processing the sensory signals between the sensor array and processors in the conventional bulky E-nose. In addition, they do not have to include a central processing unit (CPU) and memory, which are required for von Neumann computing. The spike transmission of the biological olfactory system, which is known to be the main factor for reducing power consumption, is realized with the SNN for power savings compared to the conventional E-nose with a deep neural network (DNN). Therefore, the proposed neuromorphic E-nose is promising for application to Internet of Things (IoT), which demands a highly scalable and energy-efficient system. As a practical example, it is employed as an electronic sommelier by classifying different types of wines.

Omni-directional images are becoming more prevalent for understanding the scene of all directions around a camera, as they provide a much wider field-of-view (FoV) compared to conventional images. In this work, we present a novel approach to represent omni-directional images and suggest how to apply CNNs on the proposed image representation. The proposed image representation method utilizes a spherical polyhedron to reduce distortion introduced inevitably when sampling pixels on a non-Euclidean spherical surface around the camera center. To apply convolution operation on our representation of images, we stack the neighboring pixels on top of each pixel and multiply with trainable parameters. This approach enables us to apply the same CNN architectures used in conventional euclidean 2D images on our proposed method in a straightforward manner. Compared to the previous work, we additionally compare different designs of kernels that can be applied to our proposed method. We also show that our method outperforms in monocular depth estimation task compared to other state-of-the-art representation methods of omni-directional images. In addition, we propose a novel method to fit bounding ellipses of arbitrary orientation using object detection networks and apply it to an omni-directional real-world human detection dataset.

Anchor boxes act as potential object localization candidates allow single-stage detectors to achieve real-time performance, at the cost of localization accuracy when compared to state-of-the-art two-stage detectors. Therefore, correct selection of the scale and aspect ratio associated with an anchor box is crucial for detector performance. In this work, we propose a novel architecture called DANet for improving the localization performance of single-stage object detectors, while maintaining real-time inference. The proposed network achieves this by predicting (1) the combination of aspect ratio and scale per feature map based on object density and (2) localization confidence per anchor box. We evaluate the proposed network using the benchmark dataset. On the MS COCO dataset, DANet achieves 30.9% AP at 51.8 fps using ResNet-18 and 45.3% AP at 7.4 fps using ResNeXt-101. The code and models will be available at https://github.com/PS06/AnchorNet.

Deep neural models in recent years have been successful in almost every field, including extremely complex problem statements. However, these models are huge in size, with millions (and even billions) of parameters, thus demanding more heavy computation power and failing to be deployed on edge devices. Besides, the performance boost is highly dependent on redundant labeled data. To achieve faster speeds and to handle the problems caused by the lack of data, knowledge distillation (KD) has been proposed to transfer information learned from one model to another. KD is often characterized by the so-called `Student-Teacher' (S-T) learning framework and has been broadly applied in model compression and knowledge transfer. This paper is about KD and S-T learning, which are being actively studied in recent years. First, we aim to provide explanations of what KD is and how/why it works. Then, we provide a comprehensive survey on the recent progress of KD methods together with S-T frameworks typically for vision tasks. In general, we consider some fundamental questions that have been driving this research area and thoroughly generalize the research progress and technical details. Additionally, we systematically analyze the research status of KD in vision applications. Finally, we discuss the potentials and open challenges of existing methods and prospect the future directions of KD and S-T learning., Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence(TPAMI),2021. Some references are updated in this version

Deep neural networks (DNNs) have achieved impressive performance on handling computer vision problems, however, it has been found that DNNs are vulnerable to adversarial examples. For such reason, adversarial perturbations have been recently studied in several respects. However, most previous works have focused on image classification tasks, and it has never been studied regarding adversarial perturbations on Image-to-image (Im2Im) translation tasks, showing great success in handling paired and/or unpaired mapping problems in the field of autonomous driving and robotics. This paper examines different types of adversarial perturbations that can fool Im2Im frameworks for autonomous driving purpose. We propose both quasi-physical and digital adversarial perturbations that can make Im2Im models yield unexpected results. We then empirically analyze these perturbations and show that they generalize well under both paired for image synthesis and unpaired settings for style transfer. We also validate that there exist some perturbation thresholds over which the Im2Im mapping is disrupted or impossible. The existence of these perturbations reveals that there exist crucial weaknesses in Im2Im models. Lastly, we show that our methods illustrate how perturbations affect the quality of outputs, pioneering the improvement of the robustness of current SOTA networks for autonomous driving., 8pages, Accepted on IEEE Robotics and Automation Letters (RAL)

Event cameras sense intensity changes and have many advantages over conventional cameras. To take advantage of event cameras, some methods have been proposed to reconstruct intensity images from event streams. However, the outputs are still in low resolution (LR), noisy, and unrealistic. The low-quality outputs stem broader applications of event cameras, where high spatial resolution (HR) is needed as well as high temporal resolution, dynamic range, and no motion blur. We consider the problem of reconstructing and super-resolving intensity images from LR events, when no ground truth (GT) HR images and down-sampling kernels are available. To tackle the challenges, we propose a novel end-to-end pipeline that reconstructs LR images from event streams, enhances the image qualities and upsamples the enhanced images, called EventSR. For the absence of real GT images, our method is primarily unsupervised, deploying adversarial learning. To train EventSR, we create an open dataset including both real-world and simulated scenes. The use of both datasets boosts up the network performance, and the network architectures and various loss functions in each phase help improve the image qualities. The whole pipeline is trained in three phases. While each phase is mainly for one of the three tasks, the networks in earlier phases are fine-tuned by respective loss functions in an end-to-end manner. Experimental results show that EventSR reconstructs high-quality SR images from events for both simulated and real-world data., Comment: Accepted by CVPR 2020

Most existing approaches for single image super-resolution (SISR) resort to quality low-high resolution (LR-HR) pairs and available degradation kernels to train networks for a specific task in hand in a fully supervised manner. Labeled data used for training are, however, usually limited in terms of the quantity and the diversity degradation kernels. The learned SR networks with one degradation kernel (e.g., bicubic) do not generalize well and their performance sharply deteriorates on other kernels (e.g., blurred or noise). In this paper, we address the critical challenge for SISR: limited labeled LR images and degradation kernels. We propose a novel Semi-supervised Student-Teacher Super-Resolution approach called S 2 TSR that super-resolves both labelled and unlabeled LR images via adversarial learning. To better exploit the information from labeled LR images, we propose a student-teacher framework (S-T) via knowledge transfer from supervised learning (T) to unsupervised learning (S). Specifically, the S-T knowledge transfer is based on a shared SR network, partial weight sharing of dual discriminators, and a pair matching network which also plays as a ‘latent discriminator’. Lastly, to learn better features from the limited labeled LR images, we propose a new SR network via non-local and attention mechanisms. Experiments demonstrate that our approach substantially improves unsupervised methods and performs favorably over fully supervised methods.

An event camera detects per-pixel intensity difference and produces asynchronous event stream with low latency, high dynamic range, and low power consumption. As a trade-off, the event camera has low spatial resolution. We propose an end-to-end network to reconstruct high resolution, high dynamic range (HDR) images directly from the event stream. We evaluate our algorithm on both simulated and real-world sequences and verify that it captures fine details of a scene and outperforms the combination of the state-of-the-art event to image algorithms with the state-of-the-art super resolution schemes in many quantitative measures by large margins. We further extend our method by using the active sensor pixel (APS) frames or reconstructing images iteratively., To appear in CVPR 2020 as an oral presentation

This paper addresses the multi-attributed graph matching problem, which considers multiple attributes jointly while preserving the characteristics of each attribute for graph matching. Since most of conventional graph matching algorithms integrate multiple attributes to construct a single unified attribute in an oversimplified manner, the information from multiple attributes is often not completely utilized. In order to solve this problem, we propose a novel multi-layer graph structure that can preserve the characteristics of each attribute in separated layers, and also propose a multi-attributed graph matching algorithm based on random walk centrality with the proposed multi-layer graph structure. We compare the proposed algorithm with other state-of-the-art graph matching algorithms based on a single-layer structure using synthetic and real data sets and demonstrate the superior performance of the proposed multi-layer graph structure and the multi-attributed graph matching algorithm.

Road surface profile (RSP) estimation is an important task to find the imperfections of a road surface, thereby improving ride quality. The RSP estimation has been recently studied using stereo vision owing to its affordable price. However, the existing methods provide noisy and temporally unstable results for real-world driving scenes because of noisy range measurements, noisy pitch angle estimates between the camera and the road surface, and interference from obstacles. This paper proposes a novel method for temporally consistent and robust RSP estimation to overcome these problems. The proposed method consists of three steps: free space estimation, digital elevation map (DEM) estimation, and RSP estimation. We first estimate a drivable area, i.e., free space. For robust and fast free space estimation, we propose an optimization-based non-parametric road surface modeling method and an integral disparity histogram-based free space estimation method. Then, the DEM of the road surface is estimated using range measurements on the free space to avoid obstacle interference. The DEM is temporally updated every frame using the moving average filter and the DEM reference grid update scheme. Owing to these strategies, the proposed method reduces the elevation estimation noise and the pitch angle error, and therefore provides the temporally consistent RSP. We demonstrate the superiority of the proposed method experimentally using stereo image sequences captured in real-world driving scenes.

Online two-dimensional (2D) multi-object tracking (MOT) is a challenging task when the objects of interest have similar appearances. In that case, the motion of objects is another helpful cue for tracking and discriminating multiple objects. However, when using a single moving camera for online 2D MOT, observable motion cues are contaminated by global camera movements and, thus, are not always predictable. To deal with unexpected camera motion, we propose a new data association method that effectively exploits structural constraints in the presence of large camera motion. In addition, to reduce incorrect associations with mis-detections and false positives, we develop a novel event aggregation method to integrate assignment costs computed by structural constraints. We also utilize structural constraints to track missing objects when they are re-detected again. By doing this, identities of the missing objects can be retained continuously. Experimental results validated the effectiveness of the proposed data association algorithm under unexpected camera motions. In addition, tracking results on a large number of benchmark datasets demonstrated that the proposed MOT algorithm performs robustly and favorably against various online methods in terms of several quantitative metrics, and that its performance is comparable to offline methods.

Recently developed object detectors rely on automatically generated object proposals, instead of using a dense sliding window search scheme; generating good object proposals has therefore become crucial for improving the computational cost and accuracy of object detection performance. In particular, the shape and location errors of object proposals can be directly propagated to object detection unless some additional processes are adopted to refine the shape and location of bounding boxes. In this study, the authors demonstrate an object proposal refinement algorithm that improves the localisation accuracy and refines the shape of object proposals by searching a boundary-aligned minimum bounding box. They assume that an object consists of several image regions, and that the optimal object proposal is well aligned with image region boundaries. Based on this assumption, they design novel boundary-region alignment measures and then propose a greedy refinement method based on the proposed measures. Experiments on the PASCAL VOC 2007 dataset show that the proposed method produces highly well-localised object proposals and truly improves the quality of object proposals.

Recent advances in three-dimensional (3D) video technology have extended the range of our experience while providing various 3D applications to our everyday life. Nevertheless, the so-called visual discomfort (VD) problem inevitably degrades the quality of experience in stereoscopic 3D (S3D) displays. Meanwhile, electroencephalography (EEG) has been regarded as one of the most promising brain imaging modalities in the field of cognitive neuroscience. In an effort to facilitate comfort with S3D displays, we propose a new wellness platform using EEG. We first reveal features in EEG signals that are applicable to practical S3D video systems as an index for VD perception. We then develop a framework that can automatically determine severe perception of VD based on the EEG features during S3D video viewing by capitalizing on machine-learning-based braincomputer interface technology. The proposed platform can cooperate with advanced S3D video systems whose stereo baseline is adjustable. Thus, the optimal S3D content can be reconstructed according to a viewer's sensation of VD. Applications of the proposed platform to various S3D industries are suggested, and further technical challenges are discussed for follow-up research.

In this study, the authors consider a multi-target tracking (MTT) problem in a cluttered environment. Due to the difficulty of the problem, the methods relying only on spatial information such as range, bearing and Doppler velocity can be unreliable. To overcome this, they additionally exploit the amplitude information, commonly provided by radar and sonar, for MTT. However, the usage of amplitude information is not straightforward because the signal-to-noise ratio (SNR) should be known in advance or estimated at the same time. To this end, they first propose a novel SNR estimation algorithm based on a maximum a posteriori approach, which helps the tracker to exploit the amplitude information effectively. Based on the estimated SNR, they then propose a complete framework for MTT, which is mainly composed of data association and track state update parts. They extensively evaluate the proposed system in a series of challenging scenarios, and the experimental results verify the effectiveness and robustness of the authors’ methods.

We propose an unsupervised approach to predict depth maps from images captured by a wireless endoscopic capsule. Recent advances in deep learning have shown that accurate depth maps can be predicted from a single image, where the deep network is trained via unsupervised or self-supervised learning by using monocular video sequences or stereo image pairs. However, directly applying these techniques to endoscopic images does not yield satisfactory results owing to the inherent difficulties of the wireless capsule imaging such as dim lighting and low-resolution of images, which are different from normal imaging conditions. For that reason, we exploit the environmental characteristics of endoscopic images — there is no external light source except ones attached to the capsule. Based on this condition, we propose the direct attenuation model-based depth map prediction scheme to guide depth prediction and to add meaningful cues to the loss function. We experimentally verify the proposed method with various endoscopic images.