Search

Data augmentation plays a crucial role in training a CNN-based detector. Most previous approaches were based on using a combination of general image-processing operations and could only produce limited plausible image variations. Recently, GAN (Generative Adversarial Network) based methods have shown compelling visual results. However, they are prone to fail at preserving image-objects and maintaining translation consistency when faced with large and complex domain shifts, such as day-to-night. In this paper, we propose AugGAN, a GAN-based data augmenter which could transform on-road driving images to a desired domain while image-objects would be well-preserved. The contribution of this work is three-fold: (1) we design a structure-aware unpaired image-to-image translation network which learns the latent data transformation across different domains while artifacts in the transformed images are greatly reduced; (2) we quantitatively prove that the domain adaptation capability of a vehicle detector is not limited by its training data; (3) our object-preserving network provides significant performance gain in the difficult day-to-night case in terms of vehicle detection. AugGAN could generate more visually plausible images compared to competing methods on different on-road image translation tasks across domains. In addition, we quantitatively evaluate different methods by training Faster R-CNN and YOLO with datasets generated from the transformed results and demonstrate significant improvement on the object detection accuracies by using the proposed AugGAN model.

Unpaired image-to-image translation is proven quite effective in boosting a CNN-based object detector for a different domain by means of data augmentation that can well preserve the image-objects in the translated images. Recently, multimodal GAN (Generative Adversarial Network) models have been proposed and were expected to further boost the detector accuracy by generating a diverse collection of images in the target domain, given only a single/labelled image in the source domain. However, images generated by multimodal GANs would achieve even worse detection accuracy than the ones by a unimodal GAN with better object preservation. In this work, we introduce cycle-structure consistency for generating diverse and structure-preserved translated images across complex domains, such as between day and night, for object detector training. Qualitative results show that our model, Multimodal AugGAN, can generate diverse and realistic images for the target domain. For quantitative comparisons, we evaluate other competing methods and ours by using the generated images to train YOLO, Faster R-CNN and FCN models and prove that our model achieves significant improvement and outperforms other methods on the detection accuracies and the FCN scores. Also, we demonstrate that our model could provide more diverse object appearances in the target domain through comparison on the perceptual distance metric.

Vehicle detection is a fundamental function required for advanced driver assistance systems. Extensive research has shown that good performance can be obtained on public datasets by various state-of-the-art approaches, especially the deep learning methods. However, those methods are mostly two-stage approaches which inevitably require extensive computing resources and are hard to be deployed on an embedded computing platform with real-time computing performance. We introduce a single-stage vehicle detector which can work in real-time on NVIDIA DrivePX2 platform. The main contributions of this paper are threefold. We propose a detection scheme which includes multi-scale features and multi-anchor boxes to improve the accuracy of a single-stage detector. Secondly, a new data augmentation strategy is proposed to systematically generate a lot of vehicle training images whose appearances are randomly truncated, so our detector is trained to detect partially-seen vehicles better. Thirdly, we present a multi-stage image-based online hard example mining (MSI-OHEM) framework specifically designed for single-stage detectors. MSI-OHEM performs fine-tuning on hard examples and the ones with slightly-insufficient IOU that are considered true positives. Compared to other classical object detectors, the proposed detector achieves very competitive result in terms of average precision (AP) and computational speed. For the newly-defined vehicle class (car+bus) on VOC2007 test, our detector, using MobileNetV2, GoogLeNet, Inception-v2 and ResNet-50 as basenets, achieves 85.35%/85.62%/86.49%/87.81% AP and runs at 64/58/48/28 FPS on NVIDIA DrivePX2, respectively.

This paper presents a traffic light detection system based on convolutional neural networks and lidar data. The proposed approach contains two stages. In the first detection stage, the map information is adopted to assist the object detection, and two cameras with different focal lengths are used to detect traffic lights at different distances. In the following recognition stage, we combine detector and classifier to deal with the problem of many light states. A dataset is created with urban road scenes in Taiwan. The experiments have demonstrated the advance in traffic light detection and recognition.

Image-to-image translation is potential to boost the detection accuracy of a CNN-based object detector in a different domain. Despite recent GAN (Generative Adversarial Network) based methods have shown compelling visual results, they are prone to fail at preserving image-objects and maintaining structure consistency when faced with large and complex domain shifts such as day-to-night, which reduces their practicality on tasks such as generating large-scale training data for different domains. In this work, we introduce image-translation-structure and cycle-structure consistency for generating diverse and structure-preserved translated images across complex domains, such as between day and night, for object detector training. Given only a single/labelled image at daytime, our model could generate a diverse collection of images at nighttime with different ambient light levels and rear lamp conditions (on/off) but with the same vehicle type, color and locations. Qualitative results show that our model can generate diverse and realistic images in the target domain data. For quantitative comparisons, we evaluate other competing methods and ours by using the generated images to train the Faster R-CNN and YOLO detectors and prove that our model achieves significant improvement and outperforms other methods on detection accuracy.

In this paper, we propose a deep learning system to localize and recognize Chinese texts in scenes with signage and road marks through 3D convolutional neural network. The proposed system adopts YOLO for detecting target location and exploits 3D convolutional neural network for recognizing the contents. The proposed design outperforms the existing designs based on LSTM and achieves real-time processing performance, which is feasible to be implemented on embedded platforms. The proposed system reaches over 90% accuracy in recognizing Chinese texts on bird’s-eye viewing road marks in a self-driving vehicle equipped with a fisheye camera. In addition, this system can achieve 20 fps execution speed with NVIDIA DIGITS DevBox with 1080Ti GPU, which is fast enough for autonomous driving applications.

We construct a robust road mark detector that achieves high accuracy with real-time processing performance (32 fps) under nVidia Titan-X GPU. We combine one stage deep learning detector with auxiliary CNN classifiers as a robust road marks detector. We found out that one stage detector not only detects multiple objects via single inference efficiently, but also remains a good accuracy in performance perspective. However, to make it better, we add an extra CNN classifier as the back part of the proposed architecture to reduce false positive and get better accuracy. The proposed detector can achieve 86.8% mAP in our in-house six-class road mark database.

Deep learning based image-to-image translation methods aim at learning the joint distribution of the two domains and finding transformations between them. Despite recent GAN (Generative Adversarial Network) based methods have shown compelling results, they are prone to fail at preserving image-objects and maintaining translation consistency, which reduces their practicality on tasks such as generating large-scale training data for different domains. To address this problem, we purpose a structure-aware image-to-image translation network, which is composed of encoders, generators, discriminators and parsing nets for the two domains, respectively, in a unified framework. The purposed network generates more visually plausible images compared to competing methods on different image-translation tasks. In addition, we quantitatively evaluate different methods by training Faster-RCNN and YOLO with datasets generated from the image-translation results and demonstrate significant improvement on the detection accuracies by using the proposed image-object preserving network.

In this paper, we introduce Boost-CNN, a robust stop-line detector that can detect objects (stop line) with competitive tradeoff between speed and accuracy. Boost-CNN consists of an AdaBoost classifier and a CNN. The former is our region proposal generator and it is further combined with the later to be a stop-line detector. In addition, an automatic hard mining method is proposed to reduce the number of false alarm. Our proposed detector achieves 91.5% in accuracy and has 100 FPS performance in test time (performed on NVIDIA DIGITS DevBox and Titan X GPU).

This paper presents a fast vehicle detector which can be deployed on NVIDIA DrivePX2 under real-time constraints. The network predicts bounding boxes with different aspect ratio and scale priors from the specifically-designed prediction module given concatenated multi-scale feature map. A new data augmentation strategy is proposed to systematically generate a lot of vehicle training images whose appearance is randomly truncated so our detector could detect occluded vehicles better. Besides, we propose a non-region-based online hard example mining framework which performs fine-tuning by picking (1) hard examples and (2) detection results with insufficient IOU. Compared to other classical object detectors, this work achieves very competitive result in terms of average precision (AP) and computational speed. For the newly-defined vehicle class (car+bus) on VOC2007 test, our detector achieves 85.32 AP and runs at 48 FPS and 30 FPS on NVIDIA Titan X & GP106 (DrivePX2), respectively.

Video stabilization is of fundamental importance to provide high-quality video for non-stationary camera. However, conventional video stabilization methods did not consider lens distortion because the inter-frame translation is mostly estimated under the assumption of pure geometrical transform. This paper proposes a video stabilization algorithm which models the image distortion with a division model and estimates the vertical image translation so as to better stabilize consecutive images captured by wide-angle dashcam mounted on a vehicle. Besides, in order to quantitatively assess the stabilization result, several synthetic vibration sequences with associated Ground-Truth (GT) are generated. Using these sequences, we demonstrate that our algorithm outperforms other methods in that the distortion effect resulting from wide-angle lens is also modeled. Moreover, the proposed algorithm could achieve almost equivalent stabilization results from slight to heavy degree of vibration. The proposed datasets and the experimental results for either synthetic or real-driving data are available on the web.

Advances in video technology have enabled its wide adoption in the auto industry. Today, many vehicles are equipped with backup, front-looking, and side-looking cameras that allow the driver to easily monitor traffic around the vehicle for enhancing safety. One difficulty with performing automated image analysis using a vehicle's onboard video has to do with the significant lens distortion of these sensors to cover a large field of view around the vehicle. This paper reports our research on proposing a tracking scheme that improves the accuracy and denseness of object tracking in the presence of large lens distortion. The contribution of our research is 4-fold: (1) We evaluated a large collection of state-of-the-art trackers to understand their deficiency when applied to videos with large lens distortion, (2) we showed how to derive useful evaluation metrics from public-domain, real-world driving videos that do not come with ground-truth information on pixel tracking, (3) we identified many enhancement techniques that can potentially help improve the poor performance of current trackers on videos of large lens distortion, and (4) we performed a systematic study to validate the efficacy of these enhancement techniques and proposed a new tracker design that achieved substantial improvement over the state-of-the- art, in terms of both accuracy and density, based on a rigorous precision vs. recall analysis.

Advances in video technology have enabled its wide adoption in the auto industry. Today, many vehicles are equipped with backup, front-looking, and side-looking cameras that allow the driver to easily monitor the traffic around the vehicle for enhanced safety. This paper reports our research on evaluating many existing object tracking techniques, and proposes a new tracker design and its application for 3D environmental mapping in vehicular technology applications. The contribution of our research is 4-fold: (1) We evaluate a large collection of state-of-the-art trackers using multiple criteria relevant to vehicular technology applications, (2) we show how to derive useful evaluation metrics from public-domain, real-world driving videos that do not come with ground-truth information on pixel tracking, (3) we propose a new tracker that is geared specifically for vehicular technology application and show that it achieves tracking accuracy that outperforms SIFT and is on-par with state-of-the-art optical-flow tracking algorithm, which has the best accuracy in our evaluation. Furthermore, we show that our tracker is 600 times more efficient than optical flow and 7 times more efficient than SIFT, and (4) we validated our new tracker design for 3D environmental map building application and showed that the new tracker can obtain comparable results as SIFT but with a significant saving in runtime.

This paper proposes a new intelligent surveillance system for parking lot management in underground environments. The feature based image stitching and image blending techniques are used to let video streams captured by multiple cameras be fused into single output whose blind area could be easily seen through. Such a system allows that anyone can immediately understand what is going on in the whole monitored area with a simple glance and without prior geometrical knowledge of the place because the blind area is translucent. The image stitching processes consists of four main parts:(1) feature correspondence detection and fundamental matrix estimation, (2) outlier filtering scheme based on the epipolar geometry and RANSAC, (3) computing the homography matrices between each pair of images, (4) projectively warping the images with their corresponding homography matrices, and conducting image fusion with the non-overlapping parts of the warped images. Finally, the process of translucent blending is applied to eliminate the blind areas inside the stitched image, and then the corresponding cameras behind the occluding pillar provide pixels for translucentizing. The authors have implemented the preliminary system with six surveillance cameras at underground parking lot environments, and experiment results of real world video sequences have been performed to verify the proposed design.

the goal of the project is to design intelligent and robust image-processing and augmented-reality algorithms for driver assistance and enhanced vehicular safety. In particular, the focuses were two-fold: (1) realizing the abilities to identify and localize in a vehicle's on-board video the sweeping windshield wipers during raining days and (2) designing and implementing an in-painting technique to remove the image of the windshield wipers and replace it with the corresponding pixels (not blocked by the wipers) from an adjacent frame.

This paper proposes a new algorithm to eliminate the wiper interference in a vehicle's onboard video to improve the detection rate of vision-based Advanced Driver Assistance Systems (ADAS), such as Forward Collision Warning (FCW) Systems. During raining days, the windshield wipers periodically and partially block the appearance of obstacles on the road that are to be detected in these early warning systems, and hence, impair their performance. The wiper pixels could be in-painted via (1) localizing the pixels belonging to the wipers and (2) replacing the wiper pixels with non-wiper pixels extracted from an adjacent video frame with no such blockage. Finally, since it is impossible to obtain the images captured in the same scenario with and without wiper interference, a quantitatively analyzing framework which blends wipers into non-wiper images is also proposed to quantitatively assess the detection rate of data with wipers, without wipers and with wipers but eliminated by our algorithm. The experimental results show that the wiper pixels being segmented and in-painted blend in unobtrusively with the surrounding pixels and the FCW detection rate is indeed improved.

This paper proposes a new vehicle blind spot elimination system which utilizes the on-board videos captured from other vehicles and the host vehicle. Such information is exchanged by WAVE/DSRC devices to achieve collaborative safety. The preceding vehicle which fully or partially blocks the field of view of the host vehicle could be translucentized in the video captured by the host vehicle and the driving environment of the front vehicle could be then visually checked by the host driver.

This paper proposes a monocular vision-based obstacle detection algorithm for parking assistance applications of advance safety vehicle by a rear camera. In order to efficiently detect various moving and stationary obstacles behind the vehicle, the feature of corner into the rear obstacles is firstly estimated by the Features from Accelerated Segment Test (FAST) corner detection methods. Then, the inverse perspective mapping (IPM) image can be used to determine whether every detected feature belongs to an obstacle candidate or to the ground. Based on these results, the segmentation and identification strategies are also proposed in order to determine the degree of collision risk and to filter out the non-hazardous candidates. Finally, the correct obstacle regions in IPM transformed image can be easily and quickly extracted. The system can provide a vision-based alert to the driver, helping to avoid collisions with obstacles behind the host vehicle. Through extensive experiments, we have shown that the rear obstacle detection system in typical urban situations can be used to efficiently extract obstacle regions markings. The proposed algorithm achieves high detecting rate and low computing power and is successfully implemented in ADI-BF561 600MHz dual core DSP.

Rain removal from an image/video is a challenging problem and has been recently investigated extensively. In our previous work, we have proposed the first single-image-based rain streak removal framework via properly formulating it as an image decomposition problem based on morphological component analysis (MCA) solved by performing dictionary learning and sparse coding. However, in this previous work, the dictionary learning process cannot be fully automatic, where the two dictionaries used for rain removal were selected heuristically or by human intervention. In this paper, we extend our previous work to propose an automatic self-learning-based rain streak removal framework for single image. We propose to automatically self-learn the two dictionaries used for rain removal without additional information or any assumption. We then extend our single-image-based method to video-based rain removal in a static scene by exploiting the temporal information of successive frames and reusing the dictionaries learned by the former frame(s) in a video while maintaining the temporal consistency of the video. As a result, the rain component can be successfully removed from the image/video while preserving most original details. Experimental results demonstrate the efficacy of the proposed algorithm.