Search

Describing a traffic scenario from the driver’s perspective is a challenging process for Advanced Driving Assistance System (ADAS), involving different sub-tasks of detection, tracking, segmentation, etc. Previous methods mainly focus on independent sub-tasks and have difficulties to comprehensively describe the incidents. In this study, this problem is novelly treated as a video captioning task, and a Guidance Attention Captioning Network (GAC-Network) structure is proposed for describing the incidents in a concise single sentence. In GAC-Network, an Attention based Encoder-Decoder Net (AED-Net) is built as the main network; with the temporal spatial attention mechanisms, the AED-Net make it possible to effectively reject the unimportant traffic behaviors and redundant backgrounds. Considering various driving scenarios, the Spatio-Temporal Layer Normalization is used to improve the generalization ability. To generate captions for incidents in driving, the novel Guidance Module is proposed to boost the encoder-decoder model to generate words in a caption, which have better relationship to the past and future words. Because there is no public dataset for captioning of driving scenarios, the Traffic Video Captioning (TVC) dataset is released for the video captioning task in driving scenarios. Experimental results show that the proposed methods can fulfill the captioning task for complex driving scenarios, and achieve higher performance than the methods for comparison, including at least 2.5%, 1.8%, 3.6%, and 13.1% better results on BLEU_1, METEOR, ROUGE_L and CIDEr, respectively.

Grasp detection in cluttered scenes is a very challenging task for robots. Generating synthetic grasping data is a popular way to train and test grasp methods, as is Dex-Net; yet, these methods sample training grasps on 3-D synthetic object models, but evaluate at images or point clouds with different sample distributions, which reduces performance due to covariate shift and sparse grasp labels. To solve existing problems, we propose a novel on-policy grasp detection method for parallel grippers, which can train and test on the approximate distribution with dense pixel-level grasp labels generated on RGB-D images. An Orthographic-Depth Grasp Generation (ODG-Generation) method is proposed to generate an orthographic depth image through a new imaging model of projecting points in orthographic; then this method generates multiple candidate grasps for each pixel and obtains robust positive grasps through flatness detection, force-closure metric and collision detection. Then, a comprehensive Pixel-Level Grasp Pose Dataset (PLGP-Dataset) is constructed, which is the first pixel-level grasp dataset, with the on-policy distribution. Lastly, we build a grasp detection network with a novel data augmentation process for imbalance training. Experiments show that our on-policy method can partially overcome the gap between simulation and reality, and achieves the best performance.

Yawning detection is a key means in driver fatigue detection, which suffers difficulties including head poses, facial expressions, illumination variations, occlusions, etc. Yet, most previous methods mainly focus on frontal faces, and are deficient to deal with different facial actions under arbitrary poses in the actual driving environment. In this study, we propose a novel Joint Head Pose and Facial Action Network (JHPFA-Net) for driver yawning detection across arbitrary poses in videos, with three main parts including a Geometric-based Key-frame Selection Module (GK-Module), a Face Frontalization with Warp Attention Module (FF-Module) and a dual-channel classifier for Head Pose & Facial Action Fusion Module (HF-Module). Firstly, the GK-Module is proposed to extract geometric vectors and to construct a two-stage judgment mechanism, with the purpose of dealing with frame redundancy and improving the efficiency of JHPFA-Net structure. Secondly, distinguished with existing methods, the FF-Module is proposed to synthesize photo-realistic frontal faces, which can be used for capturing the facial actions under arbitrary poses. Finally, the HF-Module is proposed to fuse head pose attributes and facial modalities together, for the purpose of achieving pose-invariant detection and improving accuracy. Extensive experiments show that the proposed JHPFA-Net achieves state-of-the-art results comparing with some representative methods on the public YawDD benchmark, and it performs well in real-time application.

Most of previous works on abnormal event detection take this task as a novelty detection problem, which employ the supervised setting that needs videos containing only normal events for learning normal patterns. However, few works are developed under the unsupervised setting that detects anomaly without labeled normal videos. In this article, we develop a novel unsupervised algorithm using the skeleton feature for detecting human-related anomalous events. Our method applies the idea of self-training regression for iteratively updating the anomaly scores of skeletons for anomaly detection. In detail, each extracted skeleton is first decomposed into global and local feature components. Then, an unsupervised anomaly detector is operated on these two components to generate the initial anomalous and normal skeleton sets. These two sets are utilized to optimize parameters of an anomaly scoring module consisting of a spatial graph convolutional network (SGCN) and fully connected layers. The trained module is then employed to recalculate anomaly scores of all skeletons to update memberships of pseudo anomalous and normal skeletons set for the next training procedure, and this process is performed in an iterative way to get superior anomaly detection performance. Experimental results on two challenging data sets and their subsets that only contain human-related anomalies demonstrate our method outperforms several state-of-the-art supervised methods.

The purpose of action prediction is to recognize an action before it is completed to reduce recognition latency. Because action prediction has lower latency than action recognition, it can be applied to a variety of surveillance scenarios and responds faster. However, action prediction is more difficult because it cannot obtain the complete action execution. In this article, we study the action prediction which is based on skeleton data and propose a new network called adaptive graph convolutional network with adversarial learning (AGCN-AL) for it. The AGCN-AL uses adversarial learning to make the features of the partial sequences as similar as possible to the features of the full sequences to learn the potential global information in the partial sequences. Besides, partial sequences with different numbers of frames contain different amounts of information. We introduce temporal-dependent loss functions to prevent the network from paying too much attention to partial sequences whose observation ratios are small, and ignoring partial sequences whose observation ratios are large. Moreover, the AGCN-AL is combined with the local AGCN into a two-stream network to enhance the prediction, proving that the local information and the potential global information in partial sequences are complementary. We evaluate the proposed approach on two data sets and show excellent performance.

Vehicle counting and traffic volume estimation on traffic videos are challenging tasks for traffic monitoring and management. Most previous methods are based on a process of vehicle detection and tracking or a process of time-spatial image (TSI)-based background subtraction, which suffer from occlusion and small size of vehicles. To overcome these difficulties, a time-spatial-based structure is proposed for vehicle counting and traffic volume estimation, without relying on tracking and TSI-based background subtraction. This structure has three main parts. First, a TSI-based density map generation model is proposed for generating density maps for TSIs, which makes it possible to automatically generate TSI training samples. Second, the time-spatial multiscale net (TM-Net) is proposed for estimating density map of TSI; with the stacked multiscale modules and the spatial attention module, the proposed TM-Net can partly overcome the difficulties brought from occlusion, small size, and deformation. Finally, a vehicle counting and traffic volume estimation model is designed for counting and volume estimation on the results of TM-Net. Experiments performed on the public UA-DETRAC data set show that the proposed TM-Net-based vehicle counting method outperforms the tested representative counting methods, and the proposed framework also can estimate traffic volume efficiently.

Rotating machines are one of the most common equipment in modern industry, the health condition of the equipment is closely linked to safety of workers and production effectiveness. Thus accurate and robust fault diagnostic approaches are vital to safety production. In practice, diagnostic accuracy is seriously affected by noises, especially in low signal-to-noise (SNR) ratio conditions, and the quality of fault features is positively link to the diagnosing accuracy. In consideration of distinguishable feature expression can improve diagnosing result and robust to wider range of experimental conditions, this paper presents a novel spectrogram based local fluctuation feature (SLFF) for low SNR conditions. Firstly, signals are transformed into spectrograms. Then a feature extracting window bank is established on spectrograms for SLFF. At last, a support vector machine (SVM) is applied as a fault classifier for evaluating the proposed feature. The proposed SLFF represents the basic spectral shape and variation which leads to robust and well distinguishable feature expression, the feature reveals the differences of spectral local variation trends between normal and fault types that can improve the discrimination under the influence of strong noises. The effectiveness of the proposed method has been proved experimentally in this paper.