Search

The regression based deep neural networks have achieved state-of-the-arts performance on depth 3D hand pose estimation task. This paper focuses on improving the regression mapping between features and pose joints. Inspired by the distribution modeling ability of Variational Autoencoders, we introduce an auxiliary variable into the regression network. During training, the auxiliary variable is modeled by an inference distribution that learns the underlying structural kinematics of human hand. Different with other regression methods on hand poses, our network estimates the pose joints from input depth features and the learned auxiliary variable as well. We show that by introducing the auxiliary variable, the regression is benefited from 1) regularization modeled by inference distribution; and 2) prior information carried by the auxiliary model. The effectiveness of the proposed regression method is evaluated with extensively self-comparative experiments and in comparison with other regression methods on hand pose datasets. The proposed network is easy to train in an end-to-end manner and can work with various feature extraction methods. We apply the proposed regression method to an existing hand pose estimation system, and improves the estimation accuracy by 18.35% and 16.65% on public hand pose datasets.

With the rapid development of the Internet, there is a pressing requirement for data security protection in Internet. However, in some applications, resource constrained devices in Internet requires that the algorithm for data encryption have high reliability and availability, as well as extremely low latency, and strong robustness in harsh environment. In this work, we propose an efficient batch images encryption/decryption scheme based on DNA encoding and piecewise linear chaotic mapping system (PWLCM). Different from all existing methods, the proposed method can encrypt/decrypt any number of images with different sizes via only single encryption computation, and the computing cost is equivalent to that of encrypting a single image. Experimental results demonstrate that the proposed method has strong robustness to resist the interference of harsh environment, such as noise contamination, JPEG compression, and image content loss. We also investigate the computing cost, encryption effect and the security. Comparing with the state-of-the-art methods, the proposed method has high encryption efficiency, stronger robustness and security.

Time series prediction problems are a difficult type of predictive modeling problem. In this paper, we propose a time series prediction method based on a variant long short-term memory (LSTM) recurrent neural network. In the proposed method, we firstly improve the memory module of the LSTM recurrent neural network by merging its forget gate and input gate into one update gate, and using Sigmoid layer to control information update. Using improved LSTM recurrent neural network, we develop a time series prediction model. In the proposed model, the parameter migration method is used model update to ensure the model has good predictive ability after predicting multi-step sequences. Experimental results show, compared with several typical time series prediction models, the proposed method have better performance for long-sequence data prediction.

Recurrent neural networks (RNNs) are capable of modeling temporal dependencies of complex sequential data. In general, current available structures of RNNs tend to concentrate on controlling the contributions of current and previous information. However, the exploration of different importance levels of different elements within an input vector is always ignored. We propose a simple yet effective Element-wise-Attention Gate (EleAttG), which can be easily added to an RNN block (e.g. all RNN neurons in an RNN layer), to empower the RNN neurons to have attentiveness capability. For an RNN block, an EleAttG is used for adaptively modulating the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Instead of modulating the input as a whole, the EleAttG modulates the input at fine granularity, i.e., element-wise, and the modulation is content adaptive. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to different tasks including the action recognition, from both skeleton-based data and RGB videos, gesture recognition, and sequential MNIST classification. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly improves the power of RNNs., Comment: IEEE Transactions on Image Processing (Accept). arXiv admin note: substantial text overlap with arXiv:1807.04445

Lane detection is one of the most important task in autonomous driving. While the semantic segmentation based method is widely explored and recognized in recent decade, some post-processing are required to estimate the exact location of the predicted lane markings and can be easily failed in complex scenarios. To tackle these limitations, this paper proposes a novel lane detection network named PCRLaneNet. Firstly, we use a fully convolutional network to predict the coordinates of lane marking points directly, which can better meet with the requirements of autonomous driving. Secondly, to take the fully advantage of the correlation of these lane marking points, a point feature fusion strategy is designed to fuse feature maps of the points on the same lane marking, which makes our method capable of handling challenging scenarios. Lastly, the robustness, accuracy and latency of the proposed method are extensively verified in two datasets (CULane and TuSimple).

Image retrieval based on deep convolutional features has demonstrated state-of-the-art performance in popular benchmarks. In this paper, we present a unified solution to address deep convolutional feature aggregation and image re-ranking by simulating the dynamics of heat diffusion. A distinctive problem in image retrieval is that repetitive or \emph{bursty} features tend to dominate final image representations, resulting in representations less distinguishable. We show that by considering each deep feature as a heat source, our unsupervised aggregation method is able to avoid over-representation of \emph{bursty} features. We additionally provide a practical solution for the proposed aggregation method and further show the efficiency of our method in experimental evaluation. Inspired by the aforementioned deep feature aggregation method, we also propose a method to re-rank a number of top ranked images for a given query image by considering the query as the heat source. Finally, we extensively evaluate the proposed approach with pre-trained and fine-tuned deep networks on common public benchmarks and show superior performance compared to previous work., Comment: The paper has been accepted to IEEE Transactions on Multimedia

Driving accident is the event that occur unexpectedly and should be detected effectively for autonomous driving systems. In this paper, we propose a method based on adversarial appearance-motion prediction for driving accident detection in dashcam videos. The novelty of this method is to consider the predictability of the frame-level and object-level motion and appearance from the current to the future. Through a self-supervised adversarial learning between the real observation in next frame and the predicted motion and appearance, the objects that may occur accident are detected. In order to evaluate the accuracy of the detection results of our method, we evaluate the performance on A3D dataset, and the effectiveness of the method is validated.

3D object detection has become a hot topic in intelligent vehicle applications in recent years. Generally, deep learning has been the primary framework used in 3D object detection, and regression of the object location and classification of the objectness are the two indispensable components. In the process of training, the $\ell_{n}\ (n=1,2)$ and the focal loss are considered as the frequent solutions to minimize the regression and classification loss, respectively. However, there are two problems to be solved in the existing methods. For regression component, there is a gap between evaluation metrics, e.g., 3D Intersection over Union (IoU), and the traditional regression loss. As for the classification component, confidence score exists ambiguous due to the binary label assignment of target. To solve these problems, we propose a loss by jointing 3D IoU and other geometric attributes (named as jointed attribute-oriented 3D loss), which can be directly used in optimizing the regression component. In addition, the jointed attribute-oriented 3D loss can assign a soft label for supervising the training of the classification. By incorporating the proposed loss function into several state-of-the-art 3D object detection methods, the significant performance improvement has been achieved on the KITTI benchmark.

Learning an optimal policy for autonomous driving task to confront with complex environment is a long- studied challenge. Imitative reinforcement learning is accepted as a promising approach to learn a robust driving policy through expert demonstrations and interactions with environments. However, this model utilizes non-smooth rewards, which have a negative impact on matching between navigation commands and trajectory (state-action pairs), and degrade the generalizability of an agent. Smooth rewards are crucial to discriminate actions generated from sub-optimal policy. In this paper, we propose a navigation command matching (NCM) model to address this issue. There are two key components in NCM, 1) a matching measurer produces smooth navigation rewards that measure matching between navigation commands and trajectory; 2) attention-guided agent performs actions given states where salient regions in RGB images (i.e. roadsides, lane markings and dynamic obstacles) are highlighted to amplify their influence on the final model. We obtain navigation rewards and store transitions to replay buffer after an episode, so NCM is able to discriminate actions generated from suboptimal policy. Experiments on CARLA driving benchmark show our proposed NCM outperforms previous state-of-the- art models on various tasks in terms of the percentage of successfully completed episodes. Moreover, our model improves generalizability of the agent and obtains good performance even in unseen scenarios.

In this study, we aim to improve the accuracy of image splicing detection. We propose a progressive image splicing detection method that can detect the position and shape of spliced region. Because image splicing is likely to destroy or change the consistent correlation pattern introduced by color filter array (CFA) interpolation process, we first used a covariance matrix to reconstruct the R, G and B channels of image and utilized the inconsistencies of the CFA interpolation pattern to extract forensics feature. Then, these forensics features were used to perform coarse-grained detection, and texture strength features were used to perform fine-grained detection. Finally, an edge smoothing method was applied to realize precise localization. As compared to the state-of-the-art CFA-based image splicing detection methods, the proposed method has a high-level detection accuracy and strong robustness against content-preserving manipulations and JPEG compression.

We present a generic unsupervised method to increase the discriminative power of image vectors obtained from a broad family of deep neural networks for object retrieval. This goal is accomplished by simultaneously selecting and weighting informative deep convolutional features using the replicator equation, commonly used to capture the essence of selection in evolutionary game theory. The proposed method includes three major steps: First, efficiently detecting features within Regions of Interest (ROIs) using a simple algorithm, as well as trivially collecting a subset of background features. Second, assigning unassigned features by optimizing a standard quadratic problem using the replicator equation. Finally, using the replicator equation again in order to partially address the issue of feature burstiness. We provide theoretical time complexity analysis to show that our method is efficient. Experimental results on several common object retrieval benchmarks using both pre-trained and fine-tuned deep networks show that our method compares favorably to the state-of-the-art. We also publish an easy-to-use Matlab implementation of the proposed method for reproducing our results.

Tunnels cause many traffic accidents in China every year, which is related to the inappropriate driving behaviors combined with less visual comfortability when driving across tunnels. The most common phenomenon when driving across the inlet and outlet sections of a tunnel is the abruptly varying visual brightness, causing a short-term blindness for a driver, known as the “ black hole ” and “ white hole ” effects, respectively. In this paper, we propose a tunnel brightness compensation model with spatial–temporal visual-content preservation. The highlights of this paper are twofold. 1) We analyze the visual brightness variation by introducing spatio-temporal orientation energy for a stable characterization of the visual comfortability of drivers in view of the distinguishable visual content caused by a sequential brightness variation. 2) We construct a tunnel brightness compensation model that preserves the spatial–temporal visual content by visual-content matching with multiple frames. Our method can manifestly improve the brightness quality and maintain the scene content simultaneously. Extensive visual brightness compensation experiments on 60 visual clips of inlet and outlet sections of tunnels demonstrate that the proposed method generates a state-of-the-art performance.

Realizing autonomy is a hot research topic for automatic vehicles in recent years. For a long time, most of the efforts to this goal concentrate on understanding the scenes surrounding the ego-vehicle (autonomous vehicle itself). By completing lowlevel vision tasks, such as detection, tracking and segmentation of the surrounding traffic participants, e.g., pedestrian, cyclists and vehicles, the scenes can be interpreted. However, for an autonomous vehicle, low-level vision tasks are largely insufficient to give help to comprehensive scene understanding. What are and how about the past, the on-going and the future of the scene participants? This deep question actually steers the vehicles towards truly full automation, just like human beings. Based on this thoughtfulness, this paper attempts to investigate the interpretation of traffic scene in autonomous driving from an event reasoning view. To reach this goal, we study the most relevant literatures and the state-of-the-arts on scene representation, event detection and intention prediction in autonomous driving. In addition, we also discuss the open challenges and problems in this field and endeavor to provide possible solutions.

As an active forensic technology, perceptual image hash has important application in image content authenticity detection and integrity authentication. In this paper, we propose a hybrid-feature-based perceptual image hash method that can be used for image tampering detection and tampering localization. In the proposed method, we use the color features of image as global features, use point-based features and block-based features as local features, and combine with the structural features to generate intermediate hash code. Then we encrypt and randomize to generate the final hash code. Using this hash code, we present a coarse-to-fine grained forensics method for image tampering detection. The proposed method can realize object-level tampering localization. Abundant experimental results show that the proposed method is sensitive to content changes caused by malicious attacks, and the tampering localization precision achieves pixel level, and it is robust to a wide range of geometric distortions and content-preserving manipulations. Compared with the state-of-the-art schemes, the proposed scheme yields superior performance.

We present a spatio-temporal energy minimization formulation for simultaneous video object discovery and co-segmentation across multiple videos containing irrelevant frames. Our approach overcomes a limitation that most existing video co-segmentation methods possess, i.e., they perform poorly when dealing with practical videos in which the target objects are not present in many frames. Our formulation incorporates a spatio-temporal auto-context model, which is combined with appearance modeling for superpixel labeling. The superpixel-level labels are propagated to the frame level through a multiple instance boosting algorithm with spatial reasoning, based on which frames containing the target object are identified. Our method only needs to be bootstrapped with the frame-level labels for a few video frames (e.g., usually 1 to 3) to indicate if they contain the target objects or not. Extensive experiments on four datasets validate the efficacy of our proposed method: 1) object segmentation from a single video on the SegTrack dataset, 2) object co-segmentation from multiple videos on a video co-segmentation dataset, and 3) joint object discovery and co-segmentation from multiple videos containing irrelevant frames on the MOViCS dataset and XJTU-Stevens, a new dataset that we introduce in this paper. The proposed method compares favorably with the state-of-the-art in all of these experiments.

Most state-of-the-art robotic cars’ perception systems are quite different from the way a human driver understands traffic environments. First, humans assimilate information from the traffic scene mainly through visual perception, while the machine perception of traffic environments needs to fuse information from several different kinds of sensors to meet safety-critical requirements. Second, a robotic car requires nearly 100% correct perception results for its autonomous driving, while an experienced human driver works well with dynamic traffic environments, in which machine perception could easily produce noisy perception results. In this paper, we propose a vision-centered multi-sensor fusing framework for a traffic environment perception approach to autonomous driving, which fuses camera, LIDAR, and GIS information consistently via both geometrical and semantic constraints for efficient self-localization and obstacle perception. We also discuss robust machine vision algorithms that have been successfully integrated with the framework and address multiple levels of machine vision techniques, from collecting training data, efficiently processing sensor data, and extracting low-level features, to higher-level object and environment mapping. The proposed framework has been tested extensively in actual urban scenes with our self-developed robotic cars for eight years. The empirical results validate its robustness and efficiency.

This paper proposes a SEG-VoxelNet that takes RGB images and LiDAR point clouds as inputs for accurately detecting 3D vehicles in autonomous driving scenarios, which for the first time introduces semantic segmentation technique to assist the 3D LiDAR point cloud based detection. Specifically, SEG-VoxelNet is composed of two sub-networks: an image semantic segmentation network (SEG-Net) and an improved-VoxelNet. The SEG-Net generates the semantic segmentation map which represents the probability of the category for each pixel. The improved-VoxelNet is capable of effectively fusing point cloud data with image semantic feature and generating accurate 3D bounding boxes of vehicles. Experiments on the KITTI 3D vehicle detection benchmark show that our approach outperforms the methods of state-of-the-art.

The modular design of product platform becomes an indispensable aspect in manufacturing industry with increasingly tailored requirements from customers. Amongst all components in this platform, the modularization mechanism addresses key issue in the design process of manufacturing product. However, the traditional modular methods only solve the modularization with single one or several objectives. In this paper, a novel hierarchical collaborative method based on multi-objective optimization is developed for modularization of product platform, which is integrated various of modular standards, including the product structure, product function, design of products, assembly of products and so on, and trades off the contradiction of these standards to achieve the optimal modularization with multiple hierarchical objectives. Moreover, for solutions to this method, a hierarchical collaborative multi-objective optimization algorithm, the Hierarchical Non-dominated Sorting Genetic Algorithm (H-NSGA), is developed to optimize the modularization with multilevel objectives. Furthermore, an industrial case illustrates the details of the proposed model and algorithm. Finally, the experimental comparison demonstrates that the hierarchical collaborative method and the hierarchical multi-objective optimization algorithm can achieve efficient modular solutions with multiple hierarchical objectives.

We propose a novel and effective approach for 3D hand pose estimation on single depth image. Instead of doing deterministic regression from depth images, our model focuses on learning a latent distribution to model the high dimensional space of pose joints, which can also be interpreted as a kinematics model for human hands. Specifically, the proposed network combines the framework of conditional variational autoencoder which learns an encoder and a decoder with standard convolutional network. The encoder models the latent variable as a prior or a regularization for the pose joints. Then probabilistic inference is performed by the decoder to generate the output prediction conditioned on input depth images. In addition, we introduce a pool-convolution module to improve the localization regression of the network. The architecture can be trained end-to-end. In experiments, we demonstrate the effectiveness of our proposed approach in comparison to various state-of-art holistic regression approaches.

In autonomous driving community, numerous benchmarks have been established to assist the tasks of 3D/2D object detection, stereo vision, semantic/instance segmentation. However, the more meaningful dynamic evolution of the surrounding objects of ego-vehicle is rarely exploited, and lacks a large-scale dataset platform. To address this, we introduce BLVD, a large-scale 5D semantics benchmark which does not concentrate on the static detection or semantic/instance segmentation tasks tackled adequately before. Instead, BLVD aims to provide a platform for the tasks of dynamic 4D (3D+temporal) tracking, 5D (4D+interactive) interactive event recognition and intention prediction. This benchmark will boost the deeper understanding of traffic scenes than ever before. We totally yield 249,129 3D annotations, 4,902 independent individuals for tracking with the length of overall 214,922 points, 6,004 valid fragments for 5D interactive event recognition, and 4,900 individuals for 5D intention prediction. These tasks are contained in four kinds of scenarios depending on the object density (low and high) and light conditions (daytime and nighttime). The benchmark can be downloaded from our project site https://github.com/VCCIV/BLVD/., To appear in ICRA2019

In recent years, with the continuous popularity of deep learning, the research on artificial intelligence has boosted the progress of many new applications, such as the autonomous driving. At present, the detection methods of vehicles, pedestrians and other objects in the self-driving technology have been investigated numerously, but there is no good solution for the detection of small objects such as stones on road. However, small targets on road seriously affects the stability of automated vehicle system. Therefore, it is important to carry out the detection of small targets on road.

Skeleton-based human action recognition has attracted great interest thanks to the easy accessibility of the human skeleton data. Recently, there is a trend of using very deep feedforward neural networks to model the 3D coordinates of joints without considering the computational efficiency. In this paper, we propose a simple yet effective semantics-guided neural network (SGN) for skeleton-based action recognition. We explicitly introduce the high level semantics of joints (joint type and frame index) into the network to enhance the feature representation capability. In addition, we exploit the relationship of joints hierarchically through two modules, i.e., a joint-level module for modeling the correlations of joints in the same frame and a framelevel module for modeling the dependencies of frames by taking the joints in the same frame as a whole. A strong baseline is proposed to facilitate the study of this field. With an order of magnitude smaller model size than most previous works, SGN achieves the state-of-the-art performance on the NTU60, NTU120, and SYSU datasets. The source code is available at https://github.com/microsoft/SGN., Comment: Accepted by CVPR2020. The source code is available at https://github.com/microsoft/SGN

Driver attention prediction is becoming an essential research problem in human-like driving systems. This work makes an attempt to predict the driver attention in driving accident scenarios (DADA). However, challenges tread on the heels of that because of the dynamic traffic scene, intricate and imbalanced accident categories. In this work, we design a semantic context induced attentive fusion network (SCAFNet). We first segment the RGB video frames into the images with different semantic regions (i.e., semantic images), where each region denotes one kind of semantic categories of the scene (e.g., road, trees, etc.), and learn the spatio-temporal features of RGB frames and semantic images in two parallel paths simultaneously. Then, the learned features are fused by an attentive fusion network to find the semantic-induced scene variation in driver attention prediction. The contributions are three folds. 1) With the semantic images, we introduce their semantic context features and verified the manifest promotion effect for helping the driver attention prediction, where the semantic context features are modeled by a graph convolution network (GCN) on semantic images; 2) We fuse the semantic context features of semantic images and the features of RGB frames in an attentive strategy, and the fused details are transferred over frames by a convolutional LSTM module to obtain the attention map of each video frame with the consideration of historical scene variation in driving situations; 3) The superiority of the proposed method is evaluated on our previously collected dataset (named as DADA-2000) and two other challenging datasets with state-of-the-art methods. DADA-2000 is available at https://github.com/JWFangit/LOTVS-DADA., Comment: published by IEEE Transactions on Intelligent Transportation Systems

In crowd scenarios, reliable trajectory prediction of pedestrians requires insightful understanding of their social behaviors. These behaviors have been well investigated by plenty of studies, while it is hard to be fully expressed by hand-craft rules. Recent studies based on LSTM networks have shown great ability to learn social behaviors. However, many of these methods rely on previous neighboring hidden states but ignore the important current intention of the neighbors. In order to address this issue, we propose a data-driven state refinement module for LSTM network (SR-LSTM), which activates the utilization of the current intention of neighbors, and jointly and iteratively refines the current states of all participants in the crowd through a message passing mechanism. To effectively extract the social effect of neighbors, we further introduce a social-aware information selection mechanism consisting of an element-wise motion gate and a pedestrian-wise attention to select useful message from neighboring pedestrians. Experimental results on two public datasets, i.e. ETH and UCY, demonstrate the effectiveness of our proposed SR-LSTM and we achieves state-of-the-art results., Comment: Accepted by CVPR2019

Content-based image retrieval is an important research topic in the multimedia field. In large-scale image search using local features, image features are encoded and aggregated into a compact vector to avoid indexing each feature individually. In the aggregation step, sum-aggregation is wildly used in many existing works and demonstrates promising performance. However, it is based on a strong and implicit assumption that the local descriptors of an image are identically and independently distributed in descriptor space and image plane. To address this problem, we propose a new aggregation method named democratic diffusion aggregation (DDA) with weak spatial context embedded. The main idea of our aggregation method is to re-weight the embedded vectors before sum-aggregation by considering the relevance among local descriptors. Different from previous work, by conducting a diffusion process on the improved kernel matrix, we calculate the weighting coefficients more efficiently without any iterative optimization. Besides considering the relevance of local descriptors from different images, we also discuss an efficient query fusion strategy which uses the initial top-ranked image vectors to enhance the retrieval performance. Experimental results show that our aggregation method exhibits much higher efficiency (about $\mathbf {\times \,14}$ faster) and better retrieval accuracy compared with previous methods, and the query fusion strategy consistently improves the retrieval quality.

In this paper, we propose an accurate and real-time positioning method for robotic cars in urban environments. The proposed method uses a robust lane marking detection algorithm, as well as an efficient shape registration algorithm between the detected lane markings and a GPS-based road shape prior, to improve the robustness and accuracy of the global localization of a robotic car. We show that, by formulating the positioning problem in a relative sense, we can estimate the global localization of a car in real time and bound its absolute error in the centimeter level by a cross-validation scheme. The cross-validation scheme integrates the vision-based lane marking detection with the shape registration, and it improves the accuracy and robustness of the overall localization system. The GPS localization can be refined by using lane marking detection when the GPS suffers from frequent satellite signal masking or blockage, whereas lane marking detection is validated and completed by the GPS-based road shape prior when it does not work well in adverse weather conditions or with poor lane signatures. We extensively evaluate the proposed method with a single forward-looking camera mounted on an autonomous vehicle that travels at 60 km/h through several urban street scenes.

Polarization is one of the important characteristics of the interaction between light and substance, which relates to the shape and the material of the target. To research the effects of the degree of polarization (DOP) caused by different materials, the polarized bidirectional reflectance distribution function (pBRDF) model has been established by the theory of microfacet. But in this model, the scattering effect of polyhedron is ignored. Based on these researches, the DOP model of satellite with different shapes (cuboid, cylinder and sphere) is set up and simulated. The simulating results show that the DOP of satellite relates with the complex refractive index, shape of satellite, incident angle and view angle. It is also proved that the polarization is a reflection of the characteristics of material, shape of satellite, which provides theoretical support for identifying satellite.

A ground based imaging polarization detector for astronomical observation is designed, the working principle is analyzed and the mathematical models of degree of polarization and detection ability of the detector are derived. The SNR (signal-to-noise ratio) and number of phoelectrons needed by the detector are analyzed by simulation with the detection demand of 1% or 2% degree of polarization. Then, the integration time of the detector is calculated when detecting stars with different brightness according to the parameters of the detector we designed. The theoretical basis is given through these results.

In the task of pedestrian trajectory prediction, social interaction could be one of the most complicated factors since it is difficult to be interpreted through simple rules. Recent studies have shown a great ability of LSTM networks in learning social behaviors from datasets, e.g., introducing LSTM hidden states of the neighbors at the last time step into LSTM recursion. However, those methods depend on previous neighboring features which lead to a delayed observation. In this paper, we propose a data-driven states refinement LSTM network (SR-LSTM) to enable the utilization of the current intention of neighbors through a message passing framework. Moreover, the model performs in the form of self-updating by jointly refining the current states of all participants, rather than an input-output mechanism served by feature concatenation. In the process of states refinement, a social-aware information selection module consisting of an element-wise motion gate and a pedestrian-wise attention is designed to serve as the guidance of the message passing process. Considering the pedestrian walking space as a graph where each pedestrian is a node and each pedestrian pair with an edge, spatial-edge LSTMs are further exploited to enhance the model capacity, where two kinds of LSTMs interact with each other so that states of them are interactively refined. Experimental results on four widely used pedestrian trajectory datasets, ETH, UCY, PWPD, and NYGC demonstrate the effectiveness of the proposed model.

Computer vision model using deep learning requires a lot of high-quality data for training. However, obtaining amounts of well-annotated data is too expensive. The state-of-the-art automatic annotation tools can accurately detect and segment a few objects. We bring together the annotation tools and the crowed engineering into a framework for object detection and instance-level segmentation. The input of model are the image need to annotate and the annotation constraints: precision, utility and cost. The output of the model are the set of detected objects and the set of instance-level segmentation results. The model can integrate the computer vision annotation model with manual annotation model. We validate human-machine collaborative annotation model on the Cityscapes dataset.

This paper proposes a multimodal fusion model for 3D car detection inputting both point clouds and RGB images and generates the corresponding 3D bounding boxes. Our model is composed of two subnetworks: one is point-based method and another is multi-view based method, which is then combined by a decision fusion model. This decision model can absorb the advantages of these two sub-networks and restrict their shortcomings effectively. Experiments on the KITTI 3D car detection benchmark show that our work can achieve state of the art performance.

Recurrent neural networks (RNNs) are capable of modeling the temporal dynamics of complex sequential information. However, the structures of existing RNN neurons mainly focus on controlling the contributions of current and historical information but do not explore the different importance levels of different elements in an input vector of a time slot. We propose adding a simple yet effective Element-wise-Attention Gate (EleAttG) to an RNN block (e.g., all RNN neurons in a network layer) that empowers the RNN neurons to have the attentiveness capability. For an RNN block, an EleAttG is added to adaptively modulate the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Specifically, the modulation of the input is content adaptive and is performed at fine granularity, being element-wise rather than input-wise. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to the action recognition tasks on both 3D human skeleton data and RGB videos. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly boosts the power of RNNs.

In this work, we propose an efficient image re-ranking method, without regional information of each indexed feature stored in the inverted file, to re-rank all retrieved images. The motivation of the proposed method is that, there are usually many visual words in the query image that only give votes to irrelevant images. With this observation, we propose to use visual words which are actually useful in finding relevant images to re-rank the retrieved images. To achieve this goal, we first initialize some images similar to the query by maximizing a quadratic function when giving an initial ranking of the retrieved images. The quadratic function is constructed by storing the similarities among a short-list of top ranked images into an affinity matrix, where the similarity between any two images is computed by the proposed graph diffusion. Then, we select a subset of visual words in the query with an alternating optimization strategy: (1) at each iteration, visual words are selected based on the set of similar images that we have found, (2) and in turn, the set of similar images is updated with the set of selected words. These two steps are repeated until convergence. Experimental results on standard benchmark datasets show that the proposed method achieves an order of magnitude speedups over the state-of-the-art spatial based re-ranking techniques, and obtains much better retrieval quality as well.

Perceptual image hash has been widely investigated in an attempt to solve the problems of image content authentication and content-based image retrieval. In this paper, we combine statistical analysis methods and visual perception theory to develop a real perceptual image hash method for content authentication. To achieve real perceptual robustness and perceptual sensitivity, the proposed method uses Watson’s visual model to extract visually sensitive features that play an important role in the process of humans perceiving image content. We then generate robust perceptual hash code by combining image-block-based features and key-point-based features. The proposed method achieves a tradeoff between perceptual robustness to tolerate content-preserving manipulations and a wide range of geometric distortions and perceptual sensitivity to detect malicious tampering. Furthermore, it has the functionality to detect compromised image regions. Compared with state-of-the-art schemes, the proposed method obtains a better comprehensive performance in content-based image tampering detection and localization.

This study proposed an online reference governor for a mobile robot to reduce the occurrence of control input saturation. For following the trajectory by a mobile robot, it is one of the practical subjects to provide appropriate control reference even if any disturbances occur. We proposed a methodology to regulate the control reference iteratively based on time-scaling approach. The time-scaling approach is a method to realize to regulate time development characteristic on the given trajectory. It is difficult to model the effect of the interaction with the road surface and the trajectory tracking error is appeared as the amount of accumulated such factors. Therefore, it is a practical approach to reduce the occurrence of control input saturation based on the evaluation of the trajectory tracking error. Proposed reference governor realizes online time scaling based on the trajectory tracking error index and a smooth transition dynamics. By introducing the proposed method, the occurrence of control input ...

Saliency detection aims to find the useful and attractive regions from an image. In many situations, there may be multiple objects in the image, and these objects may have equal attractiveness. Moreover, the appearance of pixels in one object may demonstrate large difference, which could lead to lose the object integrality when detecting saliency. To this end, this paper proposes a multi-saliency detection model via Instance Specific Element Homology (ISEH), where the integrality of an object is also considered. The ISEH is mainly formulated as a belongingness probability computation of two elements (e.g., pixels or superpixels) relative to the same object, which exploits the linking of elements within certain proposal, as well as the objectness of the proposal simultaneously. This work takes ISEH into account in the foreground and background estimation and a final optimization, and generates a superior performance for multi-saliency detection than the state-of-the-arts.

We develop a unified framework for complex event retrieval, recognition and recounting. The framework is based on a compact video representation that exploits the temporal correlations in image features. Our feature alignment procedure identifies and removes the feature redundancies across frames and outputs an intermediate tensor representation we call video imprint. The video imprint is then fed into a reasoning network, whose attention mechanism parallels that of memory networks used in language modeling. The reasoning network simultaneously recognizes the event category and locates the key pieces of evidence for event recounting. In event retrieval tasks, we show that the compact video representation aggregated from the video imprint achieves significantly better retrieval accuracy compared with existing methods. We also set new state of the art results in event recognition tasks with an additional benefit: The latent structure in our reasoning network highlights the areas of the video imprint and can be directly used for event recounting. As video imprint maps back to locations in the video frames, the network allows not only the identification of key frames but also specific areas inside each frame which are most influential to the decision process.

Road shape estimation is important for the safe driving of intelligent vehicles. The common road shape models such as line/parabola, spline and clothoid are lacking of flexibility in various urban traffic scenes. In this paper, a robust road shape model which consists of multiple overlapped submaps is proposed. Each individual submap is represented by a smooth curve generated through Gaussian process(GP). To estimate parameters of a GP submap, a framework involving pre-processing, pose correction, road shape regression and map updating/creating is proposed. Pose correction is achieved by fusion of vehicle motion model and simplified GP-based observation model. Road shape regression is used to extract a coarse road shape. Map updating/creating is used to adapt to the new coming data and generates refined road shape. A robust iterative Gaussian process regression(iGPR) is utilized in both road shape regression and map updating/creating. Extensive experimental results show the efficiency of the proposed method.

Depth information has been used in many fields because of its low cost and easy availability, since the Microsoft Kinect was released. However, the Kinect and Kinect-like RGB-D sensors show limited performance in certain applications and place high demands on accuracy and robustness of depth information. In this paper, we propose a depth sensing system that contains a laser projector similar to that used in the Kinect, and two infrared cameras located on both sides of the laser projector, to obtain higher spatial resolution depth information. We apply the block-matching algorithm to estimate the disparity. To improve the spatial resolution, we reduce the size of matching blocks, but smaller matching blocks generate lower matching precision. To address this problem, we combine two matching modes (binocular mode and monocular mode) in the disparity estimation process. Experimental results show that our method can obtain higher spatial resolution depth without loss of the quality of the range image, compared with the Kinect. Furthermore, our algorithm is implemented on a low-cost hardware platform, and the system can support the resolution of 1280 × 960, and up to a speed of 60 frames per second, for depth image sequences.

Vision+X has become the promising tendency for scene understanding in autonomous driving, where X may be the other non-vision sensors. However, it is difficult to utilize all the superiority of different sensors, mainly because of the heterogenous, asynchronous properties. To this end, this paper calibrates the commonly used RGB+3D-LiDAR data by synchronization and an online spatial structure alignment, and obtains a high-accurate calibration performance. The main highlights are that (1) we rectify the 3D points with the aid of differential inertial measurement unit (IMU), and increase the frequency of 3D laser data as the same as the ones of RGB data, and (2) this work can online high-accurately updates the external parameters of calibration by a more reliable spatial-structure matching of RGB and 3D-LiDAR data. By experimentally in-depth analysis, the superiority of the proposed method is validated.

Skeleton-based human action recognition has recently attracted increasing attention due to the popularity of 3D skeleton data. One main challenge lies in the large view variations in captured human actions. We propose a novel view adaptation scheme to automatically regulate observation viewpoints during the occurrence of an action. Rather than re-positioning the skeletons based on a human defined prior criterion, we design a view adaptive recurrent neural network (RNN) with LSTM architecture, which enables the network itself to adapt to the most suitable observation viewpoints from end to end. Extensive experiment analyses show that the proposed view adaptive RNN model strives to (1) transform the skeletons of various views to much more consistent viewpoints and (2) maintain the continuity of the action rather than transforming every frame to the same position with the same body orientation. Our model achieves significant improvement over the state-of-the-art approaches on three benchmark datasets., Comment: ICCV2017

Robust pedestrian detection has been treated as one of the main pursuits for excellent autonomous driving. Recently, some convolutional neural networks (CNN) based detectors have made large progress for this goal, such as Faster R-CNN. However, the performance of them still needs a large space to be boosted, even owning the complex learning architectures. In this paper, we novelly introduce the 3D LiDAR sensor to boost the CNN-based pedestrian detection. Facing the heterogeneous and asynchronous properties of two different sensors, we firstly introduce an accurate calibration method for visual and LiDAR sensors. Then, some physically geometrical clues acquired by 3D LiDAR are explored to eliminate the erroneous pedestrian proposals generated by the state-of-the-art CNN-based detectors. Exhaustive experiments verified the superiority of the proposed method.

Selecting correct matches from a set of tentative feature point correspondences plays a vital important role in many tasks, such as structure from motion (SfM), wide baseline stereo and image search. In this paper, we propose an efficient and effective method for identifying correct matches from an initial batch of feature correspondences. The proposed method first obtains a subset of correct matches based on the assumption that the local geometric structure among a feature point and its nearest neighbors in an image cannot be easily affected by both geometric and photometric transformations, and thus should be observed in the matched images. For efficiency, we model this local geometric structure by a set of linear coefficients that reconstruct the point from its neighbors. After obtaining a portion of correct matches, we then provide two ways to accurately estimate the correctness of each match and to efficiently estimate the number of correct matches, respectively. The proposed method is evaluated on both applications including image matching and image re-ranking. Experimental results on several public datasets show that our method outperforms state-of-the-art techniques in terms of speed and accuracy.

In this study, the authors present a method to extract moving objects in image sequences. The proposed approach is based on a graph cuts algorithm defined on a spatiotemporal superpixel neighbourhood. Presegmented superpixels are partitioned into foreground and background while preserving temporal and spatial coherence. It achieves this goal by three steps. First, instead of operating at pixel level, the superpixels are advocated as basic units of the authors segmentation scheme. Second, within the graph cuts framework, two superpixel-based data terms and two superpixel-based smoothness terms are proposed to solve segmentation problem. Finally, the proposed method yields the segmentation of all the superpixels within video volume by the graph cuts algorithm. To illustrate the advantages of this approach, the quantitative and qualitative results are compared with other state-of-the-art methods. The experimental results show that the proposed method gives better performance of segmentation with respect to these methods.

This paper proposes a video watermarking scheme based on visual saliency and secret sharing, which can be used for both synchronous authentication and copyright protection. Firstly, video is divided into several different scenes, and region of interest (ROI) for each I frames would be extracted automatically using the proto-object based saliency attention model. Secondly, the robust watermark and fragile watermark are generated according to ROI. The robust watermark is synthesized by copyright information and ROI of the first I frame for each video scene, and the edge map of ROI in each I frame is chosen as the fragile watermark. Finally, the robust watermark and fragile watermark are embedded into I frames of video scenes so that it can achieve video authentication and copyright information simultaneously. Experimental results demonstrate that the proposed method is robust to recompression and frame attack, and is also sensitive to tamper at the same time.

A method for automatically extracting salient object from a single image is presented in this paper. The proposed method is cast in an energy minimization framework. Unlike that only appearance cues are leveraged in most previous methods, an auto-context cue is used as a complementary data term. Benefitting from a generic saliency model for bootstrapping, the segmentation of the salient object and the learning of the auto-context model are iteratively performed without any user intervention. Upon convergence, the method outputs not only a clear separation of the salient object, but also an auto-context classifier which can be used to recognize the same type of object in other images. Our experiments on four benchmarks demonstrated the efficacy of the added contextual cue. It is shown that our method compares favorably with the state-of-the-art, some of which even embraced user interactions. Furthermore, we present some initial recognition results from the induced auto-context model and also show that the segmentation produced by our approach could serve as a good initialization for alpha matting.

This paper proposes a key-pose based video summarization system for a video shot facilitated by using a video object segmentation method. Firstly, we detect the camera motion and extract video objects by a 3D graph-based algorithm. Once the objects are obtained, each of them is represented by a shape descriptor. Secondly, in order to find representative frames which preserve scene content as much accurately as possible, the proposed method calculates difference between pairs of frames based on shape descriptors of objects in the video shot. Finally, key-poses (representative frames) are extracted in a global manner by clustering these shapes. Experimental results on motion video shots show that the proposed method outputs satisfactory summarizations.

Local correlation pattern indicates integrity of an image. Exposing digital forgeries by detecting local correlation pattern of images has become an important kind of approach among many others. However, local correlation pattern is sensitive to JPEG compression, since compression can be regarded as a local homogenization and attenuates the characteristics of local correlation pattern. In this paper, rather than concentrating on the differences between image textures which is common in previous works, we specifically build a gaussian model to describe the local-correlation pattern of color filter array (CFA) interpolation. Thus the model will automatically adapt to JPEG compression. With the model built from the test image, the posterior probability map of CFA interpolation is calculated. To measure the trace of CFA, frequency characteristics of the posterior probability map are calculated and weighted combined. Then the image is classified as tampered or not by a simple threshold. Experimental results from over thousands of tampered images show the validity and efficiency of the proposed method. Moreover, we examine our algorithm to the problem of distinguishing computer-generated images from photos and detecting local tamper area. The proposed method shows a good performance in these tests.