Search

This paper focuses on action recognition tasks oriented to the field of human–robot interaction, which is one of the major challenges in the robotic video understanding field. Previous approaches focus on designing temporal models, lack the ability to capture motion information and build contextual correlation models. This may result in robots being unable to effectively understand long-term video actions. To solve these two problems, this paper propose a novel video understanding framework including: an Adaptive Temporal Sensitivity and Motion Capture Network (ATSMC-Net) and a contextual scene reasoning module called Knowledge Function Graph Module (KFG-Module). The proposed ATSMC-Net can adaptively adjust the frame-level and pixel-level sensitive regions of temporal features to effectively capture motion information. To fuse contextual scene information for cross-temporal inference, the KFG-Module is introduced to achieve fine-grained video understanding based on the relationship between objects and actions. We evaluate the method using three public video understanding benchmarks, including Something-Something-V1&V2 and HMDB51. In addition, we present a dataset with real-world application scenarios of human–robot interactions to verify the effectiveness of our approach on mobile robots. The experimental results show that the proposed method can significantly improve the video understanding of the robots. [ABSTRACT FROM AUTHOR]

Currently, efficient spatio-temporal information modeling is one of the key research components to solve the action recognition problem. Previous approaches focus on enhancing the backbone features individually using hierarchical structures, and unfortunately, most of them fail to achieve a better balance between the interactional adequacy of features within the structure. In this work, we propose an effective Multi-dimensional Adaptive Fusion Network (MDAF-Net), which can be embedded into the mainstream action recognition backbone in a plug-and-play manner to fully activate the transfer and representation of action features in the deep network. Specifically, our MDAF-Net contains two main components: the Adaptive Temporal Capture Module (ATCM) and the Extended Spatial and Channel Module (ESCM). The ATCM effectively suppresses the over-expression of similar features in adjacent frames and activates the expression of motion flow information. The ESCM further improves temporal modeling efficiency by extending the spatial feature perceptual field and enhancing channel attention. Extensive experiments on several challenging action recognition benchmarks, such as Something-Something V1&V2 and Kinetics-400, demonstrate that the proposed MDAF can achieve state-of-the-art and competitive performance. [ABSTRACT FROM AUTHOR]

This paper aims to develop a simple but effective method that can estimate the number of people in still images. Inspired by the successful applications of deep learning and the appearance of crowd, we design a count-net based on Convolutional Neural Network (CNN). The count-net takes the appearance of crowd as auxiliary mechanism, thus filtering out most of the backgrounds and focusing more on people’s heads. In addition, we adopt a separated-aggregated framework since the structure of crowd in an image is rarely uniform. Firstly, we separate the crowd image into patches to treat different spatial locations discriminatively. Afterwards, these patches and the manual marked labels are fed into a count-net to train the parameters of this network. For practical application, only the counting channel of the count-net is demanded. Finally, the numbers of people in patches that belong to one image are summed as the output of the whole framework. Experimental results conducted on UCF and AHU-CROWD datasets demonstrate the superiority of our proposed method. [ABSTRACT FROM AUTHOR]

• A MemWGAN-GP network is proposed to predict future human skeleton trajectories from a given past and reconstruct the given past simultaneously for detecting abnormal human behaviors. • An adversarial loss based on the Wasserstein distance between the generator and critic of the MemWGAN-GP is utilized to improve the reconstruction and prediction ability. • The 2D convolutional and deconvolutional layers are employed in the generator to model the temporal patterns of input trajectory feature sequence. Timely detection of human-related anomaly in surveillance videos is a challenging task. Generally, the irregular human motion and action patterns can be regarded as abnormal human-related events. In this paper, we utilize the skeleton trajectories to learn the regularities of human motion and action in videos for anomaly detection. The skeleton trajectories are decomposed into global and local feature sequences, which are utilized to provide human motion and action information, respectively. Then, the global and local sequences are modeled as two separate sub-processes with our proposed Memory-augmented Wasserstein Generative Adversarial Network with Gradient Penalty (MemWGAN-GP). In each sub-process, the pre-trained MemWGAN-GP is employed to predict future feature sequences from corresponding input past sequences and reconstruct the input sequences simultaneously. The predicted and reconstructed feature sequences are compared with their groundtruth to identify anomalous sequences. The MemWGAN-GP integrates the autoencoder with a WGAN model to boost the reconstruction and prediction ability of the autoencoder. Besides, a memory module is employed in MemWGAN-GP to overcome high capacity of the autoencoder for anomalies reconstruction and prediction. Experimental results on four challenging datasets demonstrate advantages of the proposed method over other state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]