Search

Long-range spatiotemporal dependencies capturing plays an essential role in improving video features for action recognition. The previously introduced non-local block, inspired by the non-local means, is designed to address this challenge and have shown excellent performance. However, the non-local block brings significant increase in computation cost to the original network. It also lacks the ability to model regional correlations in videos. To address the above limitations, we propose the Pyramid Non-Local (PNL) module, which extends the original non-local block by incorporating regional correlations at multiple scales through a pyramid structured module. This extension upscales the effectiveness of the original non-local block by attending to the interaction between different regions. Empirical results prove the effectiveness and efficiency of our PNL module, which achieves state-of-the-art performance of 83.09 % on the Mini-Kinetics dataset, with decreased computational cost compared to the non-local block. [ABSTRACT FROM AUTHOR]

• A new module utilizing embedded key point shifts for efficient action recognition. • KPSEM extracts and embeds regional key point shifts from high-level feature maps. • Network with KPSEM achieves state-of-the-art performance on benchmark datasets. Temporal feature extraction is an essential technique in video-based action recognition. Key points have been utilized in skeleton-based action recognition methods but they require costly key point annotation. In this paper, we propose a novel temporal feature extraction module, named Key Point Shifts Embedding Module (K P S E M), to adaptively extract channel-wise key point shifts across video frames without key point annotation. Key points are adaptively extracted as feature points with maximum feature values at split regions and key point shifts are the spatial displacements of corresponding key points. The key point shifts are encoded as the overall temporal features via linear embedding layers in a multi-set manner. Our method achieves competitive performance through embedding key point shifts with trivial computational cost, achieving the state-of-the-art performance of 78.81 % on Mini-Kinetics and competitive performance on UCF101, Something-Something-v1 and HMDB51 datasets. [ABSTRACT FROM AUTHOR]