Real-time action recognition by feature-level fusion of depth and inertial sensor

Human action recognition has been an active research topic for its wide applications. Most researches focus on the recognition based on one single modality sensor. In this paper, we present a novel approach for human action recognition, which is based on feature-level fusion of depth and inertial sensor. We extract Fast Fourier Transform (FFT) coefficients from acceleration signals and Histograms of Oriented Gradients (HOG) features from Motion Response Maps (MRM). After obtaining these two modality feature vectors, we adopt Discriminant Correlation Analysis (DCA) to learn a fused feature descriptor with better discriminating ability. To evaluate the effectiveness and efficiency of the proposed approach, we conduct experiments on the multimodal human action database CAS-YNU-MHAD. Experimental results demonstrate the fused feature descriptor exhibits a strong and stable performance in improving the recognition accuracy. Moreover, our approach has a low computational complexity and can be employed in real-time systems.