Human action recognition using a convolutional neural network based on skeleton heatmaps from two-stage pose estimation

Human action recognition based on skeleton information has been extensively used in various areas, such as human–computer interaction. In this paper, we extracted human skeleton data by constructing a two-stage human pose estimation model, which combined the improved single shot detector (SSD) algorithm with convolutional pose machines (CPM) to obtain human skeleton heatmaps. The backbone of the SSD algorithm was replaced with ResNet, which can characterize images effectively. In addition, we designed multiscale transformation rules for CPM to fuse the information of different scales and a convolutional neural network for the classification of the skeleton keypoints heatmaps to complete action recognition. Indoor and outdoor experiments were conducted on the Caster Moma mobile robot platform, and without an external remote control, the real-time movement of the robot was controlled by the leader through command actions.