Gesture recognition for sign language Video Stream Translation

Due to the complexity of camera viewing angle, background and different execution speed, the difference between hand movements is low, which makes it difficult to extract features in the process of sign language recognition. In order to solve this problem, we propose a sign language recognition framework based on attention mechanism, which enables the network to pay attention to the areas that contain objects related to the activities under consideration. We use Gradient-weighted Class Activation Mapping in the pre-trained CNN for general image recognition to learn the attention map of each frame, and use convolutional LSTM to encode the video in time and space. In view of the fact that the convolution structure in LSTM has no obvious contribution to spatio-temporal feature fusion, the LSTM is adjusted to make it possible to effectively fuse spatiotemporal features with less parameters and computation.