Recognizing Gestures from Videos using a Network with Two-branch Structure and Additional Motion Cues

In this paper, we propose a method for recognizing gestures from videos which implicitly incorporates multimodal data during training, and makes classification by only using RGB modality data. The network is 3d-convolutional, and includes a shared network for implicitly incorporating multiple modalities, a generation branch for estimating motion regions and a classification branch for classifying gestures. We introduce a type of efficient modality data, binarized motion cues, which include information of moving hand regions, and are learned by using the generation network. The binarized motion cues are given as extra supervision for learning motion in the generation branch. Since features of additional motion cues learned by the generation branch are implicitly fused with features learned by the classification branch, the classification performance can be improved. Experimental results showed that the shared network can extract more discriminable intermediate features, and the network with the classification branch can achieve improved performance by only using RGB modality input data.