1. Real-Time Monocular Skeleton-Based Hand Gesture Recognition Using 3D-Jointsformer.
- Author
-
Zhong, Enmin, del-Blanco, Carlos R., Berjón, Daniel, Jaureguizar, Fernando, and García, Narciso
- Subjects
- *
CONVOLUTIONAL neural networks , *GESTURE , *TRANSFORMER models , *MONOCULARS , *RECOGNITION (Psychology) - Abstract
Automatic hand gesture recognition in video sequences has widespread applications, ranging from home automation to sign language interpretation and clinical operations. The primary challenge lies in achieving real-time recognition while managing temporal dependencies that can impact performance. Existing methods employ 3D convolutional or Transformer-based architectures with hand skeleton estimation, but both have limitations. To address these challenges, a hybrid approach that combines 3D Convolutional Neural Networks (3D-CNNs) and Transformers is proposed. The method involves using a 3D-CNN to compute high-level semantic skeleton embeddings, capturing local spatial and temporal characteristics of hand gestures. A Transformer network with a self-attention mechanism is then employed to efficiently capture long-range temporal dependencies in the skeleton sequence. Evaluation of the Briareo and Multimodal Hand Gesture datasets resulted in accuracy scores of 95.49% and 97.25%, respectively. Notably, this approach achieves real-time performance using a standard CPU, distinguishing it from methods that require specialized GPUs. The hybrid approach's real-time efficiency and high accuracy demonstrate its superiority over existing state-of-the-art methods. In summary, the hybrid 3D-CNN and Transformer approach effectively addresses real-time recognition challenges and efficient handling of temporal dependencies, outperforming existing methods in both accuracy and speed. [ABSTRACT FROM AUTHOR]
- Published
- 2023
- Full Text
- View/download PDF