A Skeleton-Based Rehabilitation Exercise Assessment System With Rotation Invariance

Automated exercise assessment is of great importance for patients under rehabilitation exercise who require professional guidance. Among the existing approaches, the skeleton-based assessment model that classifies the correctness of the exercise has attracted much attention due to its relative ease of implementation and convenience in use. However, there are two problems with this approach. The first problem is its sensitivity to the orientation of the human skeleton. To solve this problem, we propose a novel rotation-invariant descriptor, the dot product matrix of the human skeleton, and prove mathematically that this descriptor discards only the orientation message that we do not expect while preserving all other useful information. Lack of feedback from the system is the second problem, because the exercisers do not know which parts of their exercises are incorrect. Therefore, we develop a visualization method for our system based on Gradient-Weighted Class Activation Mapping (Grad-CAM) and an quantitative metric called Overlap Ratio (OvR) to measure the quality of the visualization result. To demonstrate the effect of our method, we conduct experiments on two public datasets and a self-generated push-up dataset. The experimental results show that our rotation-invariant descriptor can achieve absolute robustness to orientation even under severe angle perturbations. In terms of accuracy and OvR, our method even outperforms previous works in most cases, indicating that the rotation-invariant descriptor helps the assessment model to extract more stable features. The visualization results are also informative to correct the movements; some examples are presented in this paper. The code of this paper and our push-up dataset are publicly available at https://github.com/Kelly510/RehabExerAssess.