Search

Accurate 3D tracking of hand and fingers movements poses significant challenges in computer vision. The potential applications span across multiple domains, including human–computer interaction, virtual reality, industry, and medicine. While gesture recognition has achieved remarkable accuracy, quantifying fine movements remains a hurdle, particularly in clinical applications where the assessment of hand dysfunctions and rehabilitation training outcomes necessitate precise measurements. Several novel and lightweight frameworks based on Deep Learning have emerged to address this issue; however, their performance in accurately and reliably measuring finger movements requires validation against well-established gold standard systems. In this paper, the aim is to validate the hand-tracking framework implemented by Google MediaPipe Hand (GMH) and an innovative enhanced version, GMH-D, that exploits the depth estimation of an RGB-Depth camera to achieve more accurate tracking of 3D movements. Three dynamic exercises commonly administered by clinicians to assess hand dysfunctions, namely hand opening–closing, single finger tapping and multiple finger tapping are considered. Results demonstrate high temporal and spectral consistency of both frameworks with the gold standard. However, the enhanced GMH-D framework exhibits superior accuracy in spatial measurements compared to the baseline GMH, for both slow and fast movements. Overall, our study contributes to the advancement of hand tracking technology, and the establishment of a validation procedure as a good-practice to prove efficacy of deep-learning-based hand-tracking. Moreover, it proves that GMH-D is a reliable framework for assessing 3D hand movements in clinical applications. • Hands are vital for carrying out daily-life tasks without external aid. • Measuring objectively hand dexterity can support new digital health treatments. • Deep Learning allows for video-based hand tracking of fine motor tasks. • We validate accuracy of Google Mediapipe Hand and GMH-D against motion capture. • Both frameworks achieve good-to-high spatial and temporal hand tracking performance. [ABSTRACT FROM AUTHOR]

Parkinson's Disease (PD) demands early diagnosis and frequent assessment of symptoms. In particular, analysing hand movements is pivotal to understand disease progression. Advancements in hand tracking using Deep Learning (DL) allow for the automatic and objective disease evaluation from video recordings of standardised motor tasks, which are the foundation of neurological examinations. In view of this scenario, this narrative review aims to describe the state of the art and the future perspective of DL frameworks for hand tracking in video-based PD assessment. A rigorous search of PubMed, Web of Science, IEEE Explorer, and Scopus until October 2023 using primary keywords such as parkinson, hand tracking, and deep learning was performed to select eligible by focusing on video-based PD assessment through DL-driven hand tracking frameworks After accurate screening, 23 publications met the selection criteria. These studies used various solutions, from well-established pose estimation frameworks, like OpenPose and MediaPipe, to custom deep architectures designed to accurately track hand and finger movements and extract relevant disease features. Estimated hand tracking data were then used to differentiate PD patients from healthy individuals, characterise symptoms such as tremors and bradykinesia, or regress the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS) by automatically assessing clinical tasks such as finger tapping, hand movements, and pronation-supination. DL-driven hand tracking holds promise for PD assessment, offering precise, objective measurements for early diagnosis and monitoring, especially in a telemedicine scenario. However, to ensure clinical acceptance, standardisation and validation are crucial. Future research should prioritise large open datasets, rigorous validation on patients, and the investigation of new frontiers such as tracking hand-hand and hand-object interactions for daily-life tasks assessment. • Assessment of hand impairment is crucial for monitoring Parkinson's disease over time. • Hand tracking methods based on Deep Learning enable objective video-based assessment. • This narrative review does an in-depth evaluation of the state-of-the-art since 2017. • OpenPose, DeepLabCut and MediaPipe result the most popular frameworks in this domain. • Larger datasets, technical validation, and multihand-interactions are often missing. [ABSTRACT FROM AUTHOR]

• Reviewing recent trends in markerless vision-based 3D hand motion tracking • Proposing a new taxonomy based on hand-object interaction constraints • Distinguishing between tracking in isolation and in real/virtual interaction • Studying some datasets in the context of hand-object interaction Unlabelled Image Vision-based 3D hand tracking is a key and popular component for interaction studies in a broad range of domains such as virtual reality (VR), augmented reality (AR) and natural human-computer interaction (HCI). While this research field has been well studied in the last decades, most approaches have considered the human hand in isolation and not in action or in interaction with the surrounding environment. Even the common collaborative and strong interactions with the other hand have been ignored. However, many of today's computer applications require more and more hand-object interactions. Furthermore, employing contextual information about the object in the hand (e.g. the shape, the texture, and the pose) can remarkably constrain the tracking problem. The most studied contextual constraints involve interaction with real objects and not with virtual objects which is still a very big challenge. The goal of this survey is to develop an up-to-date taxonomy of the state-of-the-art vision-based hand pose estimation and tracking methods with a new classification scheme: hand-object interaction constraints. This taxonomy allows us to examine the strengths and weaknesses of the current state of the art and to highlight future trends in the domain. [ABSTRACT FROM AUTHOR]

Abstract Recent research demonstrates that two components of reaching behavior – initiation time (the time elapsed from stimulus presentation to movement initiation) and reach curvature (the degree to which a reach movement deviates from a direct path to the selected response) – exhibit distinct cross-trial dynamics in cognitive control tasks, indicating that these components of behavior reflect two dissociable processes underlying cognitive control: a threshold adjustment process involving the inhibition of motor output and a controlled selection process involving the recruitment of top-down resources to support goal-relevant behavior. The current study investigates the extent to which the cross-trial dynamics previously observed in reaching behavior in the Eriksen flanker task are reflected in event-related potentials during standard button-press responses. Candidate EEG measures of the threshold adjustment process (N2 and Pre-LRP amplitudes) failed to reveal the cross-trial dynamics previously observed in initiation times. Slow wave amplitudes exhibited a close correspondence to the cross-trial dynamics observed in reach curvatures, indicating that the measure is sensitive to some functions of the controlled selection process. Further, LRP slopes presented a close correspondence to the cross-trial dynamics observed in response times, indicating that this measure reflects the combined output of the threshold adjustment process and controlled selection process. The implications of these findings for future research examining the links among behavioral and neural dynamics are discussed. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel method for object manipulation in immersive virtual environments. The proposed technique exploits a novel widget, called “Smart Pin”, to enable the user to select, translate, rotate and scale objects relying entirely on the positional tracking of a single hand. It is, therefore, suitable for several real-world applications where handheld devices are not available, the surrounding environmental conditions make orientation tracking hard, and the non-dominant hand may be involved in different tasks. We evaluated the method with users on two classical tasks such as detail search and docking, comparing it with a successful two-handed manipulation technique (Handlebar). The measured execution efficiency obtained with the two methods was similar, but most users preferred the Smart Pin for its gestural comfort and ease of use. [ABSTRACT FROM AUTHOR]

Construction is an inherently less safe sector than other sectors because it exposes workers to harsh and dangerous working environments. The nature of the construction industry results in a comparatively high incidence of serious injuries and death caused by falls from a height, musculoskeletal disorders and being struck by objects. This paper presents a new concept that can tackle this problem in the future. The central hypothesis of this study is that it is possible to eliminate injuries if we move the human construction worker off-site and remotely link his/her motions to a Robotic Construction Worker (RCW) on-site. As a first steppingstone towards this ultimate goal, two systems essential for the RCW were developed in this study. First, a novel system that combines 3D body and hand position tracking was developed to capture the movements of human construction worker. This combination of tracking enables the capture of changes in the orientations and articulations of the entire human body. Second, a real-time simulation system that connects a human construction worker off-site to a virtual RCW was developed to demonstrate the proposed concept in a variety of construction scenarios. The simulation results demonstrate the future viability of the RCW concept and indicate the promise of this system for eliminating the health and safety risks faced by human construction workers. [ABSTRACT FROM AUTHOR]

In this paper, we propose a hand tracking method which was inspired by the notion of the four dukkha: birth, aging, sickness and death (BASD) in Buddhism. Based on this philosophy, we formalize the hand tracking problem in the BASD framework, and apply it to hand track hand gestures in isolated sign language videos. The proposed BASD method is a novel nature-inspired computational intelligence method which is able to handle complex real-world tracking problem. The proposed BASD framework operates in a manner similar to a standard state-space model, but maintains multiple hypotheses and integrates hypothesis update and propagation mechanisms that resemble the effect of BASD. The survival of the hypothesis relies upon the strength, aging and sickness of existing hypotheses, and new hypotheses are birthed by the fittest pairs of parent hypotheses. These properties resolve the sample impoverishment problem of the particle filter. The estimated hand trajectories show promising results for the American sign language. [ABSTRACT FROM AUTHOR]

This paper presents implementation of a hand gestures tracking system with a fully connected multilayer perceptron and a supervised sequential learning algorithm on low cost FPGAs (Field Programmable Gate Array). Two networks are used in the study. The first network is trained to detect skins and the second identifies face and hand gestures. Also, a binary image is made from the first network, which is fed into the second network as input. Skin network has two output classes, and hand network 6. The first network recognizes skin and non-skin images and the other distinguishes hands from faces while hand gestures are tracked and classified. Both networks are trained in Matlab before each was implemented on a separate FPGA and the two FPGAs are then connected to each other. Finally, the overall result is compared. Both networks are implemented on Xilinx Spartan XC3S1000 4fg456. Two hardware-implemented networks are applied to a video sequence frame by frame. Features of both hands are extracted and the network is trained using a constructed vector to track left and right hands. Results show that the implemented system enjoys a suitable trade-off between accuracy and speed, reaching 208 fps with the least memory usage. This method has been compared to other methods, as well as methods in which only one threshold is used to detect skin color. [ABSTRACT FROM AUTHOR]

Unmarked tracking of hands and fingers is a promising enabler for human–computer interaction. It can also be applied to medical monitoring and handicap aids. This will give much help to the deaf-mutes who have trouble in communicating with others or interact with the computers. In this paper, we present a 3D hand tracking and character recognition algorithm using binocular stereo camera, which can get the crucial depth information of the target. To precisely understand the characters that the user writes in the air, we define a valid gesture indicating that the user is inputting useful information and a null gesture indicating that corresponding movements are not necessary in the characters. Our approach can adapt to angle variation and make the writing freer. We show the benefits of our method by evaluating on several videos. [ABSTRACT FROM AUTHOR]

Object tracking via depth image sequences can output 3D motion trajectories, and is effective in solving many traditional problems such as illumination changes and projective distortion. However, the influence of complex scene in depth-based tracking has received little attention. Depth tracker is susceptible to many kinds of disturbances, such as nearby objects, moving distracters and noise, and these properties need to be considered in the design of a robust tracking algorithm. In this paper we investigated the depth image-based object tracking problem by a typical application: hand tracking. Firstly the properties of depth image and the disturbances are discussed. Secondly two different depth trackers are proposed to handle different scenarios: one is based on depth frame differencing, and the other one is based on the mean shift algorithm which is modified to utilize depth information. Finally, a robust fused tracker is proposed based on the two different trackers. This approach was tested on two groups of depth image sequences recorded by a Kinect device, and the experimental results show that the fused tracker is competent for accurate 3D hand tracking even under very complex scene. [ABSTRACT FROM AUTHOR]

In this paper, we present a robust real-time hand tracking system via multi-cue integration. In practice, the motion information of the hand, such as optical flow, is hard to exploit, because images of hands lack texture. As a result, the integration of the color and motion cues using conventional integration algorithms is difficult. Here, we integrate the motion and color cues from a novel feature point selection view. The hand is tracked using feature points, and the integration is realized during the feature points generation and selection process. In the generation process, a bounding box estimated by the color cue is used to provide a region for the feature points generation. Then, the RCD ( Representative , Compact and Diverse ) criteria are proposed to control the feature point selection process. After the selection process, the feature points are tracked using estimates of the motion of each feature point. The centroid of the feature points in each frame is adopted as the position of the hand. The experimental results show that our integration algorithm outperforms tracking algorithms that only use a single cue. Also the proposed tracking algorithm is more robust in complex environments than other state-of-art algorithms. [ABSTRACT FROM AUTHOR]

Post-stroke patients and people suffering from hand diseases often need rehabilitation therapy. The recovery of original skills, when possible, is closely related to the frequency, quality, and duration of rehabilitative therapy. Rehabilitation gloves are tools used both to facilitate rehabilitation and to control improvements by an evaluation system. Mechanical gloves have high cost, are often cumbersome, are not re-usable and, hence, not usable with the healthy hand to collect patient-specific hand mobility information to which rehabilitation should tend. The approach we propose is the virtual glove, a system that, unlike tools based on mechanical haptic interfaces, uses a set of video cameras surrounding the patient hand to collect a set of synchronized videos used to track hand movements. The hand tracking is associated with a numerical hand model that is used to calculate physical, geometrical and mechanical parameters, and to implement some boundary constraints such as joint dimensions, shape, joint angles, and so on. Besides being accurate, the proposed system is aimed to be low cost, not bulky (touch-less), easy to use, and re-usable. Previous works described the virtual glove general concepts, the hand model, and its characterization including system calibration strategy. The present paper provides the virtual glove overall design, both in real-time and in off-line modalities. In particular, the real-time modality is described and implemented and a marker-based hand tracking algorithm, including a marker positioning, coloring, labeling, detection and classification strategy, is presented for the off-line modality. Moreover, model based hand tracking experimental measurements are reported, discussed and compared with the corresponding poses of the real hand. An error estimation strategy is also presented and used for the collected measurements. System limitations and future work for system improvement are also discussed [ABSTRACT FROM AUTHOR]

Abstract: We propose a real-time model-based 3D hand tracker that combines image regions and the signal from an off-the-shelf 3-axis accelerometer placed on the user''s hand. The visual regions allow the tracker to cope with occlusions, motion blur and background clutter, while the latter aids with the inherent silhouette-pose ambiguities. The accelerometer and tracker are synchronised by casting the calibration problem as one of principal component analysis. Based on the assumption that, often, the number of possible hand configurations is limited by the activity the hand is engaging in, we use a multiclass pose classifier to distinguish between a number of activity dependent articulated hand configurations. We demonstrate the benefits of our method, both qualitatively and quantitatively, on a variety of video sequences and hand configurations and show a proof-of-concept human computer interface based on our system. [Copyright &y& Elsevier]

Abstract: Particle filtering and mean shift (MS) are two successful approaches to visual tracking. Both have their respective strengths and weaknesses. In this paper, we propose to integrate advantages of the two approaches for improved tracking. By incorporating the MS optimization into particle filtering to move particles to local peaks in the likelihood, the proposed mean shift embedded particle filter (MSEPF) improves the sampling efficiency considerably. Our work is conducted in the context of developing a hand control interface for a robotic wheelchair. We realize real-time hand tracking in dynamic environments of the wheelchair using MSEPF. Extensive experimental results demonstrate that MSEPF outperforms the MS tracker and the conventional particle filter in hand tracking. Our approach produces reliable tracking while effectively handling rapid motion and distraction with roughly 85% fewer particles. We also present a simple method for dynamic gesture recognition. The hand control interface based on the proposed algorithms works well in dynamic environments of the wheelchair. [Copyright &y& Elsevier]

Abstract: Recently, an optimization approach for fast visual tracking of articulated structures based on stochastic meta-descent (SMD) has been presented. SMD is a gradient descent with local step size adaptation that combines rapid convergence with excellent scalability. Stochastic sampling helps to avoid local minima in the optimization process. We have extended the SMD algorithm with new features for fast and accurate tracking by adapting the different step sizes between as well as within video frames and by introducing a robust cost function, which incorporates both depths and surface orientations. The advantages of the resulting tracker over state-of-the-art methods are supported through 3D hand tracking experiments. A realistic deformable hand model reinforces the accuracy of our tracker. [Copyright &y& Elsevier]

Abstract: A general hand-tracking algorithm is presented for tracking hands in bimanual movements. The problem is approached from a neuroscience point of view. Using a dynamic model and some motor control phenomena, the movement of the hands during a bimanual movement is recognised. By capturing the hands velocities and recognising the bimanual synchronisation, the hands are tracked and reacquired in different types of movements. This includes tracking hands when they are separated, and reacquiring them at the end of hand–hand occlusion. Different applications are demonstrated including active vision where the camera view direction and position change. [Copyright &y& Elsevier]

This paper presents a gesture recognition system for visualization navigation. Scientists are interested in developing interactive settings for exploring large data sets in an intuitive environment. The input consists of registered 3-D data. A geometric method using Bezier curves is used for the trajectory analysis and classification of gestures. The hand gesture speed is incorporated into the algorithm to enable correct recognition from trajectories with variations in hand speed. The method is robust and reliable: correct hand identification rate is 99.9% (from 1641 frames), modes of hand movements are correct 95.6% of the time, recognition rate (given the right mode) is 97.9%. An application to gesture-controlled visualization of 3D bioinformatics data is also presented. [Copyright &y& Elsevier]

In this paper, we introduce a hand gesture recognition system to recognize continuous gesture before stationary background. The system consists of four modules: a real time hand tracking and extraction, feature extraction, hidden Markov model (HMM) training, and gesture recognition. First, we apply a real-time hand tracking and extraction algorithm to trace the moving hand and extract the hand region, then we use the Fourier descriptor (FD) to characterize spatial features and the motion analysis to characterize the temporal features. We combine the spatial and temporal features of the input image sequence as our feature vector. After having extracted the feature vectors, we apply HMMs to recognize the input gesture. The gesture to be recognized is separately scored against different HMMs. The model with the highest score indicates the corresponding gesture. In the experiments, we have tested our system to recognize 20 different gestures, and the recognizing rate is above 90%. [Copyright &y& Elsevier]

A real-time computer vision system is described for tracking hands thus enabling behavioural events to be interpreted. Forearms are tracked to provide structural context, enabling mutual occlusion, which occurs when hands cross one another, to be handled robustly. No prior skin colour models are used. Instead adaptive appearance models are learned on-line. A contour distance transform is used to control model adaptation and to fit 2D geometric models robustly. Hands can be tracked whether clothed or unclothed. Results are given for a ‘smart desk’ and an in-vehicle application. The ability to interpret behavioural events of interest when tracking a vehicle driver''s hands is described. [Copyright &y& Elsevier]

• Our proposed framework can robustly infer 3D hand pose without requiring a prior. • Novel keypoint-based hand detector robust to confusing background and adjacent hands. • Two anatomy-based constraints for aiding 3D hand pose estimation network performance. • An end-to-end pipeline with state-of-the-art performance on several datasets. This work addresses the challenging problem of unconstrained 3D hand pose estimation using monocular RGB images. Most of the existing approaches assume some prior knowledge of hand (such as hand locations and side information) is available for 3D hand pose estimation. This restricts their use in unconstrained environments. Therefore, we present an end-to-end framework that robustly predicts hand prior information and accurately infers 3D hand pose by learning ConvNet models while only using keypoint annotations. To enhance the hand detector's robustness, we propose a novel keypoint-based method to simultaneously predict hand regions and side labels, unlike existing methods that suffer from background color confusion caused by using segmentation or detection-based technology. Moreover, inspired by the human hand's biological structure, we introduce two geometric constraints directly into the 3D coordinates prediction that further improves its performance. Experimental results show that our proposed framework outperforms the state-of-art methods on standard benchmark datasets while providing robust predictions. [ABSTRACT FROM AUTHOR]