Search

In virtual teaching scenarios, head-mounted display (HMD) interactions often employ traditional controller and UI interactions, which are not very conducive to teaching scenarios that require hand training. Existing improvements in this area have primarily focused on replacing controllers with gesture recognition. However, the exclusive use of gesture recognition may have limitations in certain scenarios, such as complex operations or multitasking environments. This study designed and tested an interaction method that combines simple gestures with voice assistance, aiming to offer a more intuitive user experience and enrich related research. A speech classification model was developed that can be activated via a fist-clenching gesture and is capable of recognising specific Chinese voice commands to initiate various UI interfaces, further controlled by pointing gestures. Virtual scenarios were constructed using Unity, with hand tracking achieved through the HTC OpenXR SDK. Within Unity, hand rendering and gesture recognition were facilitated, and interaction with the UI was made possible using the Unity XR Interaction Toolkit. The interaction method was detailed and exemplified using a teacher training simulation system, including sample code provision. Following this, an empirical test involving 20 participants was conducted, comparing the gesture-plus-voice operation to the traditional controller operation, both quantitatively and qualitatively. The data suggests that while there is no significant difference in task completion time between the two methods, the combined gesture and voice method received positive feedback in terms of user experience, indicating a promising direction for such interactive methods. Future work could involve adding more gestures and expanding the model training dataset to realize additional interactive functions, meeting diverse virtual teaching needs.

The detection and tracking of gesture targets is an important aspect in the dynamic gesture recognition. To meet the accuracy and speed requirements of human-computer interaction for dynamic gesture recognition, this study explores long-term gesture recognition under monocular RGB cameras. This study uses an improved particle swarm optimization algorithm as the feature extraction method, and introduces a mixed Gaussian model and kernel correlation filtering to complete gesture detection and tracking. And it has constructed a dynamic gesture tracking model on the ground of kernel correlation filtering. The experimental results show that the skin color based gesture detection algorithm has the minimum average relative error value of 0.321 on different datasets, with accuracy and recall rates higher than 0.8. The maximum correlation coefficient R-squared value is 0.823, and the detection speed reaches 36.32 frames per second. And this detection method has high repeatability on different datasets, with better detection accuracy for different gesture targets. The F1 value of the gesture tracking model has the largest area of the receiver operation characteristic curve, and the two error values of the model are small, resulting in better gesture tracking performance. In human-computer interaction systems, the detection accuracy and target rejection rate of this method have been significantly improved, and the subjective evaluation of the interaction system by the subjects is relatively high, resulting in good application effects. This study enriches the theoretical foundation of dynamic gesture detection and tracking technology, and improves the quality level of gesture tracking in the field of human-computer interaction. This helps to expand the application scope of human-computer interaction.

Utilization of hand-tracking cameras, such as Leap, for hand rehabilitation and functional assessments is an innovative approach to providing affordable alternatives for people with disabilities. However, prior to deploying these commercially-available tools, a thorough evaluation of their performance for disabled populations is necessary. In this study, we provide an in-depth analysis of the accuracy of Leap’s hand-tracking feature for both individuals with and without upper-body disabilities for common dynamic tasks used in rehabilitation. Leap is compared against motion capture with conventional techniques such as signal correlations, mean absolute errors, and digit segment length estimation. We also propose the use of dimensionality reduction techniques, such as Principal Component Analysis (PCA), to capture the complex, high-dimensional signal spaces of the hand. We found that Leap’s hand-tracking performance did not differ between individuals with and without disabilities, yielding average signal correlations between 0.7-0.9. Both low and high mean absolute errors (between 10-80mm) were observed across participants. Overall, Leap did well with general hand posture tracking, with the largest errors associated with the tracking of the index finger. Leap’s hand model was found to be most inaccurate in the proximal digit segment, underestimating digit lengths with errors as high as 18mm. Using PCA to quantify differences between the high-dimensional spaces of Leap and motion capture showed that high correlations between latent space projections were associated with high accuracy in the original signal space. These results point to the potential of low-dimensional representations of complex hand movements to support hand rehabilitation and assessment.

Utilization of hand-tracking cameras, such as Leap, for hand rehabilitation and functional assessments is an innovative approach to providing affordable alternatives for people with disabilities. However, prior to deploying these commercially-available tools, a thorough evaluation of their performance for disabled populations is necessary. In this study, we provide an in-depth analysis of the accuracy of Leap’s hand-tracking feature for both individuals with and without upper-body disabilities for common dynamic tasks used in rehabilitation. Leap is compared against motion capture with conventional techniques such as signal correlations, mean absolute errors, and digit segment length estimation. We also propose the use of dimensionality reduction techniques, such as Principal Component Analysis (PCA), to capture the complex, high-dimensional signal spaces of the hand. We found that Leap’s hand-tracking performance did not differ between individuals with and without disabilities, yielding average signal correlations between 0.7-0.9. Both low and high mean absolute errors (between 10-80mm) were observed across participants. Overall, Leap did well with general hand posture tracking, with the largest errors associated with the tracking of the index finger. Leap’s hand model was found to be most inaccurate in the proximal digit segment, underestimating digit lengths with errors as high as 18mm. Using PCA to quantify differences between the high-dimensional spaces of Leap and motion capture showed that high correlations between latent space projections were associated with high accuracy in the original signal space. These results point to the potential of low-dimensional representations of complex hand movements to support hand rehabilitation and assessment. [ABSTRACT FROM AUTHOR]

In hazardous environments, where conditions present risks for humans, the maintenance and interventions are often done with teleoperated remote systems or mobile robotic manipulators to avoid human exposure to dangers. The increasing need for safe and efficient teleoperation requires advanced environmental awareness and collision avoidance. The up-to-date screen-based 2D or 3D interfaces do not fully allow the operator to immerse in the controlled scenario. This problem can be addressed with the emerging Mixed Reality (MR) technologies with Head-Mounted Devices (HMDs) that offer stereoscopic immersion and interaction with virtual objects. Such human-robot interfaces have not yet been demonstrated in telerobotic interventions in particle physics accelerators. Moreover, the operations often require a few experts to collaborate, which increases the system complexity and requires sharing an Augmented Reality (AR) workspace. The multi-user mobile telerobotics in hazardous environments with shared control in the AR has not yet been approached in the state-of-the-art. In this work, the developed MR human-robot interface using the AR HMD is presented. The interface adapts to the constrained wireless networks in particle accelerator facilities and provides reliable high-precision interaction and specialized visualization. The multimodal operation uses hands, eyes and user motion tracking, and voice recognition for control, as well as offers video, 3D point cloud and audio feedback from the robot. Multiple experts can collaborate in the AR workspace locally or remotely, and share or monitor the robot’s control. Ten operators tested the interface in intervention scenarios in the European Organization for Nuclear Research (CERN) with complete network characterization and measurements to conclude if operational requirements were met and if the network architecture could support single and multi-user communication load. The interface system has proved to be operationally ready at the Technical Readiness Level (TRL) 8 and was validated through successful demonstration in single and multi-user missions. Some system limitations and further work areas were identified, such as optimizing the network architecture for multi-user scenarios or high-level interface actions applying automatic interaction strategies depending on network conditions.

In this paper, we focus on the problem of applying domain randomization to produce synthetic datasets for training depth image segmentation models for the task of hand localization. We provide new synthetic datasets for industrial environments suitable for various hand tracking applications, as well as ready-to-use pre-trained models. The presented datasets are analyzed to evaluate the characteristics of these datasets that affect the generalizability of the trained models, and recommendations are given for adapting the simulation environment to achieve satisfactory results when creating datasets for specialized applications. Our approach is not limited by the shortcomings of standard analytical methods, such as color, specific gestures, or hand orientation. The models in this paper were trained solely on a synthetic dataset and were never trained on real camera images; nevertheless, we demonstrate that our most diverse datasets allow the models to achieve up to 90% accuracy. The proposed hand localization system is designed for industrial applications where the operator shares the workspace with the robot.

Air-writing recognition has received wide attention due to its potential application in intelligent systems. To date, some of the fundamental problems in isolated writing have not been addressed effectively. This paper presents a simple yet effective air-writing recognition approach based on deep convolutional neural networks (CNNs). A robust and efficient hand tracking algorithm is proposed to extract air-writing trajectories collected by a single web camera. The algorithm addresses the push-to-write problem and avoids restrictions on the users’ writing without using a delimiter and an imaginary box. A novel preprocessing scheme is also presented to convert the writing trajectory into appropriate forms of data, making the CNNs trained with these forms of data simpler and more effective. Experimental results indicate that the proposed approach not only obtains much higher recognition accuracy but also reduces the network complexity significantly compared to the popular image-based methods.

Robust people tracking in an indoor or outdoor environment is an important feature for fitness applications, surveillance systems for security, and assisted living for the elderly. Camera-based systems offer the state of the art approach but raise privacy concerns and consume a significant amount of power for the sensor and the processing. On the other hand, modern millimeter-wave Radar sensors provide a low-cost and low-power alternative with very good resolution in range, radial velocity and angle of arrival. In this work, we use two popular algorithms, for tracking people using Radar, and compare them with our novel method. We show that our method is robust in certain corner cases and is suitable for scenarios with many users in close proximity. In addition, we present two consumer applications that are possible with a Radar system, multi-user macro gesture recognition and hand tracking. We believe that the combination of above methods will open new gateways for consumer applications. [ABSTRACT FROM AUTHOR]

This article proposes a new method for the tracking of three-dimensional (3-D) hand palms from the whole human standing body using fuzzy rule-based fusion of explainable fuzzy features from a monocular video. The characteristics of this method include visually and linguistically explainable fuzzy features and rules and computational efficiency. This article first tracks the 2-D palms using the following four fuzzy features: optical flows; the degree of a pixel in the foreground; skin color information; and the search area around a hand palm candidate from a segmented body. Afterward, a fuzzy system (FS) is proposed to fuse the four fuzzy features to estimate the 2D- palm positions. Localization of the elbows is based on the estimated palm locations, human body skeletons, and body contour. The 2-D palms and elbows are tracked using a modified particle filter. To estimate the depth of each palm, the locations of the palm and elbow are fed as inputs to a neural FS. The 3-D palm tracking result is applied to a robot upper-body imitation system. Experiments with comparisons of different hand palm tracking methods are performed to verify the real-time computational ability and accuracy of the proposed method. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel approach for 3D hand tracking in real-time from a set of depth images. In each frame, our approach initializes hand pose with learning and then jointly optimizes the hand pose and shape. For pose initialization, we propose a gesture classification and root location network (GCRL), which can capture the meaningful topological structure of the hand to estimate the gesture and root location of the hand. With the per-frame initialization, our approach can rapidly recover from tracking failures. For optimization, unlike most existing methods that have been using a fixed-size hand model or manual calibration, we propose a hand gesture-guided optimization strategy to estimate pose and shape iteratively, which makes the tracking results more accuracy. Experiments on three challenging datasets show that our proposed approach achieves similar accuracy as state-of-the-art approaches, while runs on a low computational resource (without GPU).

Human hands engage in interactive activities in many practical working scenarios, among which the interactions between human hands and objects are the most common. Tracking the movement of the human hand during hand-object interactions is an important research task that is also challenging due to the high-dimensionality and occlusions. In this paper, we track hand-object interactions from depth observations with a model-based method. To overcome the difficulties of optimum searching in the hand-object high-dimensional space, we propose a new algorithm - collaborative differential evolution filtering (CoDEF) - for tracking hand-object interactions. The proposed CoDEF algorithm integrates the differential evolution (DE) algorithm into a particle filtering (PF) framework to accelerate the convergence of particles. Particles are driven to the regions with a high probability by optimizing the matching error under the current observation with DE. To decompose the state space and decrease the complexity of optimum searching, CoDEF tracks the movement of the hand and object by using two collaborative trackers. Based on the proposed CoDEF algorithm, we develop a model-based tracking system with 3D graphic techniques. According to the experimental results, the proposed CoDEF algorithm can achieve robust tracking of hand-object interactions using fewer particles.

In this paper, an algorithm is presented for estimating scene flow, which is a richer, 3D analog of optical flow. The approach operates orders of magnitude faster than alternative techniques and is well suited to further performance gains through parallelized implementation. The algorithm employs multiple hypotheses to deal with motion ambiguities, rather than the traditional smoothness constraints, removing oversmoothing errors and providing significant performance improvements on benchmark data, over the previous state of the art. The approach is flexible and capable of operating with any combination of appearance and/or depth sensors, in any setup, simultaneously estimating the structure and motion if necessary. Additionally, the algorithm propagates information over time to resolve ambiguities, rather than performing an isolated estimation at each frame, as in contemporary approaches. Approaches to smoothing the motion field without sacrificing the benefits of multiple hypotheses are explored, and a probabilistic approach to occlusion estimation is demonstrated, leading to 10 and 15 percent improved performance, respectively. Finally, a data-driven tracking approach is described, and used to estimate the 3D trajectories of hands during sign language, without the need to model complex appearance variations at each viewpoint. [ABSTRACT FROM AUTHOR]

This paper presents an improved hand tracking system using pixel-based hierarchical-feature AdaBoosting (PBHFA), skin color segmentation, and codebook (CB) background cancelation. The proposed PBH feature significantly reduces the training time by a factor of at least 1440 compared to the traditional Haar-like feature. Moreover, lower computation and high tracking accuracy are also provided simultaneously. Yet, one of the disadvantages of the PBHFA is the false positive which is the consequence of the appearance of complex background in positive samples. To effectively reduce the false positive rate, the skin color segmentation and the foreground detection by applying the CB model are catered for rejecting all of the candidates which are not hand targets. As documented in the experimental results, the proposed system can achieve promising results, and thus it can be considered as an effective candidate in handling practical applications which require hand postures. [ABSTRACT FROM PUBLISHER]

A novel real-time algorithm for head and hand tracking is proposed in this paper. This approach is based on data from a range camera, which is exploited to resolve ambiguities and overlaps. The position of the head is estimated with a depth-based template matching, its robustness being reinforced with an adaptive search zone. Hands are detected in a bounding box attached to the head estimate, so that the user may move freely in the scene. A simple method to decide whether the hands are open or closed is also included in the proposal. Experimental results show high robustness against partial occlusions and fast movements. Accurate hand trajectories may be extracted from the estimated hand positions, and may be used for interactive applications as well as for gesture classification purposes. [ABSTRACT FROM PUBLISHER]

Hand motion tracking is useful in human-computer interface and many other applications requiring human-machine interactions. In this work, permanent magnets and contactless magnetic sensors are used to track finger motion. A magnet patch is affixed to each fingernail to mark the location and orientation of the fingertip. When fingers move, the combined magnetic fields produced by the magnets at fingertips are recorded by a set of magnetic sensors around a wristband. The recorded data from the sensors are utilized to calculate the magnets' locations and orientations, which can provide estimate of the hand posture based on a geometric hand model. [ABSTRACT FROM PUBLISHER]

In this paper, we present a vision-based human–computer interaction system, which integrates control components using multiple gestures, including eye gaze, head pose, hand pointing, and mouth motions. To track head, eye, and mouth movements, we present a two-camera system that detects the face from a fixed, wide-angle camera, estimates a rough location for the eye region using an eye detector based on topographic features, and directs another active pan-tilt-zoom camera to focus in on this eye region. We also propose a novel eye gaze estimation approach for point-of-regard (POR) tracking on a viewing screen. To allow for greater head pose freedom, we developed a new calibration approach to find the 3-D eyeball location, eyeball radius, and fovea position. Moreover, in order to get the optical axis, we create a 3-D iris disk by mapping both the iris center and iris contour points to the eyeball sphere. We then rotate the fovea accordingly and compute the final, visual axis gaze direction. This part of the system permits natural, non-intrusive, pose-invariant POR estimation from a distance without resorting to infrared or complex hardware setups. We also propose and integrate a two-camera hand pointing estimation algorithm for hand gesture tracking in 3-D from a distance. The algorithms of gaze pointing and hand finger pointing are evaluated individually, and the feasibility of the entire system is validated through two interactive information visualization applications. [ABSTRACT FROM PUBLISHER]

We describe a novel markerless camera tracking approach and user interaction methodology for augmented reality (AR) on unprepared tabletop environments. We propose a real-time system architecture that combines two types of feature tracking. Distinctive image features of the scene are detected and tracked frame-to-frame by computing optical flow. In order to achieve real-time performance, multiple operations are processed in a synchronized multithreaded manner: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features, and rendering an output frame. We also introduce user interaction methodology for establishing a global coordinate system and for placing virtual objects in the AR environment by tracking a user's outstretched hand and estimating a camera pose relative to it. We evaluate the speed and accuracy of our hybrid feature tracking approach and demonstrate a proof-of-concept application for enabling AR in unprepared tabletop environments, using bare hands for interaction. [ABSTRACT FROM AUTHOR]

Research in automatic analysis of sign language has largely focused on recognizing the lexical (or citation) form of sign gestures as they appear in continuous signing, and developing algorithms that scale well to large vocabularies. However, successful recognition of lexical signs is not sufficient for a full understanding of sign language communication. Nonmanual signals and grammatical processes which result in systematic variations in sign appearance are integral aspects of this communication but have received comparatively little attention in the literature. In this survey, we examine data acquisition, feature extraction and classification methods employed for the analysis of sign language gestures. These are discussed with respect to issues such as modeling transitions between signs in continuous signing, modeling inflectional processes, signer independence, and adaptation. We further examine works that attempt to analyze nonmanual signals and discuss issues related to integrating these with (hand) sign gestures. We also discuss the overall progress toward a true test of sign recognition systems--dealing with natural signing by native signers. We suggest some future directions for this research and also point to contributions it can make to other fields of research. Web-based supplemental materials (appendicies) which contain several illustrative examples and videos of signing can be found at www.computer.org/publications/dlib. [ABSTRACT FROM AUTHOR]

A novel model-based approach to 3D hand tracking from monocular video is presented. The 3D hand pose, the hand texture, and the illuminant are dynamically estimated through minimization of an objective function. Derived from an inverse problem formulation, the objective function enables explicit use of temporal texture continuity and shading information while handling important self-occlusions and time-varying illumination. The minimization is done efficiently using a quasi-Newton method, for which we provide a rigorous derivation of the objective function gradient. Particular attention is given to terms related to the change of visibility near self-occlusion boundaries that are neglected in existing formulations. To this end, we introduce new occlusion forces and show that using all gradient terms greatly improves the performance of the method. Qualitative and quantitative experimental results demonstrate the potential of the approach. [ABSTRACT FROM PUBLISHER]

This paper addresses the issue of matching rigid and articulated shapes through probabilistic point registration. The problem is recast into a missing data framework where unknown correspondences are handled via mixture models. Adopting a maximum likelihood principle, we introduce an innovative EM-like algorithm, namely, the Expectation Conditional Maximization for Point Registration (ECMPR) algorithm. The algorithm allows the use of general covariance matrices for the mixture model components and improves over the isotropic covariance case. We analyze in detail the associated consequences in terms of estimation of the registration parameters, and propose an optimal method for estimating the rotational and translational parameters based on semidefinite positive relaxation. We extend rigid registration to articulated registration. Robustness is ensured by detecting and rejecting outliers through the addition of a uniform component to the Gaussian mixture model at hand. We provide an in-depth analysis of our method and compare it both theoretically and experimentally with other robust methods for point registration. [ABSTRACT FROM AUTHOR]