Search

Purpose: The operating microscope plays a central role in middle and inner ear procedures that involve working within tightly confined spaces under limited exposure. Augmented reality (AR) may improve surgical guidance by combining preoperative computed tomography (CT) imaging that can provide precise anatomical information, with intraoperative microscope video feed. With current technology, the operator must manually interact with the AR interface using a computer. The latter poses a disruption in the surgical flow and is suboptimal for maintaining the sterility of the operating environment. The purpose of this study was to implement and evaluate free-hand interaction concepts leveraging hand tracking and gesture recognition as an attempt to reduce the disruption during surgery and improve human-computer interaction. Methods: An electromagnetically tracked surgical microscope was calibrated using a custom 3D printed calibration board. This allowed the augmentation of the microscope feed with segmented preoperative CT-derived virtual models. Ultraleap's Leap Motion Controller 2 was coupled to the microscope and used to implement hand-tracking capabilities. End-user feedback was gathered from a surgeon during development. Finally, users were asked to complete tasks that involved interacting with the virtual models, aligning them to physical targets, and adjusting the AR visualization. Results: Following observations and user feedback, we upgraded the functionalities of the hand interaction system. User feedback showed the users' preference for the new interaction concepts that provided minimal disruption of the surgical workflow and more intuitive interaction with the virtual content. Conclusion: We integrated hand interaction concepts, typically used with head-mounted displays (HMDs), into a surgical stereo microscope system intended for AR in otologic microsurgery. The concepts presented in this study demonstrated a more favorable approach to human-computer interaction in a surgical context. They hold potential for a more efficient execution of surgical tasks under microscopic AR guidance. [ABSTRACT FROM AUTHOR]

Hand tracking is relevant to such a variety of applications including human-robot interaction (HRI), human-computer interaction (HCI), virtual reality (VR), and augmented reality (AR). Accurate and robust hand tracking however is challenging due to the intricacies of dynamic motion within small space and the complex interactions with nearby objects, coupled with the hurdles in real-time hand mesh reconstruction. In this paper, we conduct a comprehensive examination and analysis of existing hand tracking technologies. Through the review of major works in the literature, we have discovered numerous studies employing a diverse array of sensors, leading us to propose their categorization into seven types: vision, soft wearable, encoder, magnetic, inertial measurement unit (IMU), electromyography (EMG), and the fusion of sensor modalities. Our findings indicate that no singular solution surpasses all others, attributing to the inherent limitations of using a single sensor modality. As a result, we assert that integrating multiple sensor modalities presents a viable path toward devising a superior hand tracking solution. Ultimately, this survey paper aims to bolster interdisciplinary research efforts across the spectrum of hand tracking technologies, thereby contributing to the advancement of the field. [ABSTRACT FROM AUTHOR]

The detection and tracking of gesture targets is an important aspect in the dynamic gesture recognition. To meet the accuracy and speed requirements of human-computer interaction for dynamic gesture recognition, this study explores long-term gesture recognition under monocular RGB cameras. This study uses an improved particle swarm optimization algorithm as the feature extraction method, and introduces a mixed Gaussian model and kernel correlation filtering to complete gesture detection and tracking. And it has constructed a dynamic gesture tracking model on the ground of kernel correlation filtering. The experimental results show that the skin color based gesture detection algorithm has the minimum average relative error value of 0.321 on different datasets, with accuracy and recall rates higher than 0.8. The maximum correlation coefficient R-squared value is 0.823, and the detection speed reaches 36.32 frames per second. And this detection method has high repeatability on different datasets, with better detection accuracy for different gesture targets. The F1 value of the gesture tracking model has the largest area of the receiver operation characteristic curve, and the two error values of the model are small, resulting in better gesture tracking performance. In human-computer interaction systems, the detection accuracy and target rejection rate of this method have been significantly improved, and the subjective evaluation of the interaction system by the subjects is relatively high, resulting in good application effects. This study enriches the theoretical foundation of dynamic gesture detection and tracking technology, and improves the quality level of gesture tracking in the field of human-computer interaction. This helps to expand the application scope of human-computer interaction.

In virtual teaching scenarios, head-mounted display (HMD) interactions often employ traditional controller and UI interactions, which are not very conducive to teaching scenarios that require hand training. Existing improvements in this area have primarily focused on replacing controllers with gesture recognition. However, the exclusive use of gesture recognition may have limitations in certain scenarios, such as complex operations or multitasking environments. This study designed and tested an interaction method that combines simple gestures with voice assistance, aiming to offer a more intuitive user experience and enrich related research. A speech classification model was developed that can be activated via a fist-clenching gesture and is capable of recognising specific Chinese voice commands to initiate various UI interfaces, further controlled by pointing gestures. Virtual scenarios were constructed using Unity, with hand tracking achieved through the HTC OpenXR SDK. Within Unity, hand rendering and gesture recognition were facilitated, and interaction with the UI was made possible using the Unity XR Interaction Toolkit. The interaction method was detailed and exemplified using a teacher training simulation system, including sample code provision. Following this, an empirical test involving 20 participants was conducted, comparing the gesture-plus-voice operation to the traditional controller operation, both quantitatively and qualitatively. The data suggests that while there is no significant difference in task completion time between the two methods, the combined gesture and voice method received positive feedback in terms of user experience, indicating a promising direction for such interactive methods. Future work could involve adding more gestures and expanding the model training dataset to realize additional interactive functions, meeting diverse virtual teaching needs.

Hand gesture recognition (HGR) is an essential technology with applications spanning human-computer interaction, robotics, augmented reality, and virtual reality. This technology enables more natural and effortless interaction with computers, resulting in an enhanced user experience. As HGR adoption increases, it plays a crucial role in bridging the gap between humans and technology, facilitating seamless communication and interaction. In this study, a novel deep learning approach is proposed for the development of a Hand Gesture Interface (HGI) that enables the control of graphical user interfaces without physical touch on personal computers. The methodology encompasses the analysis, design, implementation, and deployment of the HGI. Experimental results on a hand gesture recognition system indicate that the proposed approach improves accuracy and reduces response time compared to existing methods. The system is capable of controlling various multimedia applications, including VLC media player, Microsoft Word, and PowerPoint. In conclusion, this approach offers a promising solution for the development of HGIs that facilitate efficient and intuitive interactions with computers, making communication more natural and accessible for users. [ABSTRACT FROM AUTHOR]

The rapidly evolving field of Virtual Reality (VR)-based Human–Computer Interaction (HCI) presents a significant demand for robust and accurate hand tracking solutions. Current technologies, predominantly based on single-sensing modalities, fall short in providing comprehensive information capture due to susceptibility to occlusions and environmental factors. In this paper, we introduce a novel sensor fusion approach combined with a Long Short-Term Memory (LSTM)-based algorithm for enhanced hand tracking in VR-based HCI. Our system employs six Leap Motion controllers, two RealSense depth cameras, and two Myo armbands to yield a multi-modal data capture. This rich data set is then processed using LSTM, ensuring the accurate real-time tracking of complex hand movements. The proposed system provides a powerful tool for intuitive and immersive interactions in VR environments. [ABSTRACT FROM AUTHOR]

Hand detection and tracking are key components in many computer vision applications, including hand pose estimation and gesture recognition for human–computer interaction systems, virtual reality, and augmented reality. Despite their importance, reliable hand detection in cluttered scenes remains a challenge. This study explores the use of deep learning techniques for fast and robust hand detection and tracking. A novel algorithm is proposed by combining the Kernelized Correlation Filter (KCF) tracker with the Single-Shot Detection (SSD) method. This integration enables the detection and tracking of hands in challenging environments, such as cluttered backgrounds and occlusions. The SSD algorithm helps reinitialize the KCF tracker when it fails or encounters drift issues due to sudden changes in hand gestures or fast movements. Testing in challenging scenes showed that the proposed tracker achieved a tracking rate of over 90% and a speed of 17 frames per second (FPS). Comparison with the KCF tracker on 17 video sequences revealed an average improvement of 13.31% in tracking detection rate (TRDR) and 27.04% in object detection error (OTE). Additional comparison with MediaPipe hand tracker on 10 hand gesture videos taken from the Intelligent Biometric Group Hand Tracking (IBGHT) dataset showed that the proposed method outperformed the MediaPipe hand tracker in terms of overall TRDR and tracking speed. The results demonstrate the promising potential of the proposed method for long-sequence tracking stability, reducing drift issues, and improving tracking performance during occlusions. [ABSTRACT FROM AUTHOR]

The rapidly evolving field of Virtual Reality (VR)-based Human–Computer Interaction (HCI) presents a significant demand for robust and accurate hand tracking solutions. Current technologies, predominantly based on single-sensing modalities, fall short in providing comprehensive information capture due to susceptibility to occlusions and environmental factors. In this paper, we introduce a novel sensor fusion approach combined with a Long Short-Term Memory (LSTM)-based algorithm for enhanced hand tracking in VR-based HCI. Our system employs six Leap Motion controllers, two RealSense depth cameras, and two Myo armbands to yield a multi-modal data capture. This rich data set is then processed using LSTM, ensuring the accurate real-time tracking of complex hand movements. The proposed system provides a powerful tool for intuitive and immersive interactions in VR environments.

Hand detection and tracking are key components in many computer vision applications, including hand pose estimation and gesture recognition for human–computer interaction systems, virtual reality, and augmented reality. Despite their importance, reliable hand detection in cluttered scenes remains a challenge. This study explores the use of deep learning techniques for fast and robust hand detection and tracking. A novel algorithm is proposed by combining the Kernelized Correlation Filter (KCF) tracker with the Single-Shot Detection (SSD) method. This integration enables the detection and tracking of hands in challenging environments, such as cluttered backgrounds and occlusions. The SSD algorithm helps reinitialize the KCF tracker when it fails or encounters drift issues due to sudden changes in hand gestures or fast movements. Testing in challenging scenes showed that the proposed tracker achieved a tracking rate of over 90% and a speed of 17 frames per second (FPS). Comparison with the KCF tracker on 17 video sequences revealed an average improvement of 13.31% in tracking detection rate (TRDR) and 27.04% in object detection error (OTE). Additional comparison with MediaPipe hand tracker on 10 hand gesture videos taken from the Intelligent Biometric Group Hand Tracking (IBGHT) dataset showed that the proposed method outperformed the MediaPipe hand tracker in terms of overall TRDR and tracking speed. The results demonstrate the promising potential of the proposed method for long-sequence tracking stability, reducing drift issues, and improving tracking performance during occlusions.

Tracking and reconstructing the 3D pose and geometry of two hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, or sign language recognition. Existing works are either limited to simpler tracking settings (e.g., considering only a single hand or two spatially separated hands), or rely on less ubiquitous sensors, such as depth cameras. In contrast, in this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera that explicitly considers close interactions. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN that regresses multiple complementary pieces of information, including segmentation, dense matchings to a 3D hand model, and 2D keypoint positions, together with newly proposed intra-hand relative depth and inter-hand distance maps. These predictions are subsequently used in a generative model fitting framework in order to estimate pose and shape parameters of a 3D hand model for both hands. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline through an extensive ablation study. Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions. Moreover, our method even performs on-par with depth-based real-time methods. [ABSTRACT FROM AUTHOR]

• Reviewing recent trends in markerless vision-based 3D hand motion tracking • Proposing a new taxonomy based on hand-object interaction constraints • Distinguishing between tracking in isolation and in real/virtual interaction • Studying some datasets in the context of hand-object interaction Unlabelled Image Vision-based 3D hand tracking is a key and popular component for interaction studies in a broad range of domains such as virtual reality (VR), augmented reality (AR) and natural human-computer interaction (HCI). While this research field has been well studied in the last decades, most approaches have considered the human hand in isolation and not in action or in interaction with the surrounding environment. Even the common collaborative and strong interactions with the other hand have been ignored. However, many of today's computer applications require more and more hand-object interactions. Furthermore, employing contextual information about the object in the hand (e.g. the shape, the texture, and the pose) can remarkably constrain the tracking problem. The most studied contextual constraints involve interaction with real objects and not with virtual objects which is still a very big challenge. The goal of this survey is to develop an up-to-date taxonomy of the state-of-the-art vision-based hand pose estimation and tracking methods with a new classification scheme: hand-object interaction constraints. This taxonomy allows us to examine the strengths and weaknesses of the current state of the art and to highlight future trends in the domain. [ABSTRACT FROM AUTHOR]

We present an interactive system using one RGB-D sensor, which allows a user to use bare hands to perform sand drawing. Our system supports the common sand drawing functions, such as sand erosion, sand spilling, and sand leaking. To use hands to manipulate the virtual sand, we design four key hand gestures. The idea is that the gesture of one hand controls the drawing actions. The motion and gesture of the other hand control the drawing positions. There are three major steps. First, our system adopts a vision-based bare-hand detection method which computes the hand position and recognizes the hand gestures. Second, the drawing positions and the drawing actions are sent to a sand drawing subsystem. Finally, the subsystem performs the sand drawing actions. Experimental results show that our system enables users to draw a rich variety of sand pictures. [ABSTRACT FROM AUTHOR]

Articulated hand tracking systems have been commonly used in virtual reality applications, including systems with human–computer interaction or interaction with game consoles; hand pose estimation has various other applications comprising sign language recognition and animation synthesis. The advanced technological achievements in motion capture over the last decade allow data acquisition with high accuracy and low cost. However, due to the high complexity of the human hand, it is still challenging to animate a hand model able to deal in details with the anatomical and physiological constraints of the hand. In this paper, we present a simple and efficient methodology for tracking and reconstructing 3D hand poses. Using an optical motion capture system, where markers are positioned at strategic points, we manage to acquire the movement of the hand and establish its orientation using a minimum number of markers. An Inverse Kinematics solver was then employed to control the postures of the hand, subject to physiological constraints that restrict the allowed movements to a feasible and natural set. The proposed methodology produces smooth and biomechanically correct movements, while the required processing time remains low, enabling an effective real-time hand motion tracking and reconstruction system. [ABSTRACT FROM AUTHOR]

In this paper, We proposed a novel hand-gesture recognition algorithm using concentric-circle expanding and tracing. The proposed algorithm determines region of interest of hand image through preprocessing the original image acquired by web-camera and extracts the feature of hand gesture such as the number of stretched fingers, finger tips and finger bases, angle between the fingers which can be used as intuitive method for of human computer interaction. The proposed algorithm also reduces computational complexity compared with raster scan method through referencing only pixels of concentric-circles. The experimental result shows that the 9 hand gestures can be recognized with an average accuracy of 90.7% and an average algorithm execution time is 78ms. The algorithm is confirmed as a feasible way to a useful input method for virtual reality, augmented reality, mixed reality and perceptual interfaces of human computer interaction. [ABSTRACT FROM AUTHOR]

This paper proposes a vision-based Multi-user Human Computer Interaction (HCI) method for creating augmented reality user interfaces. In the HCI session, one of the users' hands is selected as the active hand. The fingers of the active hand are employed as input devices to trigger functionalities of the application program. To share the token of interaction among the users, the HCI session is modeled as a Finite State Machine (FSM). The FSM is composed of the initial and steady states. In the initial state, the FSM identifies the active hand by tracking the hand with the maximum moving speed. Then the FSM enters the steady state to carry out the HCI session. At the end of each individual HCI cycle, the FSM polls requests from other hands for acquiring the role of active hand. If such requests are sensed, the FSM returns to the initial state to track a new active hand. Otherwise, the HCI session is continuously carried out by the current active hand. Test results show that the resultant user interface is efficient, flexible and practical for users with problems on using ordinary input devices. In a desk-top computer equipped with a 640 × 480 resolution web-camera, the HCI session can be successfully conducted when the operation distance ranges from 30 to 90 cm. [ABSTRACT FROM AUTHOR]

Hand gesture recognition based on computer vision is considered as an efficient approach to establish communication between human and computer. This research provides a novel system to recognize dynamic hand gestures as different functions of mouse. For this purpose, a white glove is utilized in which fingertips have five different colors. Then, based on the functions of the mouse, 11 dynamic hand gestures are defined. In order to track the hand in each frame, the optical flow and GMM algorithms are used. Then, by using the mean and variance of the colors in each RGB plane, trajectories of the fingertips are detected. To make the acquired information richer, the Representative Trajectory of these five trajectories is computed. Features are extracted from the curves through a process inspired by the concept of shape context. In this process, each trajectory is normalized and then the histogram of extracted vector from normalized curve is calculated in a log-polar space. Thus, by this descriptor, a feature vector with length of 672 is created. Using the defined 11 dynamic gestures, a dataset including 220 observations is constructed and by using the aforementioned features, the relevant training matrix is formed. By employing PCA and Sequential Feature Selection techniques, dimension of the feature vector is reduced. For the experiment, different classifiers are applied and the experimental results confirm the privileged performance of the proposed system for the intelligent visual mouse. [ABSTRACT FROM AUTHOR]

Hand tracking is one of the essential elements in vision-based hand gesture recognition systems. It has a great potential as a tool for better human-computer interaction (HCI) by means of communicating in natural and articulate ways. This has motivated an active research concerning with the interpretation of hand tracking for gesture recognition system. However, due to the nature of hand motion which is flexible, erratic and always varies in its appearance, the tracking of human hand using vision-based remains a complex problem. In this paper, we present an efficient method to overcome such difficulties using integration of Adaptive Kalman Filter (AKF) and Eigenhand method. In the proposed method, the tracking task is first carried out by running the Region of Interest (ROI) based tracker. Here, by fusing the skin and motion cues as the main tracking features, the actual hand position in the current frame is measured. This initial measurement is rather inconsistent due to the working principle of ROI based tracker which greatly depends on how effectively the extracted tracking features are. To reduce this inconsistency, the measurement error is minimized by employing the AKF prediction. After the hand position is effectively estimated by the AKF tracker, a low dimensional eigenspace representation, i.e., the Eigenhand, is employed to further improve the tracking performance. This representation is necessary as the AKF tracker only treats the hand image as a set of moving pixel which consequently disregards the detail appearance of the target. Therefore, the incorporated eigenspace representation provides a compact description of the internal target appearance for better object recognition. This eigenspace adaptively learns the current state to reflect the appearance changes of the target image in each frame. The experimental results demonstrate the effectiveness of the proposed tracking algorithm in indoor and outdoor environments where the target objects undergo large pose changes, lighting variation, fast motion and partial occlusion with average detection rate above 97 % at the speed of 35 frame/second (fps). [ABSTRACT FROM AUTHOR]

Human-computer interaction, or HCI for short, concerning how people interact with computers, has long been an important and popular research field. Though not completely realistic, fancy HCI applications such as those shown in the science fiction movies Minority Report and Iron Man have impressively demonstrated the potential and trend of HCI technologies that will be very soon made available. As one can very often observe, compared with traditional keyboard/mouse interfaces, the exclusive use of hands has distinguished itself by enjoying a more intuitive and natural way for communication. Furthermore, the increasingly popular concept of ubiquitous computing has called for convenient and portable input devices, thus making hand gesture inputs even more attractive. For example, a smart phone equipped with the capability of hand gesture recognition could be a good input substitute for its intrinsically small touch screen or keypad. Rather than data gloves, which transfer hand gestures through relatively expensive electronic devices, we are more interested in recognizing the gestures of a bare hand. In this regard, there exist works that can track a 2D articulated hand model. In this paper, we make further improvement in computation efficiency and propose novel interfaces to be coupled with the hand tracking system for more user-friendliness. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a novel, real-time and robust hand tracking system, capable of tracking the articulated hand motion in full degrees of freedom (DOF) using a single depth camera. Unlike most previous systems, our system is able to initialize and recover from tracking loss automatically. This is achieved through an efficient two-stage k-nearest neighbor database searching method proposed in the paper. It is effective for searching from a pre-rendered database of small hand depth images, designed to provide good initial guesses for model based tracking. We also propose a robust objective function, and improve the Particle Swarm Optimization algorithm with a resampling based strategy in model based tracking. It provides continuous solutions in full DOF hand motion space more efficiently than previous methods. Our system runs at 40 fps on a GeForce GTX 580 GPU and experimental results show that the system outperforms the state-of-the-art model based hand tracking systems in terms of both speed and accuracy. The work result is of significance to various applications in the field of human-computer-interaction and virtual reality. [ABSTRACT FROM AUTHOR]

This paper attempts to present a vision-based interface which interacts with computers by hand gesture recognition. This work aims at creating a natural and intuitive application employing both static and dynamic hand gestures. The proposed application can be summarized in three main steps: hands detection in a video, hands tracking and converting hand shapes or trajectories into computer commands. To accomplish this application, a classification phase is paramount whether at the part of hand detection, or at the phase of 'commanding computers'. For this reason, we have proposed to use a wavelet network classifier (WNC) learnt by fast wavelet transform (FWT). To emphasize the robustness of this classifier, we have used a neural network classifier (NNC) version in order to compare the two classifiers' performances aiming at proving the strength of our proposed one. Global rates given by experimental results show the effectiveness of our proposed approaches of hand detection, hand tracking and hand gesture recognition. The comparison of the two classifier's result helps to choose the best classifier, which can improve the performances of our application. [ABSTRACT FROM AUTHOR]

Hand tracking in video is an increasingly popular research field due to the rise of novel human-computer interaction methods. However, robust and real-time hand tracking in unconstrained environments remains a challenging task due to the high number of degrees of freedom and the non-rigid character of the human hand. In this paper, we propose an unsupervised method to automatically learn the context in which a hand is embedded. This context includes the arm and any other object that coherently moves along with the hand. We introduce two novel methods to incorporate this context information into a probabilistic tracking framework, and introduce a simple yet effective solution to estimate the position of the arm. Finally, we show that our method greatly increases robustness against occlusion and cluttered background, without degrading tracking performance if no contextual information is available. The proposed real-time algorithm is shown to outperform the current state-of-the-art by evaluating it on three publicly available video datasets. Furthermore, a novel dataset is created and made publicly available for the research community. [ABSTRACT FROM AUTHOR]

With the increase of innovations in vision-based hand gesture interaction system, new techniques and algorithms are being developed by researchers. However, less attention has been paid on the scope of dismantling hand tracking problems. There is also limited publicly available database developed as benchmark data to standardize the research on hand tracking area. For this purpose, we develop a versatile hand gesture tracking database. This database consists of 60 video sequences containing a total of 15,554 RGB color images. The tracking sequences are captured in different situations ranging from an easy indoor scene to extremely high challenging outdoor scenes. Complete with annotated ground truth data, this database is made available on the web for the sake of assisting other researchers in the related fields to test and evaluate their algorithms based on standard benchmark data. [ABSTRACT FROM AUTHOR]

Human Computer Interaction moves forward in the field of sign language interpretation. Indian Sign Language (ISL) Interpretation system is a good way to help the Indian hearing impaired people to interact with normal people with the help of computer. As compared to other sign languages, ISL interpretation has got less attention by the researcher. In this paper, some historical background, need, scope and concern of ISL are given. Vision based hand gesture recognition system have been discussed as hand plays vital communication mode. Considering earlier reported work, various techniques available for hand tracking, segmentation, feature extraction and classification are listed. Vision based system have challenges over traditional hardware based approach; by efficient use of computer vision and pattern recognition, it is possible to work on such system which will be natural and accepted, in general. [ABSTRACT FROM AUTHOR]

Hand gesture-based interaction provides a natural and powerful means for human-computer interaction. It is also a good interface for human-robot interaction. However, most of the existing proposals are likely to fail when they meet some skin-coloured objects, especially the face region. In this paper, we present a novel hand tracking method which can track the features of the hand based on the obstacle avoidance flocking behaviour model to overcome skin-coloured distractions. It allows features to be split into two groups under severe distractions and merge later. The experiment results show that our method can track the hand in a cluttered background or when passing the face, while the Flocking of Features (FoF) and the Mean Shift Embedded Particle Filter (MSEPF) methods may fail. These results suggest that our method has better performance in comparison with the previous methods. It may therefore be helpful to promote the use of the hand gesture-based human-robot interaction method. [ABSTRACT FROM AUTHOR]

The proposed work is part of a project that aims for the control of a videogame based on hand gesture recognition. This goal implies the restriction of real-time response and unconstrained environments. In this paper we present a real-time algorithm to track and recognise hand gestures for interacting with the videogame. This algorithm is based on three main steps: hand segmentation, hand tracking and gesture recognition from hand features. For the hand segmentation step we use the colour cue due to the characteristic colour values of human skin, its invariant properties and its computational simplicity. To prevent errors from hand segmentation we add a second step, hand tracking. Tracking is performed assuming a constant velocity model and using a pixel labeling approach. From the tracking process we extract several hand features that are fed to a finite state classifier which identifies the hand configuration. The hand can be classified into one of the four gesture classes or one of the four different movement directions. Finally, using the system’s performance evaluation results we show the usability of the algorithm in a video game environment.

Abstract: We propose a real-time model-based 3D hand tracker that combines image regions and the signal from an off-the-shelf 3-axis accelerometer placed on the user''s hand. The visual regions allow the tracker to cope with occlusions, motion blur and background clutter, while the latter aids with the inherent silhouette-pose ambiguities. The accelerometer and tracker are synchronised by casting the calibration problem as one of principal component analysis. Based on the assumption that, often, the number of possible hand configurations is limited by the activity the hand is engaging in, we use a multiclass pose classifier to distinguish between a number of activity dependent articulated hand configurations. We demonstrate the benefits of our method, both qualitatively and quantitatively, on a variety of video sequences and hand configurations and show a proof-of-concept human computer interface based on our system. [Copyright &y& Elsevier]

Hand motion tracking is useful in human-computer interface and many other applications requiring human-machine interactions. In this work, permanent magnets and contactless magnetic sensors are used to track finger motion. A magnet patch is affixed to each fingernail to mark the location and orientation of the fingertip. When fingers move, the combined magnetic fields produced by the magnets at fingertips are recorded by a set of magnetic sensors around a wristband. The recorded data from the sensors are utilized to calculate the magnets' locations and orientations, which can provide estimate of the hand posture based on a geometric hand model. [ABSTRACT FROM PUBLISHER]

Abstract: We present a novel approach for visual tracking of structured behaviour as observed in human–computer interaction. An automatically acquired variable-length Markov model is used to represent the high-level structure and temporal ordering of gestures. Continuous estimation of hand posture is handled by combining the model with annealed particle filtering. The stochastic simulation updates and automatically switches between different model representations of hand posture that correspond to distinct gestures. The implementation executes in real time and demonstrates significant improvement in robustness over comparable methods. We provide a measurement of user performance when our method is applied to a Fitts’ law drag-and-drop task, and an analysis of the effects of latency that it introduces. [Copyright &y& Elsevier]

Abstract: Particle filtering and mean shift (MS) are two successful approaches to visual tracking. Both have their respective strengths and weaknesses. In this paper, we propose to integrate advantages of the two approaches for improved tracking. By incorporating the MS optimization into particle filtering to move particles to local peaks in the likelihood, the proposed mean shift embedded particle filter (MSEPF) improves the sampling efficiency considerably. Our work is conducted in the context of developing a hand control interface for a robotic wheelchair. We realize real-time hand tracking in dynamic environments of the wheelchair using MSEPF. Extensive experimental results demonstrate that MSEPF outperforms the MS tracker and the conventional particle filter in hand tracking. Our approach produces reliable tracking while effectively handling rapid motion and distraction with roughly 85% fewer particles. We also present a simple method for dynamic gesture recognition. The hand control interface based on the proposed algorithms works well in dynamic environments of the wheelchair. [Copyright &y& Elsevier]

This paper presents a gesture recognition system for visualization navigation. Scientists are interested in developing interactive settings for exploring large data sets in an intuitive environment. The input consists of registered 3-D data. A geometric method using Bezier curves is used for the trajectory analysis and classification of gestures. The hand gesture speed is incorporated into the algorithm to enable correct recognition from trajectories with variations in hand speed. The method is robust and reliable: correct hand identification rate is 99.9% (from 1641 frames), modes of hand movements are correct 95.6% of the time, recognition rate (given the right mode) is 97.9%. An application to gesture-controlled visualization of 3D bioinformatics data is also presented. [Copyright &y& Elsevier]