Search

We aim at creating an expressive Embodied Conversational Agent (ECA) and address the problem of synthesizing expressive agent gestures. In our previous work, we have described the gesture selection process. In this paper, we present a computational model of gesture quality. Once a certain gesture has been chosen for execution, how can we modify it to carry a desired expressive content while retaining its original semantics? We characterize bodily expressivity with a small set of dimensions derived from a review of psychology literature. We provide a detailed description of the implementation of these dimensions in our animation system, including our gesture modeling language. We also demonstrate animations with different expressivity settings in our existing ECA system. Finally, we describe two user studies that evaluate the appropriateness of our implementation for each dimension of expressivity as well as the potential of combining these dimensions to create expressive gestures that reflect communicative intent. [ABSTRACT FROM AUTHOR]

Cued Speech is an efficient method that allows orally educated deaf people to perceive a complete oral message through the visual channel. Using this system, speakers can clarify what they say with the complement of hand cues near the face; similar lip shapes are disambiguated by the addition of a manual cue. In this context, Cued Speech represents a unique system that closely links hand movements and speech since it is based on spoken language. In a previous study, we investigated the temporal organization of French Cued Speech production for a single cueing talker. A specific pattern of coordination was found: the hand anticipates the lips and speech sounds. In the present study, we investigated the cueing behavior of three additional professional cueing talkers. The same pattern of hand cues anticipation was found. Results are discussed with respect to inter-subject variability. A general pattern of coordination is proposed. [ABSTRACT FROM AUTHOR]

Rapid method for evaluation of pen-based text input techniques is necessary both for designers and consumers. We present a method that is based on an immediate performance comparison of the gesture making using the graphic templates of typefaces and the pen-based behavioral patterns. The results showed that besides the cognitive difficulty of symbolic gestures, metaphors and mnemonics, first and foremost the graphic feasibility determines handwriting performance of the gesture-based input techniques. [ABSTRACT FROM AUTHOR]

This paper presents an interactive multi-modal system for real-time multi-parametric gestural control of audio processing applications. We claim that this can ease the use / control of different tasks and for this we present the following as a demonstration: (1) A musical application, i.e. the multi-parametric control of digital audio effects, and (2) a scientific application, i.e. the interactive navigation of audifications. In the first application we discuss the use of PCA-based control axes and clustering to obtain dimensionality reduced control variables. In the second application we show how the tightly closed human-computer loop actively supports the detection and discovery of features in data under analysis. [ABSTRACT FROM AUTHOR]

This article aims to present a constrained gestural interface which allows to easily experiment spatial geometry for educational purposes. It consists in a bi-manual gestural language specifically designed in order to simplify user's interaction in geometric constructions. As the inherent complexity of geometry in 3 dimensions is combined with the cognitive difficulty of interacting with virtual environments, we propose to constrain the interaction: hand postures constrain the object type for designation and selection; objects already selected constrain the construction process; the degrees of freedom representation constrain the manipulation of constructed figures; and the deformable ray-casting method constrain the navigation. [ABSTRACT FROM AUTHOR]

This paper introduces AcouMotion as a new hard-/software system for combining human body motion, tangible interfaces and sonification to a closed-loop human computer interface that allows non-visual motor control by using sonification (non-speech auditory displays) as major feedback channel. AcouMotion's main components are (i) a sensor device for measuring motion parameters (ii) a computer simulation to represent the dynamical evolution of a model world, and (iii) a sonification engine which generates an auditory representation of objects and any interactions in the model world. The intended applications of AcouMotion range from new kinds of sport games that can be played without visual displays and therefore may be particularly interesting for people with visual impairment to further applications in data mining, physiotherapy and cognitive research. The first application of AcouMotion presented in this paper is Blindminton, a sport game similar to Badminton which is particularly adapted to the abilities of people with visual impairment. We describe our current system and its state of development, and we present first sound examples for interactive sonification using an early prototype. Finally, we discuss some interesting research directions based on the fact that AcouMotion binds auditory stimuli and body motion, and thus can represent a counterpart to the Eye-tracker device that exploits the binding of visual stimuli and eye-movement in cognitive research. [ABSTRACT FROM AUTHOR]

We present a collaborative approach towards a detailed understanding of the usage of pointing gestures accompanying referring expressions. This effort is undertaken in the context of human-machine interaction integrating empirical studies, theory of grammar and logics, and simulation techniques. In particular, we attempt to measure the precision of the focussed area of a pointing gesture, the so-called pointing cone. The pointing cone serves as a central concept in a formal account of multi-modal integration at the linguistic speech-gesture interface as well as in a computational model of processing multi-modal deictic expressions. [ABSTRACT FROM AUTHOR]

In this paper, we introduce some exploratory ideas and applications involving the gestural control of sonic textures. Three examples of how the gestural control of synthesized textures can be implemented are presented: scratching textures, based on the gesturalized exploration of a visual space; dynamic noise filtering, where gestures influence a virtual slowly moving string used to filter a noise; and breathing textures, where the metaphor of breathing is used in the sound as well as in the gestural control. Lastly, we discuss how to find connexions between appropriate gestures and sonic texture processes, with a view to producing coherent and expressive digital musical instruments. [ABSTRACT FROM AUTHOR]

In this paper we present methods for analysis of the emotional content of human movement. We have studied orchestra conductor's movements that portrayed different emotional states. Using signal processing tools and artificial neural networks we were able to determine the emotional state intended by the conductor. The test set included various musical contexts with different tempos, dynamics and nuances in the data set. Some context changes do not disturb the system while other changes cause severe performance losses. The system demonstrates that for some conductors the intended emotional content of these movements can be detected with our methods. [ABSTRACT FROM AUTHOR]

This work proposes a new way for providing feedback to expressivity in music performance. Starting from studies on the expressivity of music performance we developed a system in which a visual feedback is given to the user using a graphical representation of a human face. The first part of the system, previously developed by researchers at KTH Stockholm and at the University of Uppsala, allows the real-time extraction and analysis of acoustic cues from the music performance. Cues extracted are: sound level, tempo, articulation, attack time, and spectrum energy. From these cues the system provides an high level interpretation of the emotional intention of the performer which will be classified into one basic emotion, such as happiness, sadness, or anger. We have implemented an interface between that system and the embodied conversational agent Greta, developed at the University of Rome "La Sapienza" and "University of Paris 8". We model expressivity of the facial animation of the agent with a set of six dimensions that characterize the manner of behavior execution. In this paper we will first describe a mapping between the acoustic cues and the expressivity dimensions of the face. Then we will show how to determine the facial expression corresponding to the emotional intention resulting from the acoustic analysis, using music sound level and tempo characteristics to control the intensity and the temporal variation of muscular activation. [ABSTRACT FROM AUTHOR]

This paper focuses on the relationship between emotions induced by musical stimuli and movement. A pilot experiment has been realized with the aim to verify whether there are correlations between the emotional characterization of music excerpts and human movement. Subjects were asked to move a laser pointer on a white wall in front of them while listening to musical excerpts classified with respect to the type of emotions they can induce. Trajectories obtained moving the laser pointer have been recorded with a video camera and have been analyzed in a static and global way by using the EyesWeb platform. Results highlight a difference between trajectories associated to music stimuli classified as "fast" and "slow", in term of smoothness/angularity, suggesting the existence of a strong link between the emotional characterization of the musical excerpts listened to and the movement performed. Subfield: expressive gesture and music. Keywords: subject interfaces; emotion; expressive gesture; motor activation. [ABSTRACT FROM AUTHOR]

Both musicians and non-musicians can often be seen making sound-producing gestures in the air without touching any real instruments. Such "air playing" can be regarded as an expression of how people perceive and imagine music, and studying the relationships between these gestures and sound might contribute to our knowledge of how gestures help structure our experience of music. [ABSTRACT FROM AUTHOR]

This paper presents an approach to model the gaze behavior of an Embodied Conversational Agent in a real time multimodal dialogue interaction with a group of users. The ECA's gaze control results from the merge of the outputs of a rational dialogue engine based on natural language interaction and face tracking of users. [ABSTRACT FROM AUTHOR]

This paper presents a virtual reality application dedicated to the ergonomic evaluation and adaptation of workplaces destined for the physically disabled. Handiposte aims at assisting doctors and ergonomists as an interactive simulation based tool. After a general survey about virtual reality tools oriented towards ergonomic studies, we propose a specific framework for the design of such an application using a particular design methodology. We conclude by presenting a first prototype and by outlining future improvements. [ABSTRACT FROM AUTHOR]

This paper aims at describing a new method to simulate the locomotion of hemiplegic subjects. To this end, we propose to use inverse kinematics in order to make the feet follow a trajectory with respect to the root frame linked to the pelvis. The 11 degrees of freedom are then retrieved by inversing the kinematic function while taking other constraints into account. These constraints, termed secondary tasks impose that the solution ensures joints limits and energy minimisation. In addition to those general constraints, the main originality of this work is to take spasticity into account. This new constraint is obtained according to the specificity of the subject's pathology. The results show that angular trajectories for the pelvis, the hips and the knees for the simulated and the real motion are very similar. This preliminary work is promising and could be used to simulate the effects of reeducation or medical treatments on patients' gait. Keywords: inverse kinematics, locomotion, hemiplegia, spasticity. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of synthesizing in real time the motion of realistic virtual characters with a physics-based model from the analysis of human motion data. The synthesis is achieved by computing the motion equations of a dynamical model controlled by a sensory motor feedback loop with a non-parametric learning approach. The analysis is directly applied on end-effector trajectories captured from human motion. We have developed a Dynamic Programming Piecewise Linear Approximation model (DPPLA) that generates the discretization of these 3D Cartesian trajectories. The DPPLA algorithm leads to the identification of discrete target-patterns that constitute an adaptive sampling of the initial end-point trajectory. These sequences of samples non uniformly distributed along the trajectory are used as input of our sensory motor system. The synthesis of motion is illustrated on a dynamical model of a hand-arm system, each arm being represented by seven degrees of freedom. We show that the algorithm works on multi-dimensional variables and reduces the information flow at the command level with a good compression rate, thus providing a technique for motion data indexing and retrieval. Furthermore, the adaptive sampling seems to be correlated with some invariant law of human motion. [ABSTRACT FROM AUTHOR]

The purpose of this study is to propose a new tool to define the posture of a complete upper-limb model during grasping taking into account task and environment constraints. The developed model is based on a neural network architecture mixing both supervised and reinforcement learning. The task constraints are materialized by target points to be reached by the fingertips on the surface of the object to be grasped while environment constraints are represented by obstacles. Without few prior information on the adequate posture, the model is able to find a suitable solution. Simulation results are proposed and commented. This tool can find interesting applications in the frame of gesture definition and simulation. [ABSTRACT FROM AUTHOR]

Simulating realistic human-like figures is still a challenging task when dynamics is involved. For example, making a virtual human jump to a given position requires to control the forces involved in take-off in order to reach a given velocity vector at the beginning of the aerial phase. Several problems are addressed in this paper in order to modify a captured motion while accounting from dynamics. The method exploits a point mass approximation of the body for the Inverse Dynamics stage during the contact phase and later to optimize new trajectories. First, accurate body segment masses are required to have access to external forces thanks to inverse dynamics. Second, those forces have to be adapted to make the resulting center of mass trajectory verify new constraints (such as reaching a given point at a given time). This paper also proposes a new formalism to encode force depending on time in contact phases (called impulse). Whereas classical biomechanical analyzes focus only on the peak of forces and on the contact phase duration, our formalism provides new data to characterize the shape of an impulse. Keywords: take-off, optimization, dynamic control, constraints. [ABSTRACT FROM AUTHOR]

In this paper we explore the potential of Prioritized Inverse Kinematics for motion capture and postural control. We have two goals in mind: reducing the number of sensors to improve the usability of such systems, and allowing interactions with the environment such as manipulating objects or managing collisions on the fly. To do so, we enforce some general constraints such as balance or others that we can infer from the intended movement structure. On one hand we may loose part of the expressiveness of the original movement but this is the price to pay to ensure more precise interactions with the environment. [ABSTRACT FROM AUTHOR]

This paper presents a novel method for estimating 3D human body pose from stereo image sequences based on top-down learning. Human body pose is represented by a linear combination of prototypes of 2D depth images and their corresponding 3D body models in terms of the position of a predetermined set of joints. With a 2D depth image, we can estimate optimal coefficients for a linear combination of prototypes of the 2D depth images by solving least square minimization. The 3D body model of the input depth image is obtained by applying the estimated coefficients to the corresponding 3D body model of prototypes. In the learning stage, the proposed method is hierarchically constructed by classifying the training data into several clusters with a silhouette images and a depth images recursively. Also, in the estimating stage, the proposed method hierarchically estimates 3D human body pose with a silhouette image and a depth image. The experimental results show that our method can be efficient and effective for estimating 3D human body pose. [ABSTRACT FROM AUTHOR]

This study proposes a roadmap for the creation and specification of a virtual humanoid capable of performing expressive gestures in real time. We present a gesture motion data acquisition protocol capable of handling the main articulators involved in human expressive gesture (whole body, fingers and face). The focus is then shifted to the postprocessing of captured data leading to a motion database complying with our motion specification language and capable of feeding data driven animation techniques. [ABSTRACT FROM AUTHOR]

This paper compares different algorithms for tracking the position of fingers in a two-dimensional environment. Four algorithms have been implemented in EyesWeb, developed by DIST-InfoMus laboratory. The three first algorithms use projection signatures, the circular Hough transform, and geometric properties, and rely only on hand characteristics to locate the finger. The fourth algorithm uses color markers and is employed as a reference system for the other three. All the algorithms have been evaluated using two-dimensional video images of a hand performing different finger movements on a flat surface. Results about the accuracy, precision, latency and computer resource usage of the different algorithms are provided. Applications of this research include human-computer interaction systems based on hand gesture, sign language recognition, hand posture recognition, and gestural control of music. [ABSTRACT FROM AUTHOR]

We developed an "augmented violin", i.e. an acoustic instrument with added gesture capture capabilities to control electronic processes. We report here gesture analysis we performed on three different bow strokes, , using this augmented violin. Different features based on velocity and acceleration were considered. A linear discriminant analysis has been performed to estimate a minimum number of pertinent features necessary to model these bow stroke classes. We found that the maximum and minimum accelerations of a given stroke were efficient to parameterize the different bow stroke types, as well as differences in dynamics playing. Recognition rates were estimated using a kNN method with various training sets. We finally discuss that bow stroke recognition allows to relate the gesture data to music notation, while a bow stroke continuous parameterization can be related to continuous sound characteristics. Keywords: Music, Gesture Analysis, Bow Strokes, Violin, Augmented Instruments. [ABSTRACT FROM AUTHOR]

In the last decade speech processing has been applied in commercially available products. One of the key reasons for its success is the identification and use of an underlying set of generic symbols (phonemes) constituting all speech. In this work we follow the same approach, but for the problem of human body gestures. That is, the topic of this paper is how to define a framework for automatically finding primitives for human body gestures. This is done by considering a gesture as a trajectory and then searching for points where the density of the training data is high. The trajectories are re-sampled to enable a direct comparison between the samples of each trajectory, and enable time invariant comparisons. This work demonstrates and tests the primitive's ability to reconstruct sampled trajectories. Promising test results are shown for samples from different test persons performing gestures from a small one armed gesture set. [ABSTRACT FROM AUTHOR]

In this paper we describe O.G.R.E - Open Gestures Recognition Engine, a general purpose real time hand gesture recognition engine based on Computer Vision, able to support gesture-based communication as a modality of Human-Computer Interaction. The engine recognizes essentially, static poses of a single hand and, hand trajectory paths in simple geometrical shapes. [ABSTRACT FROM AUTHOR]

We introduce the use of appearance-based features, and tangent distance or the image distortion model to account for image variability within the hidden Markov model emission probabilities to recognize gestures. No tracking, segmentation of the hand or shape models have to be defined. The distance measures also perform well for template matching classifiers. We obtain promising first results on a new database with the German finger-spelling alphabet. This newly recorded database is freely available for further research. [ABSTRACT FROM AUTHOR]

In modern society there is an increasing demand to access, record and manipulate large amounts of information. This has inspired a new approach to thinking about and designing personal computers, where the ultimate goal is to produce a truly wearable computer. In this work we present a non-invasive hand-gesture recognition system aimed at deictic gestures. Our system is based on the powerful Sequential Monte Carlo framework which is enhanced with respect to increased robustness. This is achieved by using ratios in the likelihood function together with two image cues: edges and skin color. The system proves to be fast, robust towards noise, and quick to lock on to the object (hand). All of which is achieved without the use of special lighting or special markers on the hands, hence our system is a non-invasive solution. [ABSTRACT FROM AUTHOR]

Gestures are expressive and meaningful body motions used in daily life as a means of communication so many researchers have aimed to provide natural ways for human-computer interaction through automatic gesture recognition. However, most of researches on recognition of actions focused mainly on sign gesture. It is difficult to directly extend to recognize whole body gesture. Moreover, previous approaches used manually segmented image sequences. This paper focuses on recognition and segmentation of whole body gestures, such as walking, running, and sitting. We introduce the gesture spotting algorithm that calculates the likelihood threshold of an input pattern and provides a confirmation mechanism for the provisionally matched gesture pattern. In the proposed gesture spotting algorithm, the likelihood of non-gesture Hidden Markov Models(HMM) can be used as an adaptive threshold for selecting proper gestures. The proposed method has been tested with a 3D motion capture data, which are generated with gesture eigen vector and Gaussian random variables for adequate variation. It achieves an average recognition rate of 98.3% with six consecutive gestures which contains non-gestures. [ABSTRACT FROM AUTHOR]

Gesture and motion analysis is a highly needed process in the athletics field. This is especially true for sports dealing with acrobatics, because acrobatics mix complex spatial rotations over multiple axes and may be combined with various postures. This paper presents a new vision-based system focused on the analysis of acrobatic gestures of several sports. Instead of classical systems requiring modelizing human bodies, our system is based on the modelling and characterization of acrobatic movements. To show the robustness of the system, it was successively tested first on movements from trampoline, and also in other sports (gymnastics, diving, etc.). Within the system, the gestures analysis is mainly carried out by using global measurements, extracted from recorded movies or live video. [ABSTRACT FROM AUTHOR]

Hitherto, one major challenge to sign language recognition is how to develop approaches that scale well with increasing vocabulary size. In large vocabulary speech recognition realm, it is effective to use phonemes instead of words as the basic units. This idea can be used in large vocabulary Sign Language recognition, too. In this paper, Etyma are defined to be the smallest unit in a sign language, that is, a unit that has some meaning and distinguishes one sign from the others. They can be seen as phonemes in Sign Language. Two approaches to large vocabulary Chinese Sign Languagerecognition are discussed in this paper. One uses etyma and the other uses whole signs as the basic units. Two CyberGloves and a Pohelmus 3-D tracker with three receivers positioned on the wrist of CyberGlove and the back are used as input device. Etymon- and word- based recognition systems are introduced, which are designed to recognize 2439 etyma and 5100 signs. And then the experimental results of these two systems are given and analyzed. [ABSTRACT FROM AUTHOR]

A sign language recognition system based on Hidden Markov Models(HMMs) and Auto-regressive Hidden Markov Models(ARHMMs) has been proposed in this paper. ARHMMs fully consider the observation relationship and are helpful to discriminate signs which don't have obvious state transitions while similar in motion trajectory. ARHMM which models the observation by mixture conditional linear Gaussian is proposed for sign language recognition. The corresponding training and recognition algorithms for ARHMM are also developed. A hybrid structure to combine ARHMMs with HMMs based on the trick of using an ambiguous word set is presented and the advantages of both models are revealed in such a frame work. Keywords: Computer Vision, Sign Language Recognition, HMM, Auto-regressive HMM. [ABSTRACT FROM AUTHOR]

In this paper, we present a system for automatic sign language recognition of segmented words in American Sign Language (ASL). The system uses appearance-based features extracted directly from the frames captured by standard cameras without any special data acquisition tools. This means that we do not rely on complex preprocessing of the video signal or on an intermediate segmentation step that may produce errors. We introduce a database for ASL word recognition extracted from a publicly available set of video streams. One important property of this database is the large variability of the utterances for each word. To cope with this variability, we propose to model distinct pronunciations of each word using different clustering approaches. Automatic clustering of pronunciations improves the error rate of the system from 28.4% to 23.2%. To model global image transformations, the tangent distance is used within the Gaussian emission densities of the hidden Markov model classifier instead of the Euclidean distance. This approach can further reduce the error rate to 21.5%. [ABSTRACT FROM AUTHOR]

In Sign Language recognition, one of the problems is to collect enough data. Data collection for both training and testing is a laborious but necessary step. Almost all of the statistical methods used in Sign Language Recognition suffer from this problem. Inspired by the crossover and mutation of genetic algorithms, this paper presents a method to enlarge Chinese Sign language database through re-sampling from existing sign samples. Two initial samples of the same sign are regarded as parents. They can reproduce their children by crossover. To verify the effectiveness of the proposed method, some experiments are carried out on a vocabulary with 350 static signs. Each sign has 4 samples. Three samples are used to be the original generation. These three original samples and their offspring are used to construct the training set, and the remaining sample is used for testing. The experimental results show that the data generated by the proposed method are effective. [ABSTRACT FROM AUTHOR]

In this paper, we propose an approach for the representation of the relationship between verbs and actors in French Sign Language. This proposal comes from the results of an experiment conducted in the France Télécom R&D's Natural Language laboratory using TiLT (Linguistic Treatment of Texts), an automatic syntactic analyser and generator. The aim of this approach is to develop a model that will be used to animate an avatar through a previous computational linguistics treatment, respecting French Sign Language as a proper human language. Keywords: French Sign Language, verb typology, morphology, nominal classes, agreement, computational sciences. [ABSTRACT FROM AUTHOR]

In our paper we present a method for the automatic generation of single German Sign Language glosses from German words. Glosses are often used as a textual description of signs when transcribing Sign Language video data. For a machine translation system from German to German Sign Language we apply glosses as an intermediate notational system. Then the automatic generation from given German words is presented. This novel approach takes the orthographic similarities between glosses and written words into account. The obtained experimental results show the feasibility of our methods for word classes like adverbs, adjectives and verbs with up to 80% correctly generated glosses. [ABSTRACT FROM AUTHOR]

This paper belongs to computational linguistics, with reference to French Sign Language (FSL) which is used within French deaf community. Our proposed modelling is intended: to provide computational modelling of a semantico-cognitive formalisation of the FSL linguistic structure, and to allow its implementation and integration in applications dedicated to FSL, such as analysis in image processing, automatic recognition, generation of writing forms or realisation by signing avatars. [ABSTRACT FROM AUTHOR]

Sign language processing is often performed by processing each individual sign. Such an approach relies on an exhaustive description of the signs and does not take in account the spatial structure of the sentence. In this paper, we will present a general model of sign language sentences that uses the construction of the signing space as a representation of both the meaning and the realisation of the sentence. We will propose a computational model of this construction and explain how it can be attached to a sign language grammar model to help both analysis and generation of sign language utterances. [ABSTRACT FROM AUTHOR]

To model and simulate human gesture is a challenge which takes benefit from a close collaboration between scientists from several fields: psychology, physiology, biomechanics, cognitive and computer sciences, etc. As an a priori requirement, we need to better understand the so-called laws of biological motions, established all along the 20th century. When modelled and used to animate artificial creature, these laws makes these creatures (either virtual or robotic) move in a much more realistic, life-like, fashion. [ABSTRACT FROM AUTHOR]