Search

An evaluation study of a serious game and a system for the automatic emotion recognition designed for helping autistic children to learn to recognize and express emotions by means of their full-body movement is presented. Three-dimensional motion data of full-body movements are obtained from RGB-D sensors and used to recognize emotions by means of linear SVMs. Ten children diagnosed with High Functioning Autism or Asperger Syndrome were involved in the evaluation phase, consisting of repeated sessions to play a specifically designed serious game. Results from the evaluation study show an increase of tasks accuracy from the beginning to the end of training sessions in the trained group. In particular, while the increase of recognition accuracy was concentrated in the first sessions of the game, the increase for expression accuracy is more gradual throughout all sessions. Moreover, the training seems to produce a transfer effect on facial expression recognition.

We ran the first Affective Movement Recognition (AffectMove) challenge that brings together datasets of affective bodily behaviour across different real-life applications to foster work in this area. Research on automatic detection of naturalistic affective body expressions is still lagging behind detection based on other modalities whereas movement behaviour modelling is a very interesting and very relevant research problem for the affective computing community. The AffectMove challenge aimed to take advantage of existing body movement datasets to address key research problems of automatic recognition of naturalistic and complex affective behaviour from this type of data. Participating teams competed to solve at least one of three tasks based on datasets of different sensors types and real-life problems: multimodal EmoPain dataset for chronic pain physical rehabilitation context, weDraw-1 Movement dataset for maths problem solving settings, and multimodal Unige-Maastricht Dance dataset. To foster work across datasets, we also challenged participants to take advantage of the data across datasets to improve performances and also test the generalization of their approach across different applications.

The work reports ongoing research about a computational method, based on cooperative games on graphs, aimed at detecting the perceived origin of full-body human movement and its propagation. Compared with previous works, a larger set of movement features is considered, and a ground truth is produced, able to assess and compare the effectiveness of each such feature. This is done through the use of the Shapley Value as a centrality index. An Origin of Movement Continuum is also defined, as the basis for creating a repository of movement qualities.

Group coordination passes through an efficient integration of multimodal sources of information. This study examines complex non-verbal communication by recording movement kinematics from conductors and two sections of violinists of an orchestra adapting to a perturbation affecting their normal pattern of sensorimotor communication (rotation of half a turn of the first violinists’ section). We show that different coordination signals are channeled through ancillary (head kinematics) and instrumental movements (bow kinematics). Each one of them affect coordination either at the inter-group or intra-group levels, therefore tapping into different modes of cooperation: complementary versus imitative coordination. Our study suggests that the co-regulation of group behavior is based on the exchange of information across several layers, each one of them tuned to carry specific coordinative signals. Multi-layer sensorimotor communication may be the key musicians and, more generally humans, use to flexibly communicate between each other in interactive sensorimotor tasks.

In a joint action, a group of individuals coordinate their movements to reach a shared goal. When a change - i.e., an event that affects group functioning - occurs, the group adopts strategies to face it. This paper investigates how a change involving a strategic core role in a group affects interpersonal coordination and ultimately group effectiveness in performing a joint action. Following the entrainment theory, interpersonal coordination is addressed in terms of the rhythmic cycles of the individuals and of the group and their adjustment. Music is used as an ideal ecological scenario for investigation. Results show that whereas the change of conductor had a limited significant effect on entrainment, a significant effect was found when entrainment is used as a predictor of the external ratings. Both the obtained results and the techniques developed for measuring entrainment may open novel research directions in the area of automated analysis of group behavior, and particularly of emotion in groups.

This work investigates classification of emotions from MoCap full-body data by using Convolutional Neural Networks (CNN). Rather than addressing regular day to day activities, we focus on a more complex type of full-body movement - dance. For this purpose, a new dataset was created which contains short excerpts of the performances of professional dancers who interpreted four emotional states: anger, happiness, sadness, and insecurity. Fourteen minutes of motion capture data are used to explore different CNN architectures and data representations. The results of the four-class classification task are up to 0.79 (F1 score) on test data of other performances by the same dancers. Hence, through deep learning, this paper proposes a novel and effective method of emotion classification, which can be exploited in affective interfaces.

Social species spend considerable time observing the body movements of others to understand their actions, predict their emotions, watch their games, or enjoy their dance movements. Given the important information obtained from body movements, we still know surprisingly little about the details of brain mechanisms underlying movement perception. In this fMRI study, we investigated the relations between movement features obtained from automated computational analyses of video clips and the corresponding brain activity. Our results show that low-level computational features map to specific brain areas related to early visual- and motion-sensitive regions, while mid-level computational features are related to dynamic aspects of posture encoded in occipital–temporal cortex, posterior superior temporal sulcus and superior parietal lobe. Furthermore, behavioral features obtained from subjective ratings correlated with activity in higher action observation regions. Our computational feature-based analysis suggests that the neural mechanism of movement encoding is organized in the brain not so much by semantic categories than by feature statistics of the body movements.

We present a method to measure intrapersonal synchronization of movement from motion capture data, and we show that our method is effective in classifying the level of skills of athletes performing karate kata. Our method is based on detecting relevant peaks of acceleration of limbs (arms and legs) and measuring their synchronization. We run a multiscale analysis, based on topological persistence, to rank the importance of peaks of acceleration. The resulting impulse signals are processed next with a multievent class synchronization algorithm, in order to define an overall synchronization index that scores the level of intrapersonal synchronization with a single scalar value. We build a basic multiclass classifier, which uses just the means of indexes computed on the different classes in the training set. We make a statistical analysis and a cross validation of the classifier on real data. Performances by athletes from three levels of skill have been recorded, classified by experts, and used to test our method. Cross validation of the classifier is performed by leave-one-out and bootstrap resampling. Results show that our method can classify correctly with very high probability (beyond $\text{99}\%$ ), while it succeeds on $\text{100}\%$ of the data used in cross validation.

This abstract presents a computational model and a software library for the EyesWeb XMI platform to measure a mid-level movement quality of particular importance to convey expressivity: Postural Tension. A whole body posture can be described by a vector containing the angles between the adjacent lines identifying feet (the line connecting the barycentre of each foot), knees, hip, trunk, shoulders, head, and gaze (eyes direction). Postural Tension is the extent at which a movement exhibits rotation of these multiple horizontal planes including spirals. The abstract presents a definition of this mid-level quality, and describe a demonstration: movement of a user is captured with a low-cost wearable device, postural tension and transmission of energy through the body are then extracted, visualized and sonified.

This paper addresses the use of a remote interactive platform to support home-based rehabilitation for children with motor and cognitive impairment. The interaction between user and platform is achieved on customizable full-body interactive serious games (exergames). These exergames perform real-time analysis of multimodal signals to quantify movement qualities and postural attitudes. Interactive sonification of movement is then applied for providing a real-time feedback based on “aesthetic resonance” and engagement of the children. The games also provide log file recordings therapists can use to assess the performance of the children and the effectiveness of the games. The platform allows the customization of the games to address the children's needs. The platform is based on the EyesWeb XMI software, and the games are designed for home usage, based on Kinect for Xbox One and simple sensors including 3-axis accelerometers available in low-cost Android smartphones.

We present a method to compute a measure of karate movement quality from MoCap data. We start from well-known common assumptions: an expert athlete is able to perform movements characterized by stable and clean postures and stances, i.e., he is able to conclude the movements without hesitation, noisy small fluctuations or movement ripples. To explore this hypothesis, we collected a dataset of motion capture data of movements of five athletes while performing two different kata's for a total of 22 trials. The athletes have two distinct levels of skill and age: junior brown belt and senior black belt. For each trial, we compute the acceleration of the limbs (arms and legs) and carry out a multi-scale analysis to identify and extract relevant events. Such events correspond to maxima and minima of acceleration intensity (i.e. peaks of high acceleration or deceleration) that occur near the start and the end points of each basic movement segment in a session of kata. Significant events are then selected and an event-synchronisation approach is used to measure the amount of synchrony between the two arms and between the two legs. Results show that expert performers exhibit higher synchronisation with respect to beginners, resulting in more stable and clean movements perceived by observers.

A novel computational method for the analysis of expressive full-body movement qualities is introduced, which exploits concepts and tools from graph theory and game theory. The human skeletal structure is modeled as an undirected graph, where the joints are the vertices and the edge set contains both physical and non-physical links. Physical links correspond to connections between adjacent physical body joints (e.g., the forearm, which connects the elbow to the wrist). Nonphysical links act as "bridges" between parts of the body not directly connected by the skeletal structure, but sharing very similar feature values. The edge weights depend on features obtained by using Motion Capture data. Then, a mathematical game is constructed over the graph structure, where the vertices represent the players and the edges represent communication channels between them. Hence, the body movement is modeled in terms of a game built on the graph structure. Since the vertices and the edges contribute to the overall quality of the movement, the adopted game-theoretical model is of cooperative nature. A game-theoretical concept, called Shapley value, is exploited as a centrality index to estimate the contribution of each vertex to a shared goal (e.g., to the way a particular movement quality is transferred among the vertices). The proposed method is applied to a data set of Motion Capture data of subjects performing expressive movements, recorded in the framework of the H2020-ICT-2015 EU Project WhoLoDance, Project no. 688865. Preliminary results are presented.

In this paper we present a low-intrusive approach to the detection of expressive full-body movement qualities. We focus on two qualities: Lightness and Fragility and we detect them using the data captured by four wearable devices, two Inertial Movement Units (IMU) and two electromyographs (EMG), placed on the forearms. The work we present in the paper stems from a strict collaboration with expressive movement experts (e.g., contemporary dance choreographers) for defining a vocabulary of basic movement qualities. We recorded 13 dancers performing movements expressing the qualities under investigation. The recordings were next segmented and the perceived level of each quality for each segment was ranked by 5 experts using a 5-points Likert scale. We obtained a dataset of 150 segments of movement expressing Fragility and/or Lightness. In the second part of the paper, we define a set of features on IMU and EMG data and we extract them on the recorded corpus. We finally applied a set of supervised machine learning techniques to classify the segments. The best results for the whole dataset were obtained with a Naive Bayes classifier for Lightness (F-score 0.77), and with a Support Vector Machine classifier for Fragility (F-score 0.77). Our approach can be used in ecological contexts e.g., during artistic performances.

International audience; Laughter is a social signal capable of facilitating interaction in groups of people: it communicates interest, helps to improve creativity, and facilitates sociability. This article focuses on: endowing virtual characters with computational models of laughter synthesis, based on an expressivity-copying paradigm; evaluating how the physically co-presence of the laughing character impacts on the user’s perception of an audio stimulus and mood. We adopt music as a means to stimulate laughter. Results show that the character presence influences the user’s perception of music and mood. Expressivity-copying has an influence on the user’s perception of music, but does not have any significant impact on mood.

In this paper, we present (i) a computational model of Dynamic Symmetry of human movement, and (ii) a system to teach this movement quality (symmetry or asymmetry) by means of an interactive sonification exergame based on IMU sensors and the EyesWeb XMI software platform. The implemented system is available as a demo at the workshop.

In this paper, we investigate the detection of laughter from the user's nonverbal full-body movement in social and ecological contexts. Eight hundred and one laughter and nonlaughter segments of full-body movement were examined from a corpus of motion capture data of subjects participating in social activities that stimulated laughter. A set of 13 full-body movement features was identified, and corresponding automated extraction algorithms were developed. These features were extracted from the laughter and nonlaughter segments, and the resulting dataset was provided as input to supervised machine learning techniques. Both discriminative (radial basis function-support vector machines, k-nearest neighbor, and random forest) and probabilistic (naive Bayes and logistic regression) classifiers were trained and evaluated. A comparison of automated classification with the ratings of human observers for the same laughter and nonlaughter segments showed that the performance of our approach for automated laughter detection is comparable with that of humans. The highest F-score (0.74) was obtained by the random forest classifier, whereas the F-score obtained by human observers was 0.70. Based on the analysis techniques introduced in the paper, a vision-based system prototype for automated laughter detection was designed and evaluated. Support vector machines (SVMs) and Kohonen's self-organizing maps were used for training, and the highest F-score was obtained with SVM (0.73).

In this paper we propose a multimodal approach to distinguish between movements displaying three different expressive qualities: fluid, fragmented, and impulsive movements. Our approach is based on the Event Synchronization algorithm, which is applied to compute the amount of synchronization between two low-level features extracted from multimodal data. In more details, we use the energy of the audio respiration signal captured by a standard microphone placed near to the mouth, and the whole body kinetic energy estimated from motion capture data. The method was evaluated on 90 movement segments performed by 5 dancers. Results show that fragmented movements display higher average synchronization than fluid and impulsive movements.

Music ensembles are an ideal test-bed for quantitative analysis of social interaction. Music is an inherently social activity, and music ensembles offer a broad variety of scenarios which are particularly suitable for investigation. Small ensembles, such as string quartets, are deemed a significant example of self-managed teams, where all musicians contribute equally to a task. In bigger ensembles, such as orchestras, the relationship between a leader (the conductor) and a group of followers (the musicians) clearly emerges. This paper presents an overview of recent research on social interaction in music ensembles with a particular focus on (i) studies from cognitive neuroscience; and (ii) studies adopting a computational approach for carrying out automatic quantitative analysis of ensemble music performances.

International audience; In this paper, we study perception of intensity incongruence between auditory and visual modalities of synthesized expressions of laughter. In particular, we investigate whether incongruent expressions are perceived as 1) regulated, and 2) unsuccessful in terms of animation synthesis. For this purpose, we conducted a perceptive study with the use of a virtual agent. Congruent and incongruent multimodal expressions of laughter were synthesized from natural audiovisual laughter episodes, using machine learning algorithms. Next, the intensity of facial expressions and body movements were systematically manipulated to check whether the resulting incongruent expressions are perceived differently compared to the corresponding congruent expressions. Results show that 1) intensity incongruence lowers the perception of believability and plausibility, and 2) the in-congruent laughter expressions displaying high intensity in the audio modality and low intensity in the body movement and facial expression are perceived as more fake than the corresponding congruent expressions. Such results have implications for both animation synthesis as well as expression regulation research.

Non-verbal group dynamics are often opaque to a formal quantitative analysis of communication flow. In this context, ensemble musicians can be a reliable model of expert group coordination. In fact, bodily motion is a critical component of inter-musician coordination and thus could be used as a valuable index of sensorimotor communication. Here we measured head movement kinematics of an expert quartet of musicians and, by applying Granger Causality analysis, we numerically described the causality patterns between participants. We found a clear positive relationship between the amount of communication and complexity of the score segment. Furthermore, we also applied temporal and dynamical changes to the musical score, known by the first violin only. The perturbations were devised in order to force unidirectional communication between the leader of the quartet and the other participants. Results show that in these situations, unidirectional influence from the leader decreased, thus implying that effective leadership may require prior sharing of information between participants. In conclusion, we could measure the amount of information flow and sensorimotor group dynamics suggesting that the fabric of leadership is not built upon exclusive information knowledge but rather on sharing it.

This is a short description of the Emotional Charades serious game demo. Our goal is to focus on emotion expression through body gestures, making the players aware of the amount of affective information their bodies convey. The whole framework aims at helping children with autism to understand and express emotions. We also want to compare the performances of our automatic recognition system and the ones achieved by humans.

Despite its relevance for human-human communication, laughter has been quite under-investigated and under-exploited in human-machine interaction. Nevertheless, endowing machines with the capability of analyzing laughter (i.e., to detect when the user is laughing, to measure intensity of laughter, to distinguish between different laughter styles and types) in ecological contexts is a very challenging task. An approach to laughter recognition consisting of the real-time analysis of a single communication modality, i.e., body, is presented in this paper and positive results of an evaluation study are discussed.

When people perform a task as part of a joint action, their behavior is not the same as it would be if they were performing the same task alone, since it has to be adapted to facilitate shared understanding (or sometimes to prevent it). Joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates joint action. Here we compare the expressive movement of violinists when playing in solo and ensemble conditions. The first violinists of two string quartets (SQs), professional and student, were asked to play the same musical fragments in a solo condition and with the quartet. Synchronized multimodal recordings were created from the performances, using a specially developed software platform. Different patterns of head movement were observed. By quantifying them using an appropriate measure of entropy, we showed that head movements are more predictable in the quartet scenario. Rater evaluations showed that the change does not, as might be assumed, entail markedly reduced expression. They showed some ability to discriminate between solo and ensemble performances, but did not distinguish them in terms of emotional content or expressiveness. The data raise provocative questions about joint action in realistically complex scenarios.

This article presents an Interactive Multimodal Environment (IME), the Stanza Logo-Motoria, designed to support learning in primary schools. In particular we describe the use of this system as a tool (a) to practice listening to English as a Second Language (ESL) and (b) to enable children with severe disabilities to perform an interactive listening. We document the ongoing experimentation of the Stanza Logo-Motoria in ESL lessons and report its encouraging results. Moreover, we explain how it may be possible, by means of the Stanza Logo-Motoria, to redesign traditional learning environments in order to allow pupils to experience listening as an active and engaging experience.

Preliminary results from a study of expressivity and of non-verbal social signals in small groups of users are presented. Music is selected as experimental test-bed since it is a clear example of interactive and social activity, where affective non-verbal communication plays a fundamental role. In this experiment the orchestra is adopted as a social group characterized by a clear leader (the conductor) of two groups of musicians (the first and second violin sections). It is shown how a reduced set of simple movement features - heads movements - can be sufficient to explain the difference in the behavior of the first violin section between two performance conditions, characterized by different eye contact between the two violin sections and between the first section and the conductor.

When people perform a task as part of a joint action, their behavior is not the same as it would be if they were performing the same task alone: it is adapted to facilitate shared understanding (or sometimes to prevent it). Joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates joint action. Here we compare the expressive of violinists when playing solo Vs. in the string quartet music ensemble. The first and second violinists of a famous concert string quartet were asked to play the same musical fragment in a solo condition and with the quartet. Synchronized multimodal recordings have been created from the performances, using a specially developed software platform. The differences are not obvious to untrained observers but they are discriminated by musicians, and appropriate measures show that they exist. In particular, using an appropriate measure of entropy shows that head movements are more predictable in the quartet scenario. The change does not, as might be assumed, entail markedly reduced expression. The data pose provocative questions about joint action in realistically complex scenarios.

Within the EU ILHAIRE Project, researchers of several disciplines (e.g., computer sciences, psychology) collaborate to investigate the psychological foundations of laughter, and to bring this knowledge into shape for the use in new technologies (i.e., affective computing). Within this framework, in order to endow machines with laughter capabilities (encoding as well as decoding), one crucial task is an adequate description of laughter in terms of morphology. In this paper we present a work methodology towards automated full body laughter detection: starting from expert annotations of laughter videos we aim to identify the body features that characterize laughter.

How does the individual behavior of a musician change in solo Vs. creative joint action? In this paper we consider music performance, an ideal ecological test bed to investigate non-verbal social behavior, to compare the expressive movement of violinists when playing solo or in a string quartet ensemble. In the presented study, by measuring its Sample Entropy, we observe that the movement of a musician’s head in creative joint action is more regular with respect to the solo condition.

Our understanding of the perceived actions of those around us includes an ability to segment this continuous stream of activity into discrete events. We studied naïve observers' abilities to segment a video of an unfamiliar dance style into events using a combination of behavioural, computational vision and brain imaging methods. A 386 s video of a solo Bharatanatyam dancer was used as the basis for the study. A computational analysis provided us with, for every video frame, a Motion Index (MI) quantifying the movement of the entire dancer. A behavioural analysis using 30 naïve observers provided us with the time points where observers were most likely to place an event boundary. These behavioural and computational data were used to interpret the brain activity of another 11 participants who viewed the dance video while in an MRI scanner. Results showed that the Motion Index predicted brain activity in a single cluster in the right hemisphere that was located close to the Extrastriate Body Area (EBA). Event boundaries in the video were related to extensive clusters of bilateral activity in the Inferior Occipital Gyrus which extended towards the posterior Superior Temporal Sulcus (pSTS). Event boundaries also activated a region in the right Inferior Frontal Gyrus. These results extend our understanding of how movement kinaesthetics modulate action interpretation.

International audience; In this paper, we report on work around the I-SEARCH EU (FP7 ICT STREP) project whose objective is the development of a multimodal search engine. We present the project's objectives, and detail the achieved results, amongswhich a Rich Unified Content Description format.

This paper evaluates three different interactive sonifications of dyadic coordinated human rhythmic activity. An index of phase synchronisation of gestures was chosen as coordination metric. The sonifications are implemented as three prototype applications exploiting mobile devices: Sync’n’Moog, Sync’n’Move, and Sync’n’Mood. Sync’n’Moog sonifies the phase synchronisation index by acting directly on the audio signal and applying a nonlinear time-varying filtering technique. Sync’n’Move intervenes on the multi-track music content by making the single instruments emerge and hide. Sync’n’Mood manipulates the affective features of the music performance. The three sonifications were also tested against a condition without sonification. QC 20150623 SAME