Search

Multimodal interfaces pose the challenge of dealing with the multi-ple interactive time-scales characterizing human behavior. To dothis, innovative models and time-adaptive technologies are needed,operating at multiple time-scales and adopting a multi-layered ap-proach. The first International Workshop on Multi-Scale MovementTechnologies, hosted virtually during the 22nd ACM InternationalConference on Multimodal Interaction, is aimed at providing re-searchers from different areas with the opportunity to discuss thistopic. This paper summarizes the activities of the workshop andthe accepted papers

The work reports ongoing research about a computational method, based on cooperative games on graphs, aimed at detecting the perceived origin of full-body human movement and its propagation. Compared with previous works, a larger set of movement features is considered, and a ground truth is produced, able to assess and compare the effectiveness of each such feature. This is done through the use of the Shapley Value as a centrality index. An Origin of Movement Continuum is also defined, as the basis for creating a repository of movement qualities.

This research is a part of a broader project exploring how movement qualities can be recognized by means of the auditory channel: can we perceive an expressive full-body movement quality by means of its interactive sonification? The paper presents a sonification framework and an experiment to evaluate if embodied sonic training (i.e., experiencing interactive sonification of your own body movements) increases the recognition of such qualities through the auditory channel only, compared to a non-embodied sonic training condition. We focus on the sonification of two mid-level movement qualities: fragility and lightness. We base our sonification models, described in the first part, on the assumption that specific compounds of spectral features of a sound can contribute to the cross-modal perception of a specific movement quality. The experiment, described in the second part, involved 40 participants divided into two groups (embodied sonic training vs. no training). Participants were asked to report the level of lightness and fragility they perceived in 20 audio stimuli generated using the proposed sonification models. Results show that (1) both expressive qualities were correctly recognized from the audio stimuli, (2) a positive effect of embodied sonic training was observed for fragility but not for lightness. The paper is concluded by the description of the artistic performance that took place in 2017 in Genoa (Italy), in which the outcomes of the presented experiment were exploited.

Group coordination passes through an efficient integration of multimodal sources of information. This study examines complex non-verbal communication by recording movement kinematics from conductors and two sections of violinists of an orchestra adapting to a perturbation affecting their normal pattern of sensorimotor communication (rotation of half a turn of the first violinists’ section). We show that different coordination signals are channeled through ancillary (head kinematics) and instrumental movements (bow kinematics). Each one of them affect coordination either at the inter-group or intra-group levels, therefore tapping into different modes of cooperation: complementary versus imitative coordination. Our study suggests that the co-regulation of group behavior is based on the exchange of information across several layers, each one of them tuned to carry specific coordinative signals. Multi-layer sensorimotor communication may be the key musicians and, more generally humans, use to flexibly communicate between each other in interactive sensorimotor tasks.

In a joint action, a group of individuals coordinate their movements to reach a shared goal. When a change - i.e., an event that affects group functioning - occurs, the group adopts strategies to face it. This paper investigates how a change involving a strategic core role in a group affects interpersonal coordination and ultimately group effectiveness in performing a joint action. Following the entrainment theory, interpersonal coordination is addressed in terms of the rhythmic cycles of the individuals and of the group and their adjustment. Music is used as an ideal ecological scenario for investigation. Results show that whereas the change of conductor had a limited significant effect on entrainment, a significant effect was found when entrainment is used as a predictor of the external ratings. Both the obtained results and the techniques developed for measuring entrainment may open novel research directions in the area of automated analysis of group behavior, and particularly of emotion in groups.

We present non-intrusive visual observation and estimation of movement parameters using RGB data for detecting the effect of olfactory stimulation (essential oils) on movement patterns of high school students during the lessons. In particular, we examine the effect of exposure to aromatherapy has on students’ movement kinetics of upper-body: velocity, acceleration, jerk and energy. The Lavender essential oil was used because of antiseptic, antimicrobial, anti-inflammatory and calming properties that may be used for treating anxiety, insomnia and depression [8, 11, 12]. Two classes were studied, as control and experimental group during two days with week of pause in between. First group had both days without aromatherapy, instead the second - two settings without and with aromatherapy for separate days. For post processing of the recorded data we use OpenPose [7] for estimation of position of joints, Matlab for processing positional data and tracking of the subjects, EyesWeb XMI for the extraction of movement features at a small time scale. Data showed significant differences in velocity, acceleration and jerk for left shoulder and elbow joints of experimental group in comparison between aroma and no aroma settings with Mann-Whitney U test at p < .05. In conclusion, this is an ongoing study shows the possibility of using movement qualities, such as kinematic movement features, extracted ecologically using non-invasive equipment, as a method to measure change of movement behavior, in the cases when no other type of data capture is possible. Future studies will involve further experiments and wider collection of movement features with higher level notations as fluidity, smoothness, rigidity of the movements.

This work investigates classification of emotions from MoCap full-body data by using Convolutional Neural Networks (CNN). Rather than addressing regular day to day activities, we focus on a more complex type of full-body movement - dance. For this purpose, a new dataset was created which contains short excerpts of the performances of professional dancers who interpreted four emotional states: anger, happiness, sadness, and insecurity. Fourteen minutes of motion capture data are used to explore different CNN architectures and data representations. The results of the four-class classification task are up to 0.79 (F1 score) on test data of other performances by the same dancers. Hence, through deep learning, this paper proposes a novel and effective method of emotion classification, which can be exploited in affective interfaces.

In this paper we present a pilot study on non-intrusive visual observation and estimation of affective parameter using recorded videos (RGB). We aim at estimating student engagement analyzing upper-body movement comparing two different classroom settings: with the introduction of aromatherapy during the class vs standard lesson. Following previous studies on how aromatherapy can alter movement behaviour, we chose Lavender essential oil. We used computer vision techniques for pose estimation and developed software modules for the extraction of movement features from media data. Data show significant increases in overall velocity and acceleration when the participants are exposed to the aromatherapy condition. Significant decreases in neck flexion angle has been also observed, that shows students had a straighter head posture (i.e. sitting up straighter). No significant differences were observed for the overall kinetic energy of the joints and spinal extension.

Social species spend considerable time observing the body movements of others to understand their actions, predict their emotions, watch their games, or enjoy their dance movements. Given the important information obtained from body movements, we still know surprisingly little about the details of brain mechanisms underlying movement perception. In this fMRI study, we investigated the relations between movement features obtained from automated computational analyses of video clips and the corresponding brain activity. Our results show that low-level computational features map to specific brain areas related to early visual- and motion-sensitive regions, while mid-level computational features are related to dynamic aspects of posture encoded in occipital–temporal cortex, posterior superior temporal sulcus and superior parietal lobe. Furthermore, behavioral features obtained from subjective ratings correlated with activity in higher action observation regions. Our computational feature-based analysis suggests that the neural mechanism of movement encoding is organized in the brain not so much by semantic categories than by feature statistics of the body movements.

We present a method to measure intrapersonal synchronization of movement from motion capture data, and we show that our method is effective in classifying the level of skills of athletes performing karate kata. Our method is based on detecting relevant peaks of acceleration of limbs (arms and legs) and measuring their synchronization. We run a multiscale analysis, based on topological persistence, to rank the importance of peaks of acceleration. The resulting impulse signals are processed next with a multievent class synchronization algorithm, in order to define an overall synchronization index that scores the level of intrapersonal synchronization with a single scalar value. We build a basic multiclass classifier, which uses just the means of indexes computed on the different classes in the training set. We make a statistical analysis and a cross validation of the classifier on real data. Performances by athletes from three levels of skill have been recorded, classified by experts, and used to test our method. Cross validation of the classifier is performed by leave-one-out and bootstrap resampling. Results show that our method can classify correctly with very high probability (beyond $\text{99}\%$ ), while it succeeds on $\text{100}\%$ of the data used in cross validation.

Designing multimodal systems for interactive computer play, as tools for supporting rehabilitation of children with motor and/or cognitive impairment is a complex process, requiring interaction of many stakeholders, including clinicians, therapists, engineers, human factors experts, parents, and children themselves. This paper discusses the participatory design process, leading us to conceive, develop, and iteratively refine a flexible and modular open platform for nonverbal interactive child rehabilitation exergames, with a specific focus on interactive sonification of motor behavior. The process was implemented through a collection of case studies carried out using an early prototype of the platform.

As they are mainly based on bodily experiences and embodied knowledge, dance and movement practices present a great diversity and complexity across genre and context. Thus, developing a conceptual framework for archiving, managing, curating and analysing movement data, in order to develop reusable datasets and algorithms for a variety of purposes, remains a challenge. In this work, based on relevant literature on movement representation and existing systems such as Laban Movement Analysis, as well as working with dance experts through workshops, focus groups, and interviews, we propose a conceptual framework for creating, and analysing dance learning content. The conceptual framework has been developed within an interdisciplinary project, that brings together technology and human-computer interaction researchers, computer science engineers, motion capture experts from industry and academia, as well as dance experts with background on four different dance genres: contemporary, ballet, Greek folk, and flamenco. The framework has been applied: a) as a guidance to systematically create a movement library with multimodal recordings for dance education, including four different dance genres, b) as the basis for developing controlled vocabularies of dance for manual and automated annotation, and c) as the conceptual framework to define the requirements for similarity search and feature extraction.

Analysis of human movement data is a core topic of many research studies in human-human and human-computer interaction. Whereas, on the one side, automated movement analysis is often based on the application of sophisticated computer science techniques (e.g., motion tracking from video recordings), on the other side the interdisciplinary nature of research in this area requires the availability of tools that can be used by researchers who may not have an advanced computer science expertise. This paper presents a system enabling users, who are not necessarily computer scientists, to perform motion tracking from a dataset of video recordings. The system - consisting of a set of (freely downloadable) tools accessible by means of user friendly graphical interfaces - was designed, developed, and tested in the context of a project for automated analysis of entrainment in ensemble music performance, following the needs and requirements of musicologists and psychologists.

This abstract presents a computational model and a software library for the EyesWeb XMI platform to measure a mid-level movement quality of particular importance to convey expressivity: Postural Tension. A whole body posture can be described by a vector containing the angles between the adjacent lines identifying feet (the line connecting the barycentre of each foot), knees, hip, trunk, shoulders, head, and gaze (eyes direction). Postural Tension is the extent at which a movement exhibits rotation of these multiple horizontal planes including spirals. The abstract presents a definition of this mid-level quality, and describe a demonstration: movement of a user is captured with a low-cost wearable device, postural tension and transmission of energy through the body are then extracted, visualized and sonified.

This paper addresses the use of a remote interactive platform to support home-based rehabilitation for children with motor and cognitive impairment. The interaction between user and platform is achieved on customizable full-body interactive serious games (exergames). These exergames perform real-time analysis of multimodal signals to quantify movement qualities and postural attitudes. Interactive sonification of movement is then applied for providing a real-time feedback based on “aesthetic resonance” and engagement of the children. The games also provide log file recordings therapists can use to assess the performance of the children and the effectiveness of the games. The platform allows the customization of the games to address the children's needs. The platform is based on the EyesWeb XMI software, and the games are designed for home usage, based on Kinect for Xbox One and simple sensors including 3-axis accelerometers available in low-cost Android smartphones.

The measurement and tracking of body movement within musical performances can provide valuable sources of data for studying interpersonal interaction and coordination between musicians. The continued development of tools to extract such data from video recordings will offer new opportunities to research musical movement across a diverse range of settings, including field research and other ecological contexts in which the implementation of complex motion capture systems is not feasible or affordable. Such work might also make use of the multitude of video recordings of musical performances that are already available to researchers. The present study made use of such existing data, specifically, three video datasets of ensemble performances from different genres, settings, and instrumentation (a pop piano duo, three jazz duos, and a string quartet). Three different computer vision techniques were applied to these video datasets—frame differencing, optical flow, and kernelized correlation filters (KCF)—with the aim of quantifying and tracking movements of the individual performers. All three computer vision techniques exhibited high correlations with motion capture data collected from the same musical performances, with median correlation (Pearson’s r) values of .75 to .94. The techniques that track movement in two dimensions (optical flow and KCF) provided more accurate measures of movement than a technique that provides a single estimate of overall movement change by frame for each performer (frame differencing). Measurements of performer’s movements were also more accurate when the computer vision techniques were applied to more narrowly-defined regions of interest (head) than when the same techniques were applied to larger regions (entire upper body, above the chest or waist). Some differences in movement tracking accuracy emerged between the three video datasets, which may have been due to instrument-specific motions that resulted in occlusions of the body part of interest (e.g. a violinist’s right hand occluding the head whilst tracking head movement). These results indicate that computer vision techniques can be effective in quantifying body movement from videos of musical performances, while also highlighting constraints that must be dealt with when applying such techniques in ensemble coordination research.

The Routledge Companion to Music, Technology, and Education is a comprehensive resource that draws together burgeoning research on the use of technology in music education around the world. Rather than following a procedural how-to approach, this companion considers technology, musicianship, and pedagogy from a philosophical, theoretical, and empirically-driven perspective, offering an essential overview of current scholarship while providing support for future research. The 37 chapters in this volume consider the major aspects of the use of technology in music education.

We present a method to compute a measure of karate movement quality from MoCap data. We start from well-known common assumptions: an expert athlete is able to perform movements characterized by stable and clean postures and stances, i.e., he is able to conclude the movements without hesitation, noisy small fluctuations or movement ripples. To explore this hypothesis, we collected a dataset of motion capture data of movements of five athletes while performing two different kata's for a total of 22 trials. The athletes have two distinct levels of skill and age: junior brown belt and senior black belt. For each trial, we compute the acceleration of the limbs (arms and legs) and carry out a multi-scale analysis to identify and extract relevant events. Such events correspond to maxima and minima of acceleration intensity (i.e. peaks of high acceleration or deceleration) that occur near the start and the end points of each basic movement segment in a session of kata. Significant events are then selected and an event-synchronisation approach is used to measure the amount of synchrony between the two arms and between the two legs. Results show that expert performers exhibit higher synchronisation with respect to beginners, resulting in more stable and clean movements perceived by observers.

In this paper we present a low-intrusive approach to the detection of expressive full-body movement qualities. We focus on two qualities: Lightness and Fragility and we detect them using the data captured by four wearable devices, two Inertial Movement Units (IMU) and two electromyographs (EMG), placed on the forearms. The work we present in the paper stems from a strict collaboration with expressive movement experts (e.g., contemporary dance choreographers) for defining a vocabulary of basic movement qualities. We recorded 13 dancers performing movements expressing the qualities under investigation. The recordings were next segmented and the perceived level of each quality for each segment was ranked by 5 experts using a 5-points Likert scale. We obtained a dataset of 150 segments of movement expressing Fragility and/or Lightness. In the second part of the paper, we define a set of features on IMU and EMG data and we extract them on the recorded corpus. We finally applied a set of supervised machine learning techniques to classify the segments. The best results for the whole dataset were obtained with a Naive Bayes classifier for Lightness (F-score 0.77), and with a Support Vector Machine classifier for Fragility (F-score 0.77). Our approach can be used in ecological contexts e.g., during artistic performances.

International audience; Laughter is a social signal capable of facilitating interaction in groups of people: it communicates interest, helps to improve creativity, and facilitates sociability. This article focuses on: endowing virtual characters with computational models of laughter synthesis, based on an expressivity-copying paradigm; evaluating how the physically co-presence of the laughing character impacts on the user’s perception of an audio stimulus and mood. We adopt music as a means to stimulate laughter. Results show that the character presence influences the user’s perception of music and mood. Expressivity-copying has an influence on the user’s perception of music, but does not have any significant impact on mood.

The present study investigates expressive non-verbal interaction in the musical context starting from behavioral features extracted at individual and group levels. Four groups of features are defined, which are related to head movement and direction, and may help gaining insight on the expressivity and cohesion of the performance, discriminating between different performance conditions. Then, the features are evaluated both at a global scale and at a local scale. The findings obtained from the analysis of a string quartet recorded in an ecological setting show that using these features alone or in their combination may help in distinguishing between two types of performance: (a) a concert-like condition, where all musicians aim at performing at best, (b) a perturbed one, where the 1 $$\mathrm{st}$$ violinist devises alternative interpretations of the music score without discussing them with the other musicians. In the global data analysis, the discriminative power of the features is investigated through statistical tests. Then, in the local data analysis, a larger amount of data is used to exploit more sophisticated machine learning techniques to select suitable subsets of the features, which are then used to train an SVM classifier to perform binary classification. Interestingly, the features whose discriminative power is evaluated as large (respectively, small) in the global analysis are also evaluated in a similar way in the local analysis. When used together, the 22 features that have been defined in the paper demonstrate to be efficient for classification, leading to a percentage of about 90 % successfully classified examples among the ones not used in the training phase. Similar results are obtained considering only a subset of 15 features.

Concepts and tools from cooperative game theory are exploited to quantify the role played by each member of a team in estimating the position of an observed point of interest. The measure of importance known as “Shapley value” is used to this end. From the theoretical point view, we propose a specific form of the characteristic function for the class of cooperative games under investigation. In the numerical analysis, different configurations of a group of individuals are considered: all individuals looking at a mobile point of interest, one of them replaced with an artificially-generated one who looks exactly toward the point of interest, and directions of the heads replaced with randomly-generated directions. The corresponding experimental outcomes are compared.