Search

Human communication entails subtle non-verbal modes of expression, which can be analyzed quantitatively using computational approaches and thus support human sciences. In this paper we present huSync, a computational framework and system that utilizes trajectory information extracted using pose estimation algorithms from video sequences to quantify synchronization between individuals in small groups. The system is exploited to study interpersonal coordination in musical ensembles. Musicians communicate with each other through sounds and gestures, providing nonverbal cues that regulate interpersonal coordination. huSync was applied to recordings of concert performances by a professional instrumental ensemble playing two musical pieces. We examined effects of different aspects of musical structure (texture and phrase position) on interpersonal synchronization, which was quantified by computing phase locking values of head motion for all possible within-group pairs. Results indicate that interpersonal coupling was stronger for polyphonic textures (ambiguous leadership) than homophonic textures (clear melodic leader), and this difference was greater in early portions of phrases than endings (where coordination demands are highest). Results were cross-validated against an analysis of audio features, showing links between phase locking values and event density. This research produced a system, huSync, that can quantify synchronization in small groups and is sensitive to dynamic modulations of interpersonal coupling related to ambiguity in leadership and coordination demands, in standard video recordings of naturalistic human group interaction. huSync enabled a better understanding of the relationship between interpersonal coupling and musical structure, thus enhancing collaborations between human and computer scientists.

The objectives of this study were to investigate the feasibility of applying computer vision techniques and to analyse changes in behaviour and movement of high school students during class. The study is performed over two phases. Phase one focuses on developing a feasible method to use computer vision-based techniques in high school classes and phase two focuses on the testing of aromatherapy to affect student’s movement. All camera data was processed and analysed by OpenPose, Matlab and EyesWeb. Movement features such as velocity, acceleration, and kinetic energy and postural variables, spinal extension and neck flexion were calculated. Results of phase one, shows significant differences in the overall segment velocity, acceleration, energy, and neck flexion. Similarly, the second phase shows significant differences in velocity, acceleration and jerk for the left shoulder and elbow joints of the group exposed to aroma. In conclusion, the results show the feasibility of using computer vision techniques to apply in a classroom setting.

Peer reviewed: True, Acknowledgements: We acknowledge the musicians from the Quartetto di Cremona. We thank the reviewers and the editor for their advice and encouragement to improve the manuscript. Scanning resources were provided by the Centre for Cognitive Neuroimaging, University of Glasgow., We use functional Magnetic Resonance Imaging (fMRI) to explore synchronized neural responses between observers of audiovisual presentation of a string quartet performance during free viewing. Audio presentation was accompanied by visual presentation of the string quartet as stick figures observed from a static viewpoint. Brain data from 18 musical novices were obtained during audiovisual presentation of a 116 s performance of the allegro of String Quartet, No. 14 in D minor by Schubert played by the 'Quartetto di Cremona.' These data were analyzed using intersubject correlation (ISC). Results showed extensive ISC in auditory and visual areas as well as parietal cortex, frontal cortex and subcortical areas including the medial geniculate and basal ganglia (putamen). These results from a single fixed viewpoint of multiple musicians are greater than previous reports of ISC from unstructured group activity but are broadly consistent with related research that used ISC to explore listening to music or watching solo dance. A feature analysis examining the relationship between brain activity and physical features of the auditory and visual signals yielded findings of a large proportion of activity related to auditory and visual processing, particularly in the superior temporal gyrus (STG) as well as midbrain areas. Motor areas were also involved, potentially as a result of watching motion from the stick figure display of musicians in the string quartet. These results reveal involvement of areas such as the putamen in processing complex musical performance and highlight the potential of using brief naturalistic stimuli to localize distinct brain areas and elucidate potential mechanisms underlying multisensory integration.

Full-body human movement is characterized by fine-grain expressive qualities that humans are easily capable of exhibiting and recognizing in others’ movement. In sports (e.g., martial arts) and performing arts (e.g., dance), the same sequence of movements can be performed in a wide range of ways characterized by different qualities, often in terms of subtle (spatial and temporal) perturbations of the movement. Even a non-expert observer can distinguish between a top-level and average performance by a dancer or martial artist. The difference is not in the performed movements--the same in both cases--but in the “quality” of their performance. In this article, we present a computational framework aimed at an automated approximate measure of movement quality in full-body physical activities. Starting from motion capture data, the framework computes low-level (e.g., a limb velocity) and high-level (e.g., synchronization between different limbs) movement features. Then, this vector of features is integrated to compute a value aimed at providing a quantitative assessment of movement quality approximating the evaluation that an external expert observer would give of the same sequence of movements. Next, a system representing a concrete implementation of the framework is proposed. Karate is adopted as a testbed. We selected two different katas (i.e., detailed choreographies of movements in karate) characterized by different overall attitudes and expressions (aggressiveness, meditation), and we asked seven athletes, having various levels of experience and age, to perform them. Motion capture data were collected from the performances and were analyzed with the system. The results of the automated analysis were compared with the scores given by 14 karate experts who rated the same performances. Results show that the movement-quality scores computed by the system and the ratings given by the human observers are highly correlated (Pearson’s correlations r = 0.84, p = 0.001 and r = 0.75, p = 0.005).

Our daily human life is filled with a myriad of joint action moments, be it children playing, adults working together (i.e., team sports), or strangers navigating through a crowd. Joint action brings individuals (and embodiment of their emotions) together, in space and in time. Yet little is known about how individual emotions propagate through embodied presence in a group, and how joint action changes individual emotion. In fact, the multi-agent component is largely missing from neuroscience-based approaches to emotion, and reversely joint action research has not found a way yet to include emotion as one of the key parameters to model socio-motor interaction. In this review, we first identify the gap and then stockpile evidence showing strong entanglement between emotion and acting together from various branches of sciences. We propose an integrative approach to bridge the gap, highlight five research avenues to do so in behavioral neuroscience and digital sciences, and address some of the key challenges in the area faced by modern societies., Pages 44, Figures 6, Table 1, Article in press, Neuroscience and Biobehavioral Reviews

Multimodal interfaces pose the challenge of dealing with the multi-ple interactive time-scales characterizing human behavior. To dothis, innovative models and time-adaptive technologies are needed,operating at multiple time-scales and adopting a multi-layered ap-proach. The first International Workshop on Multi-Scale MovementTechnologies, hosted virtually during the 22nd ACM InternationalConference on Multimodal Interaction, is aimed at providing re-searchers from different areas with the opportunity to discuss thistopic. This paper summarizes the activities of the workshop andthe accepted papers

In this paper, we explore how the audio respiration signal can contribute to multimodal analysis of movement qualities. Within this aim, we propose two novel techniques which use the audio respiration signal captured by a standard microphone placed near to mouth and supervised machine learning algorithms. The first approach consists of the classification of a set of acoustic features extracted from exhalations of a person performing fluid or fragmented movements. In the second approach, the intrapersonal synchronization between the respiration and kinetic energy of body movements is used to distinguish the same qualities. First, the value of synchronization between modalities is computed using the Event Synchronization algorithm. Next, a set of features, computed from the value of synchronization, is used as an input to machine learning algorithms. Both approaches were applied to the multimodal corpus composed of short performances by three professionals performing fluid and fragmented movements. The total duration of the corpus is about 17 min. The highest F-score (0.87) for the first approach was obtained for the binary classification task using Support Vector Machines (SVM-LP). The best result for the same task using the second approach was obtained using Naive Bayes algorithm (F-score of 0.72). The results confirm that it is possible to infer information about the movement qualities from respiration audio.

This research is a part of a broader project exploring how movement qualities can be recognized by means of the auditory channel: can we perceive an expressive full-body movement quality by means of its interactive sonification? The paper presents a sonification framework and an experiment to evaluate if embodied sonic training (i.e., experiencing interactive sonification of your own body movements) increases the recognition of such qualities through the auditory channel only, compared to a non-embodied sonic training condition. We focus on the sonification of two mid-level movement qualities: fragility and lightness. We base our sonification models, described in the first part, on the assumption that specific compounds of spectral features of a sound can contribute to the cross-modal perception of a specific movement quality. The experiment, described in the second part, involved 40 participants divided into two groups (embodied sonic training vs. no training). Participants were asked to report the level of lightness and fragility they perceived in 20 audio stimuli generated using the proposed sonification models. Results show that (1) both expressive qualities were correctly recognized from the audio stimuli, (2) a positive effect of embodied sonic training was observed for fragility but not for lightness. The paper is concluded by the description of the artistic performance that took place in 2017 in Genoa (Italy), in which the outcomes of the presented experiment were exploited.

This paper introduces the EU FET PROACTIVE 4-year project EnTimeMent. EnTimeMent aims at a radical change in scientific research and enabling technologies for human movement qualitative analysis, entrainment and prediction, based on a novel neuro-cognitive approach of the multiple, mutually interactive time scales characterizing human behaviour. The main technological breakthrough of EnTimeMent will be promoting novel perspectives on understanding, measuring and predicting the qualities of movement (at individual and group level) in motion capture, multisensory interfaces, wearables, affective and IoT technologies.

We present non-intrusive visual observation and estimation of movement parameters using RGB data for detecting the effect of olfactory stimulation (essential oils) on movement patterns of high school students during the lessons. In particular, we examine the effect of exposure to aromatherapy has on students’ movement kinetics of upper-body: velocity, acceleration, jerk and energy. The Lavender essential oil was used because of antiseptic, antimicrobial, anti-inflammatory and calming properties that may be used for treating anxiety, insomnia and depression [8, 11, 12]. Two classes were studied, as control and experimental group during two days with week of pause in between. First group had both days without aromatherapy, instead the second - two settings without and with aromatherapy for separate days. For post processing of the recorded data we use OpenPose [7] for estimation of position of joints, Matlab for processing positional data and tracking of the subjects, EyesWeb XMI for the extraction of movement features at a small time scale. Data showed significant differences in velocity, acceleration and jerk for left shoulder and elbow joints of experimental group in comparison between aroma and no aroma settings with Mann-Whitney U test at p < .05. In conclusion, this is an ongoing study shows the possibility of using movement qualities, such as kinematic movement features, extracted ecologically using non-invasive equipment, as a method to measure change of movement behavior, in the cases when no other type of data capture is possible. Future studies will involve further experiments and wider collection of movement features with higher level notations as fluidity, smoothness, rigidity of the movements.

In this paper we present a pilot study on non-intrusive visual observation and estimation of affective parameter using recorded videos (RGB). We aim at estimating student engagement analyzing upper-body movement comparing two different classroom settings: with the introduction of aromatherapy during the class vs standard lesson. Following previous studies on how aromatherapy can alter movement behaviour, we chose Lavender essential oil. We used computer vision techniques for pose estimation and developed software modules for the extraction of movement features from media data. Data show significant increases in overall velocity and acceleration when the participants are exposed to the aromatherapy condition. Significant decreases in neck flexion angle has been also observed, that shows students had a straighter head posture (i.e. sitting up straighter). No significant differences were observed for the overall kinetic energy of the joints and spinal extension.

We present a computational model and a system for the automated recognition of emotions starting from full-body movement. Three-dimensional motion data of full-body movements are obtained either from professional optical motion-capture systems (Qualisys) or from low-cost RGB-D sensors (Kinect and Kinect2). A number of features are then automatically extracted at different levels, from kinematics of a single joint to more global expressive features inspired by psychology and humanistic theories (e.g., contraction index, fluidity, and impulsiveness). An abstraction layer based on dictionary learning further processes these movement features to increase the model generality and to deal with intraclass variability, noise, and incomplete information characterizing emotion expression in human movement. The resulting feature vector is the input for a classifier performing real-time automatic emotion recognition based on linear support vector machines. The recognition performance of the proposed model is presented and discussed, including the tradeoff between precision of the tracking measures (we compare the Kinect RGB-D sensor and the Qualisys motion-capture system) versus dimension of the training dataset. The resulting model and system have been successfully applied in the development of serious games for helping autistic children learn to recognize and express emotions by means of their full-body movement.

Designing multimodal systems for interactive computer play, as tools for supporting rehabilitation of children with motor and/or cognitive impairment is a complex process, requiring interaction of many stakeholders, including clinicians, therapists, engineers, human factors experts, parents, and children themselves. This paper discusses the participatory design process, leading us to conceive, develop, and iteratively refine a flexible and modular open platform for nonverbal interactive child rehabilitation exergames, with a specific focus on interactive sonification of motor behavior. The process was implemented through a collection of case studies carried out using an early prototype of the platform.

As they are mainly based on bodily experiences and embodied knowledge, dance and movement practices present a great diversity and complexity across genre and context. Thus, developing a conceptual framework for archiving, managing, curating and analysing movement data, in order to develop reusable datasets and algorithms for a variety of purposes, remains a challenge. In this work, based on relevant literature on movement representation and existing systems such as Laban Movement Analysis, as well as working with dance experts through workshops, focus groups, and interviews, we propose a conceptual framework for creating, and analysing dance learning content. The conceptual framework has been developed within an interdisciplinary project, that brings together technology and human-computer interaction researchers, computer science engineers, motion capture experts from industry and academia, as well as dance experts with background on four different dance genres: contemporary, ballet, Greek folk, and flamenco. The framework has been applied: a) as a guidance to systematically create a movement library with multimodal recordings for dance education, including four different dance genres, b) as the basis for developing controlled vocabularies of dance for manual and automated annotation, and c) as the conceptual framework to define the requirements for similarity search and feature extraction.

Analysis of human movement data is a core topic of many research studies in human-human and human-computer interaction. Whereas, on the one side, automated movement analysis is often based on the application of sophisticated computer science techniques (e.g., motion tracking from video recordings), on the other side the interdisciplinary nature of research in this area requires the availability of tools that can be used by researchers who may not have an advanced computer science expertise. This paper presents a system enabling users, who are not necessarily computer scientists, to perform motion tracking from a dataset of video recordings. The system - consisting of a set of (freely downloadable) tools accessible by means of user friendly graphical interfaces - was designed, developed, and tested in the context of a project for automated analysis of entrainment in ensemble music performance, following the needs and requirements of musicologists and psychologists.

The measurement and tracking of body movement within musical performances can provide valuable sources of data for studying interpersonal interaction and coordination between musicians. The continued development of tools to extract such data from video recordings will offer new opportunities to research musical movement across a diverse range of settings, including field research and other ecological contexts in which the implementation of complex motion capture systems is not feasible or affordable. Such work might also make use of the multitude of video recordings of musical performances that are already available to researchers. The present study made use of such existing data, specifically, three video datasets of ensemble performances from different genres, settings, and instrumentation (a pop piano duo, three jazz duos, and a string quartet). Three different computer vision techniques were applied to these video datasets—frame differencing, optical flow, and kernelized correlation filters (KCF)—with the aim of quantifying and tracking movements of the individual performers. All three computer vision techniques exhibited high correlations with motion capture data collected from the same musical performances, with median correlation (Pearson’s r) values of .75 to .94. The techniques that track movement in two dimensions (optical flow and KCF) provided more accurate measures of movement than a technique that provides a single estimate of overall movement change by frame for each performer (frame differencing). Measurements of performer’s movements were also more accurate when the computer vision techniques were applied to more narrowly-defined regions of interest (head) than when the same techniques were applied to larger regions (entire upper body, above the chest or waist). Some differences in movement tracking accuracy emerged between the three video datasets, which may have been due to instrument-specific motions that resulted in occlusions of the body part of interest (e.g. a violinist’s right hand occluding the head whilst tracking head movement). These results indicate that computer vision techniques can be effective in quantifying body movement from videos of musical performances, while also highlighting constraints that must be dealt with when applying such techniques in ensemble coordination research.

The Routledge Companion to Music, Technology, and Education is a comprehensive resource that draws together burgeoning research on the use of technology in music education around the world. Rather than following a procedural how-to approach, this companion considers technology, musicianship, and pedagogy from a philosophical, theoretical, and empirically-driven perspective, offering an essential overview of current scholarship while providing support for future research. The 37 chapters in this volume consider the major aspects of the use of technology in music education.

The present study investigates expressive non-verbal interaction in the musical context starting from behavioral features extracted at individual and group levels. Four groups of features are defined, which are related to head movement and direction, and may help gaining insight on the expressivity and cohesion of the performance, discriminating between different performance conditions. Then, the features are evaluated both at a global scale and at a local scale. The findings obtained from the analysis of a string quartet recorded in an ecological setting show that using these features alone or in their combination may help in distinguishing between two types of performance: (a) a concert-like condition, where all musicians aim at performing at best, (b) a perturbed one, where the 1 $$\mathrm{st}$$ violinist devises alternative interpretations of the music score without discussing them with the other musicians. In the global data analysis, the discriminative power of the features is investigated through statistical tests. Then, in the local data analysis, a larger amount of data is used to exploit more sophisticated machine learning techniques to select suitable subsets of the features, which are then used to train an SVM classifier to perform binary classification. Interestingly, the features whose discriminative power is evaluated as large (respectively, small) in the global analysis are also evaluated in a similar way in the local analysis. When used together, the 22 features that have been defined in the paper demonstrate to be efficient for classification, leading to a percentage of about 90 % successfully classified examples among the ones not used in the training phase. Similar results are obtained considering only a subset of 15 features.

Concepts and tools from cooperative game theory are exploited to quantify the role played by each member of a team in estimating the position of an observed point of interest. The measure of importance known as “Shapley value” is used to this end. From the theoretical point view, we propose a specific form of the characteristic function for the class of cooperative games under investigation. In the numerical analysis, different configurations of a group of individuals are considered: all individuals looking at a mobile point of interest, one of them replaced with an artificially-generated one who looks exactly toward the point of interest, and directions of the heads replaced with randomly-generated directions. The corresponding experimental outcomes are compared.

This paper presents a conceptual framework for the analysis of expressive qualities of movement. Our perspective is to model an observer of a dance performance. The conceptual framework is made of four layers, ranging from the physical signals that sensors capture to the qualities that movement communicate (e.g., in terms of emotions). The framework aims to provide a conceptual background the development of computational systems can build upon, with a particular reference to systems analyzing a vocabulary of expressive movement qualities, and translating them to other sensory channels, such as the auditory modality. Such systems enable their users to "listen to a choreography" or to "feel a ballet", in a new kind of cross-modal mediated experience.

In this paper, we present a new multimodal repository for the analysis of expressive movement qualities in dance. First, we discuss guidelines and methodology that we applied to create this repository. Next, the technical setup of recordings and the platform for capturing the synchronized audio-visual, physiological, and motion capture data are presented. The initial content of the repository consists of about 90 minutes of short dance performances movement sequences, and improvisations performed by four dancers, displaying three expressive qualities: Fluidity, Impulsivity, and Rigidity.

In this paper we present the objectives and preliminary work of WhoLoDancE a Research and Innovation Action funded under the European Union's Horizon 2020 programme, aiming at using new technologies for capturing and analyzing dance movement to facilitate whole-body interaction learning experiences for a variety of dance genres. Dance is a diverse and heterogeneous practice and WhoLoDancE will develop a protocol for the creation and/or selection of dance sequences drawn from different dance styles for different teaching and learning modalities. As dance learning practice lacks standardization beyond dance genres and specific schools and techniques, one of the first project challenges is to bring together a variety of dance genres and teaching practices and work towards a methodology for selecting the appropriate shots for motion capturing, to acquire kinetic material which will provide a satisfying proof of concept for Learning scenarios of particular genres. The four use cases we are investigating are 1) classical ballet, 2) contemporary dance, 3) flamenco and 4) Greek folk dance.

Intrapersonal synchronization of limb movements is a relevantfeature for assessing coordination of motoric behaviour.In this paper, we show that it can also distinguishbetween full-body movements performed with different expressivequalities, namely rigidity, fluidity, and impulsivity.For this purpose, we collected a dataset of movements performedby professional dancers and annotated the perceived movement qualities with the help of a group of experts inexpressive movement analysis. We computed intrapersonal synchronization by applying the Event Synchronization algorithmto the time-series of the speed of arms and hands.Results show that movements performed with different qualitiesdisplay a significantly different amount of intrapersonalsynchronization: impulsive movements are the mostsynchronized, thefluid ones show the lowest values of synchronization, and the rigid ones lay in between.

This article introduces a model and a system for embodied social active listening to sound and music content. The model is based on the simultaneous navigation/exploration of multiple maps, starting from a low-level physical map, up to a high-level emotional, affective map. The paper discusses the concepts underlying such a model, the system implementing it, and two concrete examples: The Orchestra Explorer designed for the Museo degli Strumenti Musicali at Accademia Nazionale di Santa Cecilia, Roma, Italy, and the interactive dance and music performance The Bow Is bent and Drawn (composer Nicola Ferrari), presented at Casa Paganini, Genova, Italy, in occasion of the opening concert of the 8th Intl. Conference on New Interfaces for Musical Expression (NIME08), June 4, 2008, and in museum interactive experiences at Festival della Scienza, Genova. This work is part of current research at Casa Paganini—InfoMus on embodied social active listening to sound and music content through the analysis and processing of expressiveness in human full-body movement and gesture. A user-centric interactive multimedia system architecture is proposed, operating on prerecorded music. From the perspective of valorization of cultural heritage, this research provides engaging paradigms of interaction with prerecorded music content, enabling a large number of nonexpert users to rediscover the musical heritage (e.g., classical and contemporary music) they may not be familiar with. Research was carried out in the framework of the EU-ICT Project SAME (Sound and Music for Everyone, Everyday, Everywhere, Every Way, www.sameproject.eu, 2008–2010) and has been recently extended to active experience of audiovisual content, and in particular to cultural heritage and museum scenarios: the novel interactive permanent museum exhibition Viaggiatori di Sguardo has been designed and developed at Palazzo Ducale, Genova, enabling visitors to explore virtually the UNESCO Heritage Palazzi dei Rolli in Genova.

This paper presents a framework for analysis of affective behavior starting with a reduced amount of visual information related to human upper-body movements. The main goal is to individuate a minimal representation of emotional displays based on nonverbal gesture features. The GEMEP (Geneva multimodal emotion portrayals) corpus was used to validate this framework. Twelve emotions expressed by 10 actors form the selected data set of emotion portrayals. Visual tracking of trajectories of head and hands were performed from a frontal and a lateral view. Postural/shape and dynamic expressive gesture features were identified and analyzed. A feature reduction procedure was carried out, resulting in a 4D model of emotion expression that effectively classified/grouped emotions according to their valence (positive, negative) and arousal (high, low). These results show that emotionally relevant information can be detected/measured/obtained from the dynamic qualities of gesture. The framework was implemented as software modules (plug-ins) extending the EyesWeb XMI Expressive Gesture Processing Library and is going to be used in user centric, networked media applications, including future mobiles, characterized by low computational resources, and limited sensor systems.

This paper presents a multimodal system for real-time analysis of nonverbal affective social interaction in small groups of users. The focus is on two major aspects of affective social interaction: the synchronization of the affective behavior within a small group and the emergence of functional roles, such as leadership. A small group of users is modeled as a complex system consisting of single interacting components that can auto-organize and show global properties. Techniques are developed for computing quantitative measures of both synchronization and leadership. Music is selected as experimental test-bed since it is a clear example of interactive and social activity, where affective nonverbal communication plays a fundamental role. The system has been implemented as software modules for the EyesWeb XMI platform (http://www.eyesweb.org). It has been used in experimental frameworks (a violin duo and a string quartet) and in real-world applications (in user-centric applications for active music listening). Further application scenarios include entertainment, edutainment, therapy and rehabilitation, cultural heritage, and museum applications. Research has been carried out in the framework of the EU-ICT FP7 Project SAME (http://www.sameproject.eu).

Based on the current context of the Sound and Music Computing (SMC) field, the state of the art in research and the open issues that have been identified and described in other articles of this journal issue, in this article we make a step forward and try to identify the broad SMC challenges and we propose strategies with which to tackle them. On the research side we identify a clear need for designing better sound objects and environments and for promoting research to understand, model, and improve human interaction with sound and music. In the education domain we feel the need for better training our multidisciplinary researchers and to make sure that they can contribute to the multicultural society we live in. There is also a clear need for improving the transferring of the knowledge and technologies generated by our community. Finally we claim that the SMC field should be very much concerned with its social context and that a number of current social concerns should be addressed. We accompany each of these challenges with strategies that should help researchers, educators and policy makers take specific actions to advance in the proposed SMC roadmap. This work was partially funded by the S2S2 Coordination Action under the FET initiative of the 6th Framework Programme of the EC.

This paper introduces an algorithm for automatically measuring impulsivity. This can be used as a major expressive movement feature in the development of systems for real-time analysis of emotion expression from human full-body movement, a research area which has received increased attention in the affective computing community. In particular, our algorithm is developed in the framework of the EU-H2020-ICT Project DANCE aiming at investigating techniques for sensory substitution in blind people, in order to enable perception of and participation in non-verbal, artistic whole-body experiences. The algorithm was tested by applying it to a reference archive of short dance performances. The archive includes a collection of both impulsive and fluid movements. Results show that our algorithm can reliably distinguish impulsive vs. sudden performances.

Background in musicology Understanding the gesture-based foundations of musical involvement opens a number of new perspectives for musicology, with the likely effect of a change in approach. Giving justice to the role of gesture in music calls for an integrated view of perception and action, which implies a shift from auditory-based cognition towards embodied cognition, an approach that includes the whole human body as mediator between mental processes and physical energy. Background in computing With the help of new measurement tools, new computational models of multi-modal interaction, and new interactive multimedia platforms, it becomes easier to study music from a multi-modal point of view, both in terms of scientific and artistic perspectives. Music also offers an appropriate domain for the testing and developing of interactive systems and computational models for handling non-verbal multi-modal communication. Aims In studying musical gesture, both musicology and engineering can profit from a joint collaboration. While technology provides new tools for the measurement and modelling of musical involvement, knowledge of its corporeal foundation may largely contribute to the development of technology that in turn contributes to measurement and modelling. Furthermore, this knowledge may be exploited in artistic applications. As such, the interaction between musicology and music engineering may lead to interesting developments in the domain of embodied cognition and interactive multimedia platforms. Main contribution An overview of recent work on the development of an interactive multimedia platform for the measurement and modelling of musical expressiveness is presented. The paper introduces the notion of corporeal imitation and mirroring behavior as a core concept fur understanding musical expressiveness. Implications The approach advocated in this paper implies a shift of paradigm in systematic musicology, away from an all too narrow focus on auditory perception and sound structure, towards a broader view that encompasses multi-modal musical involvement. This approach is grounded in multi-sensory integration, the coupling of perception and action, the study of motor imitation, and issues that relate to affect, emotions and subjectivity. The advantage of using a interactive multimedia platform for music research, rather than ad hoc solutions, pertains to the use of modular functionalities, which allow rapid prototyping of experimental setup, flexible modelling, and the subsequent use of results in artistic applications.

Analysis of expressiveness in human gesture can lead to new paradigms for the design of improved human-machine interfaces, thus enhancing users’ participation and experience in mixed reality applications and context-aware mediated environments. The development of expressive interfaces decoding the highly affective information gestures convey opens novel perspectives in the design of interactive multimedia systems in several application domains: performing arts, museum exhibits, edutainment, entertainment, therapy, and rehabilitation. This paper describes some recent developments in our research on expressive interfaces by presenting computational models and algorithms for the real-time analysis of expressive gestures in human full-body movement. Such analysis is discussed both as an example and as a basic component for the development of effective expressive interfaces. As a concrete result of our research, a software platform named EyesWeb was developed (http://www.eyesweb.org). Besides supporting research, EyesWeb has also been employed as a concrete tool and open platform for developing real-time interactive applications.

This paper presents and discusses some experiments having the purpose of planning, developing and validating aesthetically resonant environments for different types of sensorimotor impairments which affect Parkinson's patients. From a technical point of view the aim is to develop a computational open architecture in which it is possible to integrate modules for gesture analysis and recognition and for interactive construction of therapeutic exercises based on multimedia stimulation in real time. The clinical objective is to experiment with a device of sensorimotor stimulation that supports akinesia compensation by controlling movement rhythmic structures in subjects with Parkinson's disease. The EyesWeb open architecture has been used to analyse patients' motion and to produce visual feedback during therapy sessions in real time. A pilot study has been conducted on two Parkinson's disease patients in the framework of the EU IST project CARE-HERE. Copyright © 2003 John Wiley & Sons, Ltd.

This article introduces the approach, the developed models, and ongoing research issues on expressiveness and physical interaction in interactive systems, with a special focus on human-robot communication. Research issues and related concrete examples from the interactive multimedia performance “L‘Ala dei Sensi” (literally, “The Wing of the Senses”), held in Ferrara (Italy) in November 1999, are presented. “L‘Ala dei Sensi” is formed by different episodes, a subset of which include one or more dancers interacting in real-time with a robot on stage. This interaction can influence and generate computer-generated music and video. Experiments have been conduced on the concept of musical and visual “clones” on large screens interacting with humans and the robot. The underlying models and approaches are introduced, with a special focus on expressiveness in the physical interaction of dancers with mobile robots and with visual/music “clones”. An overview of the hardware and software architecture is shortly prese...

Music content processing is receiving more and more attention from the multimedia community. On the one hand, research developments in computer music during the last decades are conducing toward more sophisticated and novel musical applications; on the other hand, such research is useful for the development of non-musical applications embedding music content processing to enhance effectiveness and usability. Intelligent user interfaces, games and entertainment, edutainment, and rehabilitation are some examples. This paper focuses on the role of analysis and synthesis of expressive content in multimedia signals, with particular emphasis on music, integrated with human movement and visual media in the framework of intelligent interactive systems. Research on “affective computing” in the USA and on “KANSEI information processing” (KIP) in Japan are consolidated research areas dealing with artificial emotions and expressivity. We focus on a third way, where research on expressivity and artificial emotions is ...

The Art Exhibition of ACM Multimedia 2013 has attracted significant work from a variety of digital artists collaborating with research institutions. We have endeavored to select exhibits that achieved an interesting balance between technology and artistic intent. The techniques underpinning these artworks are relevant to several technical tracks of the conference, in particular those dealing with human-centered and interactive media. We briefly review how the various installations revisit current topics in Multimedia research, focusing more specifically on their approach to dynamic content generation, user experience, multimodality, and affective interfaces. Once again, the unique blend of technology and user experience does not limit itself to showcasing recent advances in interactive media, and should be of interest to all conference participants as well as the general public the exhibition space will be open to.

The present study investigates expressive non-verbal interaction in musical context starting from behavioral features extracted at individual and group level. We define four features related to head movement and direction that may help gaining insight on the expressivity and cohesion of the performance. Our preliminary findings obtained from the analysis of a string quartet recorded in ecological settings show that these features may help in distinguishing between two types of performance: (a) a concert-like condition where all musicians aim at performing at best, (b) a perturbed one where the 1st violinist devises alternative interpretations of the music score without discussing them with the other musicians.

We propose a paradigm and an end-to-end system to support authoring and real-time active listening experience of prerecorded music. Users are prosumers who navigate and express themselves in a shared (physical or virtual) orchestra space, populated by the sections of an orchestra playing a prerecorded music piece. The system consists of two components: an authoring system enabling the user to edit and shape personalized versions of active listening experiences, and a run-time environment implementing the mechanisms of interaction with the active music content. Both components exploit metadata defining the mapping rules of users expressive gesture and context information onto sound and music processing. A user interacts with content by holding her smartphone in her hand and by using environmental sensors and context-aware information to shape the experience, possibly in cooperation with other users.

A framework and software system for real-time tracking and analysis of non-verbal expressive and emotional behavior is proposed. The objective is to design and create multimodal interactive systems for the automated analysis of emotions. The system will give audiovisual feedback to support the therapy of children affected by Autism Spectrum Conditions (ASC). The system is based on the EyesWeb XMI open software platform and on Kinect depth sensors.

Since its first edition in 2009, the International Workshop on Social Behaviour in Music (SBM) has been an occasion for researchers and practitioners for discussing recent advances in automated analysis of social behaviour, being music the selected test-bed and application scenario. The first edition of SBM was held in Vancouver, Canada, in the framework of the 2009 IEEE International Conference on Social Computing (SocialCom 2009). The second one was held in Genova, Italy, in the framework of the 4th International ICST Conference on Intelligent Technologies for Interactive Entertainment (Intetain 2011). SBM is now at its third edition, which takes place in the framework of the 14th International Conference on Multimodal Interaction (ICMI 2012), Santa Monica, California, USA. Again, SBM aims at providing a picture of current research breakthrough and issues, giving at the same time directions for future works and collaborations.

This paper presents the interactive installation “Come un’Onda premuta da un’Onda” (“As Wave impels a Wave”, a citation from Ovidio’s “Metamorphoses” as a metaphor of time). The installation, presented in its early version at the Festival della Scienza 2009, introduces visitors to the rich history and artistic content of a monumental building: a virtual walk through the time. The core idea is to support an active experience based on novel paradigms of interaction and narration. The active experience is grounded on an informational environment characterized by an invisible “sound scent” map. The research is partially supported by the EU FP7 ICT I-SEARCH project.

An Invisible Line is an installation focusing on the remote communication between 2 human users, based on the analysis of full-body expressivity. It aims at creating shared, networked, social experiences. It is the result of a scientific and artistic collaboration between Casa Paganini - InfoMus Lab (Genova, Italy), IRCAM (Paris, Italy) and The Hochschule fur Musik und Theater (Hamburg, Germany).

In this article, a unified framework for multimodal search and retrieval is introduced. The framework is an outcome of the research that took place within the I-SEARCH European Project. The proposed system covers all aspects of a search and retrieval process, namely low-level descriptor extraction, indexing, query formulation, retrieval and visualisation of the search results. All I-SEARCH components advance the state of the art in the corresponding scientific fields. The I-SEARCH multimodal search engine is dynamically adapted to end-user's devices, which can vary from a simple mobile phone to a high-performance PC.

This article introduces a cognitive model and an experimental system for the representation and real‐time processing of music and multimedia, based on both artificial intelligence (AI) and traditional techniques. First, it introduces some basic issues on the requirements of Al‐based music systems and the class of problems faced by our system (music and multimedia composition, real‐time performance, and analysis). Then, our proposal of a representation scheme — on the basis of an implemented system, called HARP — is described. It is based on the integration of different formalisms, able to manage the different nature and levels of music and multimedia objects, from the symbolic, knowledge level, dealing with multilevel, abstract representations and reasoning mechanisms, to the subsymbolic level, dealing with low‐level processes, images, and physical signals. In our model, the symbolic level integrates a semantic network language of the family of KL‐ONE with a temporal logic language and production...

The paper introduces a system and a software architecture for the representation and real-time processing of sound, music, and multimedia based on artificial intelligence techniques. This system, called WinProcne/HARP, is able to represent objects in a two-fold formalism—symbolic and analogical—at different levels of abstraction, and to carry out plans according to the user's goals. It also provides both formal and informal analysis capabilities for extracting information. In WinProcne/HARP the user can build, update, browse, and merge various knowledge bases of sound, music, and multimedia material, as well as enter queries, start and manage real time performance, using a high-level graphical user interface. The system is currently used by researchers and composers in various experiments, including (a) advanced robotics projects, in which the system is used as a tool for interacting, cotrolling and simulating robot movements, and (b) theatrical automation, where the system is delegated to manage and integrate sound, music, and three-dimensional computer animations of humanoid figures. The paper explicitly refers to some applications in the music field.

This chapter focuses on systems and interfaces for multimodal analysis of expressive gesture as a key element of music performance. Research on expressive gesture became particularly relevant in recent years. Psychological studies have been a fundamental source for automatic analysis of expressive gesture since their contribution in identifying the most significant features to be analysed. A further relevant source has been research in the humanistic tradition, in particular choreography. As a major example, in his Theory of Effort, choreographer Rudolf Laban describes the most significant qualities of movement. Starting from these sources, several models, systems, and techniques for analysis of expressive gesture were developed. This chapter presents an overview of methods for the analysis, modelling, and understanding of expressive gesture in musical performance. It introduces techniques resulted from the research developed over the years by the authors: from early experiments of human-robot interaction in the context of music performance up to recent set-ups of innovative interfaces and systems for active experience of sound and music content. The chapter ends with an overview of possible future research challenges.

Social interaction and embodiment are key issues for future User Centric Media. Social networks and games are more and more characterized by an active, physical participation of the users. The integration in mobile devices of a growing number of sensors to capture users' physical activity (e.g., accelerometers, cameras) and context information (GPS, location) supports novel systems capable to connect audiovisual content processing and communication to users social behavior, including joint movement and physical engagement. In this paper, a system enabling a novel paradigm for social, active experience of sound and music content is presented. An instance of such a system, named Sync`n'Move, allowing two users to explore a multi-channel pre-recorded music piece as the result of their social interaction, and in particular of their synchronization, is introduced. This research has been developed in the framework of the EU-ICT Project SAME ( www.sameproject.eu ) and has been presented at Agora Festival (IRCAM, Centre Pompidou, Paris, June 2009). In that occasion, Sync`n'Move has been evaluated by both expert and non expert users, and results are briefly presented. Perspectives on the impact of such a novel paradigm and system in future User Centric Media are finally discussed, with a specific focus on social active experience of audiovisual content.