Search

Full-body human movement is characterized by fine-grain expressive qualities that humans are easily capable of exhibiting and recognizing in others’ movement. In sports (e.g., martial arts) and performing arts (e.g., dance), the same sequence of movements can be performed in a wide range of ways characterized by different qualities, often in terms of subtle (spatial and temporal) perturbations of the movement. Even a non-expert observer can distinguish between a top-level and average performance by a dancer or martial artist. The difference is not in the performed movements--the same in both cases--but in the “quality” of their performance. In this article, we present a computational framework aimed at an automated approximate measure of movement quality in full-body physical activities. Starting from motion capture data, the framework computes low-level (e.g., a limb velocity) and high-level (e.g., synchronization between different limbs) movement features. Then, this vector of features is integrated to compute a value aimed at providing a quantitative assessment of movement quality approximating the evaluation that an external expert observer would give of the same sequence of movements. Next, a system representing a concrete implementation of the framework is proposed. Karate is adopted as a testbed. We selected two different katas (i.e., detailed choreographies of movements in karate) characterized by different overall attitudes and expressions (aggressiveness, meditation), and we asked seven athletes, having various levels of experience and age, to perform them. Motion capture data were collected from the performances and were analyzed with the system. The results of the automated analysis were compared with the scores given by 14 karate experts who rated the same performances. Results show that the movement-quality scores computed by the system and the ratings given by the human observers are highly correlated (Pearson’s correlations r = 0.84, p = 0.001 and r = 0.75, p = 0.005).

The work reports ongoing research about a computational method, based on cooperative games on graphs, aimed at detecting the perceived origin of full-body human movement and its propagation. Compared with previous works, a larger set of movement features is considered, and a ground truth is produced, able to assess and compare the effectiveness of each such feature. This is done through the use of the Shapley Value as a centrality index. An Origin of Movement Continuum is also defined, as the basis for creating a repository of movement qualities.

A multimodal dataset is presented, which has been collected for analyzing and measuring the quality of movement performed during sport activities. Martial arts (namely karate) are taken as test-beds for investigation. Karate encompasses predefined sequences of movements (“katas”) that can be carried out with different qualities, e.g., by performers at different skill levels (highly vs. poorly skilled). The experimental setup and method are described. The dataset is composed of motion capture (MoCap) data, synchronized with video and audio recordings, of several participants with different levels of experience. The raw MoCap data are independent of any particular post-processing algorithm and can be used for other research purposes. In the second part of the paper, a set of measures is proposed to evaluate a kata performance. They are based on the geometrical and kinematic features, such as posture correctness and synchronization between limbs. and were chosen according to karate experts' opinion.

This paper addresses the use of a remote interactive platform to support home-based rehabilitation for children with motor and cognitive impairment. The interaction between user and platform is achieved on customizable full-body interactive serious games (exergames). These exergames perform real-time analysis of multimodal signals to quantify movement qualities and postural attitudes. Interactive sonification of movement is then applied for providing a real-time feedback based on “aesthetic resonance” and engagement of the children. The games also provide log file recordings therapists can use to assess the performance of the children and the effectiveness of the games. The platform allows the customization of the games to address the children's needs. The platform is based on the EyesWeb XMI software, and the games are designed for home usage, based on Kinect for Xbox One and simple sensors including 3-axis accelerometers available in low-cost Android smartphones.

International audience; Laughter is a social signal capable of facilitating interaction in groups of people: it communicates interest, helps to improve creativity, and facilitates sociability. This article focuses on: endowing virtual characters with computational models of laughter synthesis, based on an expressivity-copying paradigm; evaluating how the physically co-presence of the laughing character impacts on the user’s perception of an audio stimulus and mood. We adopt music as a means to stimulate laughter. Results show that the character presence influences the user’s perception of music and mood. Expressivity-copying has an influence on the user’s perception of music, but does not have any significant impact on mood.

The present study investigates expressive non-verbal interaction in the musical context starting from behavioral features extracted at individual and group levels. Four groups of features are defined, which are related to head movement and direction, and may help gaining insight on the expressivity and cohesion of the performance, discriminating between different performance conditions. Then, the features are evaluated both at a global scale and at a local scale. The findings obtained from the analysis of a string quartet recorded in an ecological setting show that using these features alone or in their combination may help in distinguishing between two types of performance: (a) a concert-like condition, where all musicians aim at performing at best, (b) a perturbed one, where the 1 $$\mathrm{st}$$ violinist devises alternative interpretations of the music score without discussing them with the other musicians. In the global data analysis, the discriminative power of the features is investigated through statistical tests. Then, in the local data analysis, a larger amount of data is used to exploit more sophisticated machine learning techniques to select suitable subsets of the features, which are then used to train an SVM classifier to perform binary classification. Interestingly, the features whose discriminative power is evaluated as large (respectively, small) in the global analysis are also evaluated in a similar way in the local analysis. When used together, the 22 features that have been defined in the paper demonstrate to be efficient for classification, leading to a percentage of about 90 % successfully classified examples among the ones not used in the training phase. Similar results are obtained considering only a subset of 15 features.

This paper presents a conceptual framework for the analysis of expressive qualities of movement. Our perspective is to model an observer of a dance performance. The conceptual framework is made of four layers, ranging from the physical signals that sensors capture to the qualities that movement communicate (e.g., in terms of emotions). The framework aims to provide a conceptual background the development of computational systems can build upon, with a particular reference to systems analyzing a vocabulary of expressive movement qualities, and translating them to other sensory channels, such as the auditory modality. Such systems enable their users to "listen to a choreography" or to "feel a ballet", in a new kind of cross-modal mediated experience.

In this paper, we present a new multimodal repository for the analysis of expressive movement qualities in dance. First, we discuss guidelines and methodology that we applied to create this repository. Next, the technical setup of recordings and the platform for capturing the synchronized audio-visual, physiological, and motion capture data are presented. The initial content of the repository consists of about 90 minutes of short dance performances movement sequences, and improvisations performed by four dancers, displaying three expressive qualities: Fluidity, Impulsivity, and Rigidity.

In this paper we present the objectives and preliminary work of WhoLoDancE a Research and Innovation Action funded under the European Union's Horizon 2020 programme, aiming at using new technologies for capturing and analyzing dance movement to facilitate whole-body interaction learning experiences for a variety of dance genres. Dance is a diverse and heterogeneous practice and WhoLoDancE will develop a protocol for the creation and/or selection of dance sequences drawn from different dance styles for different teaching and learning modalities. As dance learning practice lacks standardization beyond dance genres and specific schools and techniques, one of the first project challenges is to bring together a variety of dance genres and teaching practices and work towards a methodology for selecting the appropriate shots for motion capturing, to acquire kinetic material which will provide a satisfying proof of concept for Learning scenarios of particular genres. The four use cases we are investigating are 1) classical ballet, 2) contemporary dance, 3) flamenco and 4) Greek folk dance.

This article introduces a model and a system for embodied social active listening to sound and music content. The model is based on the simultaneous navigation/exploration of multiple maps, starting from a low-level physical map, up to a high-level emotional, affective map. The paper discusses the concepts underlying such a model, the system implementing it, and two concrete examples: The Orchestra Explorer designed for the Museo degli Strumenti Musicali at Accademia Nazionale di Santa Cecilia, Roma, Italy, and the interactive dance and music performance The Bow Is bent and Drawn (composer Nicola Ferrari), presented at Casa Paganini, Genova, Italy, in occasion of the opening concert of the 8th Intl. Conference on New Interfaces for Musical Expression (NIME08), June 4, 2008, and in museum interactive experiences at Festival della Scienza, Genova. This work is part of current research at Casa Paganini—InfoMus on embodied social active listening to sound and music content through the analysis and processing of expressiveness in human full-body movement and gesture. A user-centric interactive multimedia system architecture is proposed, operating on prerecorded music. From the perspective of valorization of cultural heritage, this research provides engaging paradigms of interaction with prerecorded music content, enabling a large number of nonexpert users to rediscover the musical heritage (e.g., classical and contemporary music) they may not be familiar with. Research was carried out in the framework of the EU-ICT Project SAME (Sound and Music for Everyone, Everyday, Everywhere, Every Way, www.sameproject.eu, 2008–2010) and has been recently extended to active experience of audiovisual content, and in particular to cultural heritage and museum scenarios: the novel interactive permanent museum exhibition Viaggiatori di Sguardo has been designed and developed at Palazzo Ducale, Genova, enabling visitors to explore virtually the UNESCO Heritage Palazzi dei Rolli in Genova.

This paper presents a multimodal system for real-time analysis of nonverbal affective social interaction in small groups of users. The focus is on two major aspects of affective social interaction: the synchronization of the affective behavior within a small group and the emergence of functional roles, such as leadership. A small group of users is modeled as a complex system consisting of single interacting components that can auto-organize and show global properties. Techniques are developed for computing quantitative measures of both synchronization and leadership. Music is selected as experimental test-bed since it is a clear example of interactive and social activity, where affective nonverbal communication plays a fundamental role. The system has been implemented as software modules for the EyesWeb XMI platform (http://www.eyesweb.org). It has been used in experimental frameworks (a violin duo and a string quartet) and in real-world applications (in user-centric applications for active music listening). Further application scenarios include entertainment, edutainment, therapy and rehabilitation, cultural heritage, and museum applications. Research has been carried out in the framework of the EU-ICT FP7 Project SAME (http://www.sameproject.eu).

Based on the current context of the Sound and Music Computing (SMC) field, the state of the art in research and the open issues that have been identified and described in other articles of this journal issue, in this article we make a step forward and try to identify the broad SMC challenges and we propose strategies with which to tackle them. On the research side we identify a clear need for designing better sound objects and environments and for promoting research to understand, model, and improve human interaction with sound and music. In the education domain we feel the need for better training our multidisciplinary researchers and to make sure that they can contribute to the multicultural society we live in. There is also a clear need for improving the transferring of the knowledge and technologies generated by our community. Finally we claim that the SMC field should be very much concerned with its social context and that a number of current social concerns should be addressed. We accompany each of these challenges with strategies that should help researchers, educators and policy makers take specific actions to advance in the proposed SMC roadmap. This work was partially funded by the S2S2 Coordination Action under the FET initiative of the 6th Framework Programme of the EC.

An emerging trend in interactive music performance consists of the audience directly participating in the performance by means of mobile devices. This is a step forward with respect to concepts like active listening and collaborative music making: non-expert members of an audience are enabled to directly participate in a creative activity such as the performance. This requires the availability of technologies for capturing and analysing in real-time the natural behaviour of the users/performers, with particular reference to non- verbal expressive and social behaviour. This paper presents a prototype of a non-verbal expressive and social search engine and active listening system, enabling two teams of non-expert users to act as performers. The performance consists of real-time sonic manipulation and mixing of music pieces selected according to features characterising performers’ movements captured by mobile devices. The system is described with specific reference to the SIEMPRE Podium Performance, a non-verbal socio-mobile music performance presented at the Art & ICT Exhibition that took place in Vilnius (LI) in November 2013.

This paper presents and discusses some experiments having the purpose of planning, developing and validating aesthetically resonant environments for different types of sensorimotor impairments which affect Parkinson's patients. From a technical point of view the aim is to develop a computational open architecture in which it is possible to integrate modules for gesture analysis and recognition and for interactive construction of therapeutic exercises based on multimedia stimulation in real time. The clinical objective is to experiment with a device of sensorimotor stimulation that supports akinesia compensation by controlling movement rhythmic structures in subjects with Parkinson's disease. The EyesWeb open architecture has been used to analyse patients' motion and to produce visual feedback during therapy sessions in real time. A pilot study has been conducted on two Parkinson's disease patients in the framework of the EU IST project CARE-HERE. Copyright © 2003 John Wiley & Sons, Ltd.

This article introduces the approach, the developed models, and ongoing research issues on expressiveness and physical interaction in interactive systems, with a special focus on human-robot communication. Research issues and related concrete examples from the interactive multimedia performance “L‘Ala dei Sensi” (literally, “The Wing of the Senses”), held in Ferrara (Italy) in November 1999, are presented. “L‘Ala dei Sensi” is formed by different episodes, a subset of which include one or more dancers interacting in real-time with a robot on stage. This interaction can influence and generate computer-generated music and video. Experiments have been conduced on the concept of musical and visual “clones” on large screens interacting with humans and the robot. The underlying models and approaches are introduced, with a special focus on expressiveness in the physical interaction of dancers with mobile robots and with visual/music “clones”. An overview of the hardware and software architecture is shortly prese...

Music content processing is receiving more and more attention from the multimedia community. On the one hand, research developments in computer music during the last decades are conducing toward more sophisticated and novel musical applications; on the other hand, such research is useful for the development of non-musical applications embedding music content processing to enhance effectiveness and usability. Intelligent user interfaces, games and entertainment, edutainment, and rehabilitation are some examples. This paper focuses on the role of analysis and synthesis of expressive content in multimedia signals, with particular emphasis on music, integrated with human movement and visual media in the framework of intelligent interactive systems. Research on “affective computing” in the USA and on “KANSEI information processing” (KIP) in Japan are consolidated research areas dealing with artificial emotions and expressivity. We focus on a third way, where research on expressivity and artificial emotions is ...

When people perform a task as part of a joint action, their behavior is not the same as it would be if they were performing the same task alone, since it has to be adapted to facilitate shared understanding (or sometimes to prevent it). Joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates joint action. Here we compare the expressive movement of violinists when playing in solo and ensemble conditions. The first violinists of two string quartets (SQs), professional and student, were asked to play the same musical fragments in a solo condition and with the quartet. Synchronized multimodal recordings were created from the performances, using a specially developed software platform. Different patterns of head movement were observed. By quantifying them using an appropriate measure of entropy, we showed that head movements are more predictable in the quartet scenario. Rater evaluations showed that the change does not, as might be assumed, entail markedly reduced expression. They showed some ability to discriminate between solo and ensemble performances, but did not distinguish them in terms of emotional content or expressiveness. The data raise provocative questions about joint action in realistically complex scenarios.

The Art Exhibition of ACM Multimedia 2013 has attracted significant work from a variety of digital artists collaborating with research institutions. We have endeavored to select exhibits that achieved an interesting balance between technology and artistic intent. The techniques underpinning these artworks are relevant to several technical tracks of the conference, in particular those dealing with human-centered and interactive media. We briefly review how the various installations revisit current topics in Multimedia research, focusing more specifically on their approach to dynamic content generation, user experience, multimodality, and affective interfaces. Once again, the unique blend of technology and user experience does not limit itself to showcasing recent advances in interactive media, and should be of interest to all conference participants as well as the general public the exhibition space will be open to.

This article presents an Interactive Multimodal Environment (IME), the Stanza Logo-Motoria, designed to support learning in primary schools. In particular we describe the use of this system as a tool (a) to practice listening to English as a Second Language (ESL) and (b) to enable children with severe disabilities to perform an interactive listening. We document the ongoing experimentation of the Stanza Logo-Motoria in ESL lessons and report its encouraging results. Moreover, we explain how it may be possible, by means of the Stanza Logo-Motoria, to redesign traditional learning environments in order to allow pupils to experience listening as an active and engaging experience.

We propose a paradigm and an end-to-end system to support authoring and real-time active listening experience of prerecorded music. Users are prosumers who navigate and express themselves in a shared (physical or virtual) orchestra space, populated by the sections of an orchestra playing a prerecorded music piece. The system consists of two components: an authoring system enabling the user to edit and shape personalized versions of active listening experiences, and a run-time environment implementing the mechanisms of interaction with the active music content. Both components exploit metadata defining the mapping rules of users expressive gesture and context information onto sound and music processing. A user interacts with content by holding her smartphone in her hand and by using environmental sensors and context-aware information to shape the experience, possibly in cooperation with other users.

Within the EU ILHAIRE Project, researchers of several disciplines (e.g., computer sciences, psychology) collaborate to investigate the psychological foundations of laughter, and to bring this knowledge into shape for the use in new technologies (i.e., affective computing). Within this framework, in order to endow machines with laughter capabilities (encoding as well as decoding), one crucial task is an adequate description of laughter in terms of morphology. In this paper we present a work methodology towards automated full body laughter detection: starting from expert annotations of laughter videos we aim to identify the body features that characterize laughter.

International audience; In this paper, we report on work around the I-SEARCH EU (FP7 ICT STREP) project whose objective is the development of a multimodal search engine. We present the project's objectives, and detail the achieved results, amongswhich a Rich Unified Content Description format.

This paper evaluates three different interactive sonifications of dyadic coordinated human rhythmic activity. An index of phase synchronisation of gestures was chosen as coordination metric. The sonifications are implemented as three prototype applications exploiting mobile devices: Sync’n’Moog, Sync’n’Move, and Sync’n’Mood. Sync’n’Moog sonifies the phase synchronisation index by acting directly on the audio signal and applying a nonlinear time-varying filtering technique. Sync’n’Move intervenes on the multi-track music content by making the single instruments emerge and hide. Sync’n’Mood manipulates the affective features of the music performance. The three sonifications were also tested against a condition without sonification. QC 20150623 SAME

An Invisible Line is an installation focusing on the remote communication between 2 human users, based on the analysis of full-body expressivity. It aims at creating shared, networked, social experiences. It is the result of a scientific and artistic collaboration between Casa Paganini - InfoMus Lab (Genova, Italy), IRCAM (Paris, Italy) and The Hochschule fur Musik und Theater (Hamburg, Germany).

This article introduces a cognitive model and an experimental system for the representation and real‐time processing of music and multimedia, based on both artificial intelligence (AI) and traditional techniques. First, it introduces some basic issues on the requirements of Al‐based music systems and the class of problems faced by our system (music and multimedia composition, real‐time performance, and analysis). Then, our proposal of a representation scheme — on the basis of an implemented system, called HARP — is described. It is based on the integration of different formalisms, able to manage the different nature and levels of music and multimedia objects, from the symbolic, knowledge level, dealing with multilevel, abstract representations and reasoning mechanisms, to the subsymbolic level, dealing with low‐level processes, images, and physical signals. In our model, the symbolic level integrates a semantic network language of the family of KL‐ONE with a temporal logic language and production...

The paper introduces a system and a software architecture for the representation and real-time processing of sound, music, and multimedia based on artificial intelligence techniques. This system, called WinProcne/HARP, is able to represent objects in a two-fold formalism—symbolic and analogical—at different levels of abstraction, and to carry out plans according to the user's goals. It also provides both formal and informal analysis capabilities for extracting information. In WinProcne/HARP the user can build, update, browse, and merge various knowledge bases of sound, music, and multimedia material, as well as enter queries, start and manage real time performance, using a high-level graphical user interface. The system is currently used by researchers and composers in various experiments, including (a) advanced robotics projects, in which the system is used as a tool for interacting, cotrolling and simulating robot movements, and (b) theatrical automation, where the system is delegated to manage and integrate sound, music, and three-dimensional computer animations of humanoid figures. The paper explicitly refers to some applications in the music field.

This chapter focuses on systems and interfaces for multimodal analysis of expressive gesture as a key element of music performance. Research on expressive gesture became particularly relevant in recent years. Psychological studies have been a fundamental source for automatic analysis of expressive gesture since their contribution in identifying the most significant features to be analysed. A further relevant source has been research in the humanistic tradition, in particular choreography. As a major example, in his Theory of Effort, choreographer Rudolf Laban describes the most significant qualities of movement. Starting from these sources, several models, systems, and techniques for analysis of expressive gesture were developed. This chapter presents an overview of methods for the analysis, modelling, and understanding of expressive gesture in musical performance. It introduces techniques resulted from the research developed over the years by the authors: from early experiments of human-robot interaction in the context of music performance up to recent set-ups of innovative interfaces and systems for active experience of sound and music content. The chapter ends with an overview of possible future research challenges.

Perception and attention mechanisms are of great importance for entities situated within complex dynamic environments. With roles extending greatly beyond passive information services about the external environment, such mechanisms actively prioritise, augment and expedite information to ensure that the potentially relevant is made available so appropriate action can take place. Here, we describe the rationale behind endowing artificial entities, or virtual agents, with real-time perception and attention systems. We cover the fundamentals of designing and building such systems. Once equipped, the resulting agents can achieve a more substantial connection with their environment for the purposes of reacting, planning, decision making and, ultimately, behaving.

Social interaction and embodiment are key issues for future User Centric Media. Social networks and games are more and more characterized by an active, physical participation of the users. The integration in mobile devices of a growing number of sensors to capture users' physical activity (e.g., accelerometers, cameras) and context information (GPS, location) supports novel systems capable to connect audiovisual content processing and communication to users social behavior, including joint movement and physical engagement. In this paper, a system enabling a novel paradigm for social, active experience of sound and music content is presented. An instance of such a system, named Sync`n'Move, allowing two users to explore a multi-channel pre-recorded music piece as the result of their social interaction, and in particular of their synchronization, is introduced. This research has been developed in the framework of the EU-ICT Project SAME ( www.sameproject.eu ) and has been presented at Agora Festival (IRCAM, Centre Pompidou, Paris, June 2009). In that occasion, Sync`n'Move has been evaluated by both expert and non expert users, and results are briefly presented. Perspectives on the impact of such a novel paradigm and system in future User Centric Media are finally discussed, with a specific focus on social active experience of audiovisual content.

Theatre stage and artistic performances aim at joining participants to act together, to share, shape, and co-create cultural content by means of active experiences. Future networked and social media technologies promote new ways of experiencing cultural events and artefacts. Nonverbal social signals are major components of future social media characterised by embodiment and physical engagement of users. This contribution introduces research results from the EU-ICT-FET SIEMPRE Project, presenting how these results were shown in the final performance at FET11.

In this paper, we propose a class of algorithms for the sub-optimal solution of a particular class of problems of process scheduling, particularly focusing on a case study in the area of flexible manufacturing systems (FMSs). The general class of problems we face in our approach is characterized as follows: there is a set of concurrent processes, each formed by a number of temporally related tasks (segments). Tasks are executable by alternate resource sets, different both in performance and costs. Processes and tasks are characterized by release times, due dates, and deadlines. Time constraints are also present in the availability of each resource in resource sets. It has been proven that such a problem does not admit an algorithm for an optimal solution in polynomial time. Our proposed algorithm finds a sub-optimal schedule according to a set of optimization criteria, based on task and process times (earliness, tardiness), and/or time independent costs of resources. Our approach to process scheduling is based on Timed Coloured Petri Nets. We describe the structure of the coordination and scheduling algorithms, concentrating on (i) the general-purpose component, and (ii) the application-dependent component. In particular, the paper focuses on the following issues: (i) theautomatic synthesis of Petri net models of the coordination subsystem, starting from the problem knowledge base; (ii) the dynamic behavior of the coordination subsystem, whose kernel is a High Level Petri net executor, a coordination process based on an original, general purpose algorithm; (iii) the structure of the real-time scheduling subsystem, based on particular heuristic sub-optimal multi-criteria algorithms. Furthermore, the paper defines the interaction mechanisms between the coordination and scheduling subsystems. Our approach clearly distinguishes the mechanism of the net execution from the decision support system. Two conceptually distinct levels, which correspond to two different, interacting implementation modules in the prototype CASE tool, have been defined: theexecutor and thescheduler levels. One of the outstanding differences between these levels is that the executor is conceived as a fast, efficient coordination process, without special-purpose problem-solving capabilities in case of conflicts. The scheduler, on the other hand, is the adaptive, distributed component, whose behavior may heavily depend on the problem class. If the scheduler fails, the executor is, in any case, able to proceed with a general-purpose conflict resolution strategy. Experimental results on the real-time performance of the kernel of the implemented system are finally shown in the paper. The approach described in this paper is at the basis of a joint project with industrial partners for the development of a CASE tool for the simulation of blast furnaces.

This paper presents a brief overview of the multidisciplinary scientific and technological research projects at Casa Paganini — InfoMus on ICT for therapy and rehabilitation, integrating computer science, multimedia, psychology and humanistic theories. Research is grounded on the understanding and modeling of non-verbal human behavior, including the role of expressive gesture and social signals in multimodal non-verbal communication.

This paper surveys a collection of sample applications for networked user-centric context-aware embodied music listening. The applications have been designed and developed in the framework of the EU-ICT Project SAME (www.sameproject.eu) and have been presented at Agora Festival (IRCAM, Paris, France) in June 2009. All of them address in different ways the concept of embodied, active listening to music, i.e., enabling listeners to interactively operate in real-time on the music content by means of their movements and gestures as captured by mobile devices. In the occasion of the Agora Festival the applications have also been evaluated by both expert and non-expert users. This work has been partially supported by the EU-ICT Project SAME.

This paper describes the main novel developments which have been introduced and experimented with in "Viaggiatori di Sguardo", a permanent interactive installation. The objectives and resulting interaction design choices are briefly explained, and the system architecture briefly sketched. Then, the paper focuses on the novel full-body pointing interaction mechanism based on the mimicking of the use of binoculars. The permanent installation "Viaggiatori di Sguardo" was opened on December 2009 at Palazzo Ducale, Fondazione per la Cultura, Genova, and has been visited up to now by about 5,000 visitors. Results from the evaluation of such a novel interface are presented, and future developments briefly discussed.

In this paper we describe the SAME networked platform for context-aware, experience-centric mobile music applications, and we present an implementation of the SAME active music listening paradigm: the Mobile Conductor. It allows the user to express herself in conducting a virtual ensemble playing a MIDI piece of music by means of her mobile phone. The mobile phone detects the user’s hand movement and molds the music performance style by modulating its speed, volume, and intonation.

In this paper we present a survey of our research on analysis of expressive gesture and how it is evolving towards the analysis of expressive social interaction in groups of users. Social interaction and its expressive implications (e.g., emotional contagion, empathy) is an extremely relevant component for analysis of expressive gesture, since it provides significant information on the context expressive gestures are performed in. However, most of the current systems analyze expressive gestures according to basic emotion categories or simple dimensional approaches. Moreover, almost all of them are intended for a single user, whereas social interaction is often neglected. After briefly recalling our pioneering studies on collaborative robot-human interaction, this paper presents two steps in the direction of novel computational models and techniques for measuring social interaction: (i) the interactive installation Mappe per Affetti Erranti for active listening to sound and music content, and (ii) the techniques we developed for explicitly measuring synchronization within a group of users. We conclude with the research challenges we will face in the near future.

This paper introduces a software architecture for the representation and real-time processing of sound and music, based on Artificial Intelligence techniques. In particular, a system able to store and process music and sounds at different levels of abstraction, and to carry out plans for manipulating this material according to the user's goals is described. This system is able to generate new music pieces as well as manipulate existent ones, on the basis of given material: to this end, it provides also formal analysis capabilities on both music and sounds for extracting information useful in subsequent synthesis processes.

This research is concerned with synchronization (or entrainment), and in particular on emotional synchronization, which is likely to be an important driver of collaborative processes. We focus on ensemble musical performance, an ideal test-bed for the development of models and techniques for measuring creative social interaction in an ecologically valid framework. Ongoing work and early results on the automated analysis in real-time of non-verbal cues related to expressive gesture in social interaction is briefly presented.

Tangible Acoustic Interfaces (TAIs) exploit the propagation of sound in physical objects in order to localize touching positions and to analyse user's gesture on the object. Designing and developing TAIs consists of exploring how physical objects, augmented surfaces, and spaces can be transformed into tangible-acoustic embodiments of natural seamless unrestricted interfaces. Our research focuses on Expressive TAIs , i.e., TAIs able at processing expressive user's gesture and providing users with natural multimodal interfaces that fully exploit expressive, emotional content. This paper presents a concrete example of analysis of expressive gesture in TAIs: hand gestures on a TAI surface are classified according to the Space and Time dimensions of Rudolf Laban's Theory of Effort. Research started in the EU-IST Project TAI-CHI (Tangible Acoustic Interfaces for Computer-Human Interaction) and is currently going on in the EU-ICT Project SAME (Sound and Music for Everyone, Everyday, Everywhere, Every way, www.sameproject.eu). Expressive gesture analysis and multimodal and cross-modal processing are achieved in the new EyesWeb XMI open platform (available at www.eyesweb.org) by means of a new version of the EyesWeb Expressive Gesture Processing Library.

Research on HCI and multimedia systems for art and entertainment based on non-verbal, full-body, emotional and social interaction is the main topic of this paper. A short review of previous research projects in this area at our centre are presented, to introduce the main issues discussed in the paper. In particular, a case study based on novel paradigms of social active music listening is presented. Active music listening experience enables users to dynamically mould expressive performance of music and of audiovisual content. This research is partially supported by the 7FP EU-ICT Project SAME (Sound and Music for Everyone, Everyday, Everywhere, Every Way, www.sameproject.eu).

This paper presents a number of algorithms andrelated software modules for the automated analysisin real-time of non-verbal cues related to expressivegesture in social interaction. Analysis is based onPhase Synchronisation (PS) and RecurrenceQuantification Analysis. We start from the hypothesisthat PS is one of the low-level social signalsexplaining empathy and dominance in a small groupof users. Real- time implementation of the algorithmsis available in the EyesWeb XMI Social SignalProcessing Library. Applications of the approach isin user-centric media and in particular in activemusic listening (EU ICT FP7 Project SAME).

HARP (Hybrid Action Representation and Planning), a prototype high-level system for computer-aided composition, is introduced. HARP represents and manipulates music knowledge using a twofold formalism. An object-oriented concurrent environment, the analogical subsystem, provides procedures to manage the sound itself (samples, codes, and algorithms) and particular analysis processes. The symbolic subsystem, a declarative symbolic environment, stores higher level scores, composition rules, definitions in general, and descriptions of pieces of music. The symbolic subsystem is based on a multiple-inheritance semantic network formalism derived from KL-One. The HARP knowledge base, system interface, and reasoning mechanisms are discussed. An important set of analogical descriptions in HARP, based on the metaphor of force fields, is examined. >

This work aims at defining a computational model of human emotional entrainment. Music, as a non-verbal language to express emotions, is chosen as an ideal test bed for these aims. We start from multimodal gesture and motion signals, recorded in a real world collaborative condition in an ecological setting. Four violin players were asked to play, alone or in duo, a music fragment in two different perceptual feedback modalities and in four different emotional states. We focused our attention on phase synchronisation of the head motions of the players. From observation by subjects (musicians and observers), an evidence of entrainment emerges between players. The preliminary results, based on a reduced data set, however do not grasp fully this phenomenon. A more extended analysis is current subject of investigation.

This paper illustrates our recent work on the analysis of expressive gesture related to the motion of the upper body (the head and the hands) in the context of emotional portrayals performed by professional actors. An experiment is presented which is the result of a multidisciplinary joint work. The experiment aims at (i) developing models and algorithms for analysis of such expressive content (ii) individuating which motion cues are involved in conveying the actorpsilas expressive intentions to portray four emotions (anger, joy, relief, sadness) via a scenario approach. The paper discusses the experiment in detail with reference to related conceptual issues, developed techniques, and the obtained results.

With active experience and active listening we mean a novel approach to music listening, where listeners are enabled (interactively and in real-time) to access, retrieve, and operate on music content, including the possibility to modify and mould such content. Active listening is the basic concept for a novel generation of interactive music systems, which are particularly addressed to the general users. A key focus is on content personalization and context awareness in the framework of mobile music applications. This paper discusses the research challenges that need to be faced for developing systems and devices for active listening in shared, distributed, and social collaborative environments. It also illustrates a first concrete application demonstrated in a multimodal interactive dance/music installation ldquoMappe per Affetti Errantirdquo, that we presented at our site in Casa Paganini, Genova, Italy, in the framework of ldquoFestival delta Scienzardquo, an international science exhibit held in October November 2007. The work here described includes some novel improvements emerging from this public testing. Research on active listening is carried out in the EU-ICT Project SAME (Sound and Music for Everyone, Everyday, Everywhere, Every Way, www.sameproject.eu).

In this paper, we present a new system, the Orchestra Explorer, enabling a novel paradigm for active fruition of sound and music content. The Orchestra Explorer allows users to physically navigate inside a virtual orchestra, to actively explore the music piece the orchestra is playing, to modify and mold the sound and music content in real-time through their expressive full-body movement and gesture. An implementation of the Orchestra Explorer was developed and presented in the framework of the science exhibition "Cimenti di Invenzione e Armonia", held at Casa Paganini, Genova, from October 2006 to January 2007.

In this paper we describe a system which allows users to use their full-body for controlling in real-time the generation of an expressive audio-visual feedback. The system extracts expressive motion features from the user's full-body movements and gestures. The values of these motion features are mapped both onto acoustic parameters for the real-time expressive rendering of a piece of music, and onto real-time generated visual feedback projected on a screen in front of the user.