Search

Neurodevelopmental disorders (NDDs) are characterised by impairments in communication. Socially assistive robots have been identified as a promising avenue to alleviate their burden. Since NDDs have different needs, their way of communicating with a robot (e.g., speech-based) could differ among individuals. This paper aims to investigate the most suitable modality to communicate with a robot for NDDs - among voice, cards, and buttons - and explore their opinion on this matter. We ran an exploratory study involving 29 NDDs participants: 13 of them could freely communicate with an autonomous QT robot, 9 took part in a group discussion, and 7 first interacted individually with the robot, and then they participated in a group discussion. Our results showed that i) the cards were the most used communication modality, ii) voice can be used for counting games, buttons for multiple-choice games, and cards for memory-like games, iii) opinions did not differ much among groups. [ABSTRACT FROM AUTHOR]

The uninterruptible trend of developing high performance human-robot communication systems worldwide stimulates the research efforts focused to engine novel hardware configurations suitable for high performance multimodal communication abilities implementation. Being fully integrated in this major trend, the paper presents a specially conceived robotic configuration designed for human gestures and speech recognition abilities enhancement. Having the standard configuration of the NI SbRIO-9631 prototype robot, a novel multimodal communication structure has been developed by gradually extending this mobile device with voice recognition and real-time image processing units. The result is very flexible and powerful development tool for human-robot multimodal communication research and implementation. The hardware resources of this robot are wisely integrated then by using the LabView graphical software toolkit, enabling a large amount of voice signals and image processing algorithms testing. The designed software drivers also allow various sensor modality including vision and speech signals analyzing. Endowed with these novel capabilities, the mobile robot displays intelligence and cooperativeness in its behavior, and becomes a versatile hardware system for human-robot multimodal communication implementation. [ABSTRACT FROM PUBLISHER]

Discourse markers, a group of pragmatic elements, are used widely in our daily communication. Among these markers, well is found to be the most frequently used one in conversation. Although much research has been carried out on its theoretical bases such as its definition, functions and categorizations, few studies have taken a multimodal approach in analyzing this marker. The aim of the present study was to provide a multimodal description of the use of well as a discourse marker in spontaneous speech. Firstly, we wanted to investigate whether the use of this marker in conversation, regardless of its function, was accompanied with any kind of non-verbal behavior (posture shift, head shift, gaze and eyebrow movement) and secondly, we wanted to determine whether any of these non-verbal cues (if any) were specific to one pragmatic function of well and if they could be used as a signal to disambiguate the pragmatic function of well. The study of discourse markers is considered important due to the fact that it can contribute both to dialogue interpretation and generation resulting in a more natural dialogue modeling useful for human-infocommunications interface. [ABSTRACT FROM PUBLISHER]

Spoken dialogs represent a considerable challenge for language technology for at least two reasons: (a) due to the frequent incompleteness of syntactic structure a given sequence of words cannot often undergo satisfactory parsing, (b) as a result, the associated fragmented prosody may not be used to substitute for missing elements. However, spoken dialogs are essentially multimodal with nonverbal means having the potential of offering some particular structural cues as well as functional-pragmatic ones. An incomplete syntax should in principle break down automatic syntactic analysis, however, its combination with the output of other; non-verbal modalities may generate cues to some possible interpretation and decision making. An important challenge is to identify a proper complex of such aligning cues for a suitable decision making. In this context relying on larger amounts of data gleaned from longer stretches of communicative events may contribute to a more successful representation of the cognitive content behind similar instances of fragmented communication if these sets of data are based on a theoretically founded understanding of multimodal communication. The HuComTech multimodal corpus of 60 hours of spoken dialogues of Hungarian [1] consists of a multi-level annotation of both physical and interpretative (functional, pragmatic) aspects of communication. Annotation is not purely descriptive: it is based on a novel generative model of communication allowing for the capturing of the two-way, top down and bottom up character of communication with the purpose to identify the individual minimal building blocks of communication and to get a deeper understanding of how these components make up a complex communicative event as a whole. Based on these insights, a novel scheme of annotation for syntax, prosody as well as uni- and multimodal pragmatics is introduced aiming at describing and understanding the multimodal nature of human communication, cognitive info-communication in general, and, on the longer run, contributing to the further development of human-computer interaction systems. [ABSTRACT FROM PUBLISHER]

Spoken dialogue frameworks have traditionally been designed to handle a single stream of data — the speech signal. Research on human-human communication has been providing large evidence and quantifying the effects and the importance of a multitude of other multimodal nonverbal signals that people use in their communication, that shape and regulate their interaction. Driven by findings from multimodal human spoken interaction, and the advancements of capture devices and robotics and animation technologies, new possibilities are rising for the development of multimodal human-machine interaction that is more affective, social, and engaging. In such face-to-face interaction scenarios, dialogue systems can have a large set of signals at their disposal to infer context and enhance and regulate the interaction through the generation of verbal and nonverbal facial signals. This paper summarizes several design decision, and experiments that we have followed in attempts to build rich and fluent multimodal interactive systems using a newly developed hybrid robotic head called Furhat, and discuss issues and challenges that this effort is facing. [ABSTRACT FROM PUBLISHER]

To fully utilize the social potential of virtual environments, support for seamless and automatic integration of nonverbal communication is essential. In this paper we propose a conceptual and architectural design for a framework, which provides an integrated, flexible environment for the cooperation of heterogeneous modules specialized in different aspects of the acquirement, analysis and presentation of nonverbal communication cues. We propose a semantic framework for the knowledge exchange between these modules, as well as a general purpose architecture for the semantically correct fusion of nonverbal features. [ABSTRACT FROM PUBLISHER]