Incompleteness and fragmentation in spoken language syntax and its relation to prosody and gesturing: Cognitive processes vs. Possible formal cues.

Spoken dialogs represent a considerable challenge for language technology for at least two reasons: (a) due to the frequent incompleteness of syntactic structure a given sequence of words cannot often undergo satisfactory parsing, (b) as a result, the associated fragmented prosody may not be used to substitute for missing elements. However, spoken dialogs are essentially multimodal with nonverbal means having the potential of offering some particular structural cues as well as functional-pragmatic ones. An incomplete syntax should in principle break down automatic syntactic analysis, however, its combination with the output of other; non-verbal modalities may generate cues to some possible interpretation and decision making. An important challenge is to identify a proper complex of such aligning cues for a suitable decision making. In this context relying on larger amounts of data gleaned from longer stretches of communicative events may contribute to a more successful representation of the cognitive content behind similar instances of fragmented communication if these sets of data are based on a theoretically founded understanding of multimodal communication. The HuComTech multimodal corpus of 60 hours of spoken dialogues of Hungarian [1] consists of a multi-level annotation of both physical and interpretative (functional, pragmatic) aspects of communication. Annotation is not purely descriptive: it is based on a novel generative model of communication allowing for the capturing of the two-way, top down and bottom up character of communication with the purpose to identify the individual minimal building blocks of communication and to get a deeper understanding of how these components make up a complex communicative event as a whole. Based on these insights, a novel scheme of annotation for syntax, prosody as well as uni- and multimodal pragmatics is introduced aiming at describing and understanding the multimodal nature of human communication, cognitive info-communication in general, and, on the longer run, contributing to the further development of human-computer interaction systems. [ABSTRACT FROM PUBLISHER]