Search

Human hand gestures are very swift and difficult to observe from the (often) distant perspective of a scientific overhearer. Not uncommonly, fingers are occluded by other body parts or context objects and the true hand posture is often only revealed to the addressee. In addition to that, as the hand has many degrees of freedom and the annotation has to cover positions and orientations in a 3D world – which is less accessible from the typical computer-desktop workplace of an annotator than, let’s say, spoken language – the annotation of hand postures is quite expensive and complex. Fortunately, the research on virtual reality technology has brought about data gloves in the first place, which were meant as an interaction device allowing humans to manipulate entities in a virtual world. Since its release, however, many different applications have been found. Data gloves are devices that track most of the joints of the human hand and generate data-sets describing the posture of the hand several times a second. The article reviews different types of data gloves, discusses representation formats and ways to support annotation, and presents best practices for study design using data gloves as recording devices.

For the scientific observation of non-verbal communication behavior, video recordings are the state of the art. However, everyone who has conducted at least one video-based study has probably made the experience, that it is difficult to get the setup right, with respect to image resolution, illumination, perspective, occlusions, etc. And even more effort is needed for the annotation of the data. Frequently, even short interaction sequences may consume weeks or even months of rigorous full-time annotations. One way to overcome some of these issues is the use of motion capturing for assessing (not only) communicative body movements. There are several competing tracking technologies available, each with its own benefits and drawbacks. The article provides an overview of the basic types of tracking systems, presents representation formats and tools for the analysis of motion data, provides pointers to some studies using motion capture and discusses best practices for study design. However, the article also stresses that motion capturing still requires some expertise and is only starting to become mobile and reasonably priced – arguments not to be neglected.

This chapter discusses and assesses the feasibility of operationalising Growth Point Theory's model of language production in embodied conversational agents (ECAs). First, the chapter outlines the cornerstones of non-modular Growth Point Theory and its empirical basis. It then gives an overview of gesture and speech production models that are currently realised in ECAs, and discuss their potential and limitations with respect to which characteristics of natural speech and gesture they can account for. Finally, it discusses which requirements a technical model must meet in order to be more compatible with Growth Point Theory.

Publication in the conference proceedings of EUSIPCO, Toulouse, France, 2002

Publication in the conference proceedings of EUSIPCO, Toulouse, France, 2002

The signature form of abstract deixis is the classic pointing G-hand aimed at absolutely nothing. This gesture is extremely widespread and frequent in conversational discourse.[2] A speaker says "and there was [this guy]", and flexes an index finger at a certain locus, although there is no person there. In effect, the gesture `creates' a reference, in this case an imaginary person identified as "this guy". What I am terming `abstract deixis' is, according to Bühler (1982), concrete deixis that has been transposed to the realm of imaginary referents. His term for this process was deixis at phantasma. As in all cases of pointing, abstract deixis entails a coordinate center, or origo, and an indicated line connecting this center to a locus (the G-hand is an iconic sign of this line). A `target' then is created at this locus that may be used for later anaphoric references. The creative use of pointing in abstract deixis fleshes out the deictic paradigm with new meanings for the target and origo cross-hairs. Another component of pointing is the deictic field or ground over which the pointing act is understood to extend. In the concrete case, the deictic field is spatial and ranges from the immediate perceptual present to vast regions such as the entire Yucatan peninsula that Haviland (in press) describes in acts of pointing to distant locales. In the abstract case, the ground takes on non-spatial meanings. The components of deixis are all part of the referential creation of abstract pointing, and can be analyzed as parameters set by the speaker for the context. Creative deictic use is prominent in the example that I will now present.

The chapters in this volume have generally accepted the argument that speech-gesture integration is basic to language use. But what explains the integration itself? I will attempt to make the case that it can be understood with the concept of a `growth point' or GP (McNeill & Duncan this volume) It is called a GP since it is a theoretical unit in which principles that explain mental growth -- differentiation, internalization, dialectic, and reorganization -- apply to realtime utterance generation by adults (and children). It is also called a GP since it is meant to be the initial form of a thinking-while-speaking unit out of which a dynamic process of organization emerges. The emergence unpacks the GP into a surface utterance and gesture that articulates its meaning implications.

Many bilingual speakers believe they engage in different forms of thinking when they shift languages. This experience of entering different thought worlds can be explained with the hypothesis that languages induce different forms of `thinking-for-speaking'-- thinking generated, as Slobin (1987) says, because of the requirements of a linguistic code. "`Thinking for speaking' involves picking those characteristics that (a) fit some conceptualization of the event, and (b) are readily encodable in the language"[2] (p. 435). That languages differ in their thinking-for-speaking demands is a version of the linguistic relativity hypothesis, the proposition that language influences thought and that different languages influence thought in different ways.

The article on body movements in robotics describes why body movements are a relevant object of interest for research on robots and categorizes different types of movement based on their functional purpose and intent. Following an overview of robot appearances and application areas, a focus is placed on robotic platforms that are recognized as “social robots” and which may come in anthropomorphic, zoomorphic, caricatured, or functional designs. Body movements are discussed with respect to locomotion, manipulative movement, and expressive movement, with an emphasis on movements that express communicative functions, such as gesture and facial expression. With respect to motion generation, off-line methods employing hardcoded action sequences or pre-recorded motions are contrasted with on-line methods which may be based on imitation learning or which transfer multimodal motion scheduling from virtual humanoid agents to humanoid robots. In particular, research challenges with regard to motor control for arbitrary, expressive hand-arm movement and its coordination with other interaction modalities are discussed.