Search

Beyond the words a speaker produces, meaning can be recovered from the many ways in which the words are delivered. During everyday discourse, the way in which a speaker produces an utterance - both in their spoken delivery and accompanying movements | is an important part of the communication process. This thesis investigates the ways in which meaning is carried within these nonlinguistic behaviours, focussing on how they may provide signals about upcoming message content and about speakers intentions. Previous research has shown that the manner of spoken delivery (for instance, rates of speech, intonation, and fluency) influences listeners content-based expectations as well as pragmatic comprehension. These effects have been evidenced both post-hoc and during the moment-to-moment processing of speech. Considerably less attention has been paid on whether speakers movements and non-verbal behaviours have similar effects: Research on gesture has tended to focus on the content represented by gesture (rather than its potential to signal information about the message and/or speaker). We focus on two ways in which non-linguistic behaviours influence comprehension: Firstly, as signals of speech planning difficulty (and so of upcoming content), and secondly as indicators of a speakers intention to deceive. The former has been studied in relation to speech disfluency, but not to gestures. The latter has been studied extensively in relation to many linguistic and non-linguistic behaviours, but has only recently begun to be addressed with respect to the time course of the process. Focussing on these two areas, we address the broader question about the perceptual relevance of non-linguistic signals in comprehension through a dialogue study and a series of comprehension experiments combining eye and mouse tracking techniques. Extending the VisualWorld Paradigm|commonly used in comprehension studies - to include a video component showing the speaker, we measure listeners' eye movements and mouse coordinates as they select objects in the on-screen display. By directly manipulating the presence of different non-linguistic behaviours, we investigate whether and when these behaviours are interpreted as signals of 1) upcoming difficulty in speech, and 2) the speaker's intention to deceive. Our results demonstrate that, like for speech disfluency, listeners interpret the presence of representational gesturing to inform explicit predictions about upcoming referents. This follows from the findings from our dialogue study which suggest that speakers produce more of this type of gesturing relative to speech when describing shapes which are more conceptually difficult. We also show that listeners reliably perceive certain motoric behaviours of a speaker to indicate deception, and that the influence these behavioural cues have on listeners' interpretation can be detected alongside the unfolding linguistic input. Non-linguistic cues to perceived deception are found to be robust to contexts where speakers produce a variety of cues in different modalities. However, findings point to differences in how listeners link visual and spoken cues with deception. Results indicate that when cues are present in both modalities, visual cues tend to drive listeners' biases towards interpreting an utterance as dishonest. The time course supports a view in which non-linguistic cues influence pragmatic comprehension at an early stage, and suggests that linking visual cues with deception may be more resource demanding than it is for spoken cues. Listeners' associations between the presence of certain non-linguistic behaviours and judgements of deception also hold in situations where there are other alternative explanations for a given cue. However, the availability of an alternative explanations for a given non-linguistic behaviour are found to influence early stages of comprehension, suggesting that listeners may engage in dynamic reasoning about the possible causes of a speaker's manner of delivery. Taken together, the results from studies presented in this thesis highlight the role of both the spoken and visual delivery of an utterance in shaping comprehension, highlighting the fact that communication is fundamentally multi-modal.