Search

This file contains information about 1) full results; 2) participants and procedures; 3) materials; 4) establishing the effect of surprisal; 5) quantification of multimodal cues; 6) EEG recording and preprocessing; 7) LMER analysis

Communication naturally occurs in dynamic face-to-face environments where spoken words are embedded in linguistic discourse and accompanied by multimodal cues. Existing research supports predictive brain models where prior discourse but also prosody, mouth movements, and hand-gestures individually contribute to comprehension. In electroencephalography (EEG) studies, more predictable words show reduced negativity, peaking at approximately 400ms after onset of the word (N400). Meaningful gestures and prosody also reduce the N400, while the effects of rhythmic gestures and mouth movements remain unclear. However, these studies have only focused on individual cues while in the real world, communicative cues co-occur and potentially interact. We measured EEG elicited by words while participants watched videos of a speaker producing short naturalistic passages. For each word, we quantified the information carried by prior linguistic discourse (surprisal), prosody (mean pitch), mouth informativeness (lip-reading), and presence of meaningful and/or rhythmic gestures. Discourse predictability reduced the N400 amplitude and multimodal cues impacted and interacted with this effect. Specifically, higher pitch and meaningful gestures reduced N400 amplitude, beyond the effect of discourse, while rhythmic gestures increased the N400 amplitude. Moreover, higher pitch overall reduced N400 amplitude while informative mouth movements only showed effects when no gestures were present. These results can constrain existing predictive brain models in that they demonstrate that the brain uses cues selectively in a dynamic and contextually determined manner in real-world language comprehension. Significance Statement Language originated, is learnt and is mostly used in face-to-face contexts comprising linguistic information and multimodal cues such as intonation of the speech, meaningful/rhythmic hand gestures and mouth movements. Yet, the rich multimodal context of real-world language use is usually stripped away in language studies. Here, we developed quantifications of the visual and acoustic non-linguistic cues accompanying speech and we investigate how, jointly, the cues shape brain activity. We find that the brain activity associated with processing is always modulated by the multimodal non-linguistic cues and their interactions. Thus, this work questions the validity of language comprehension models only based on speech. It further presents new methods for the study of real-world language.

How is speech understood despite the lack of a deterministic relationship between the sounds reaching auditory cortex and what we perceive? One possibility is that unheard words that are unconsciously activated in association with listening context are used to constrain interpretation. We hypothesized that a mechanism for doing so involves reusing the ability of the brain to predict the sensory effects of speaking associated words. Predictions are then compared to signals arriving in auditory cortex, resulting in reduced processing demands when accurate. Indeed, we show that sensorimotor brain regions are more active prior to words predictable from listening context. This activity resembles lexical and speech production related processes and, specifically, subsequent but still unpresented words. When those words occur, auditory cortex activity is reduced, through feedback connectivity. In less predictive contexts, activity patterns and connectivity for the same words are markedly different. Results suggest that the brain reorganizes to actively use knowledge about context to construct the speech we hear, enabling rapid and accurate comprehension despite acoustic variability.

Functional magnetic resonance imaging (fMRI) studies of speech sound categorization often compare conditions in which a stimulus is presented repeatedly to conditions in which multiple stimuli are presented. This approach has established that a set of superior temporal and inferior parietal regions respond more strongly to conditions containing stimulus change. Here, we examine whether this contrast is driven by habituation to a repeating condition or by selective responding to change. Experiment 1 directly tests this by comparing the observed response to long trains of stimuli against a constructed hemodynamic response modeling the hypothesis that no habituation occurs. The results are consistent with the view that enhanced response to conditions involving phonemic variability reflect change detection. In a second experiment, the specificity of these responses to linguistically relevant stimulus variability was studied by including a condition in which the talker, rather than phonemic category, was variable from stimulus to stimulus. In this context, strong change detection responses were observed to changes in talker, but not to changes in phoneme category. The results prompt a reconsideration of two assumptions common to fMRI studies of speech sound categorization: they suggest that temporoparietal responses in passive paradigms such as those used here are better characterized as reflecting change detection than habituation, and that their apparent selectivity to speech sound categories may reflect a more general preference for variability in highly salient or behaviorally relevant stimulus dimensions.

Does “the motor system” play “a role” in speech perception? If so, where, how, and when? We conducted a systematic review that addresses these questions using both qualitative and quantitative methods. The qualitative review of behavioural, computational modelling, non-human animal, brain damage/disorder, electrical stimulation/recording, and neuroimaging research suggests that distributed brain regions involved in producing speech play specific, dynamic, and contextually determined roles in speech perception. The quantitative review employed region and network based neuroimaging meta-analyses and a novel text mining method to describe relative contributions of nodes in distributed brain networks. Supporting the qualitative review, results show a specific functional correspondence between regions involved in non-linguistic movement of the articulators, covertly and overtly producing speech, and the perception of both nonword and word sounds. This distributed set of cortical and subcortical speech production regions are ubiquitously active and form multiple networks whose topologies dynamically change with listening context. Results are inconsistent with motor and acoustic only models of speech perception and classical and contemporary dual-stream models of the organization of language and the brain. Instead, results are more consistent with complex network models in which multiple speech production related networks and subnetworks dynamically self-organize to constrain interpretation of indeterminant acoustic patterns as listening context requires.