More than words: The online orchestration of word predictability, prosody, gesture, and mouth movements during natural language comprehension

Communication naturally occurs in dynamic face-to-face environments where spoken words are embedded in linguistic discourse and accompanied by multimodal cues. Existing research supports predictive brain models where prior discourse but also prosody, mouth movements, and hand-gestures individually contribute to comprehension. In electroencephalography (EEG) studies, more predictable words show reduced negativity, peaking at approximately 400ms after onset of the word (N400). Meaningful gestures and prosody also reduce the N400, while the effects of rhythmic gestures and mouth movements remain unclear. However, these studies have only focused on individual cues while in the real world, communicative cues co-occur and potentially interact. We measured EEG elicited by words while participants watched videos of a speaker producing short naturalistic passages. For each word, we quantified the information carried by prior linguistic discourse (surprisal), prosody (mean pitch), mouth informativeness (lip-reading), and presence of meaningful and/or rhythmic gestures. Discourse predictability reduced the N400 amplitude and multimodal cues impacted and interacted with this effect. Specifically, higher pitch and meaningful gestures reduced N400 amplitude, beyond the effect of discourse, while rhythmic gestures increased the N400 amplitude. Moreover, higher pitch overall reduced N400 amplitude while informative mouth movements only showed effects when no gestures were present. These results can constrain existing predictive brain models in that they demonstrate that the brain uses cues selectively in a dynamic and contextually determined manner in real-world language comprehension. Significance Statement Language originated, is learnt and is mostly used in face-to-face contexts comprising linguistic information and multimodal cues such as intonation of the speech, meaningful/rhythmic hand gestures and mouth movements. Yet, the rich multimodal context of real-world language use is usually stripped away in language studies. Here, we developed quantifications of the visual and acoustic non-linguistic cues accompanying speech and we investigate how, jointly, the cues shape brain activity. We find that the brain activity associated with processing is always modulated by the multimodal non-linguistic cues and their interactions. Thus, this work questions the validity of language comprehension models only based on speech. It further presents new methods for the study of real-world language.