Search

It is well known that visual information derived from mouth movements (i.e., lipreading) can have profound effects on auditory speech identification (e.g. the McGurk-effect). Here we examined the reverse phenomenon, namely whether auditory speech affects lipreading. We report that speech sounds dubbed onto lipread speech affect immediate identification of lipread tokens. This effect likely reflects genuine cross-modal integration of sensory signals and not just a simple response bias because we also observed adaptive shifts in visual identification of the ambiguous lipread tokens after exposure to incongruent audiovisual adapter stimuli. Presumably, listeners had learned to label the lipread stimulus in accordance with the sound, thus demonstrating that the interaction between hearing and lipreading is genuinely bi-directional., ((c) 2010 Elsevier Ireland Ltd. All rights reserved.)

Here we demonstrate that the perceptual system adapts to tactile-visual temporal asynchronies (i.e., temporal recalibration). Participants were exposed to a train of tactile and visual stimuli with a constant time lag (either -100ms, 0ms, or 100ms; with negative values indicating that the tactile stimulus came first). Following exposure, they were presented tactile-visual test stimulus pairs and judged whether the tactile or the visual stimulus was presented first (Temporal Order Judgement). Results show that subjective simultaneity (the PSS) was shifted in the direction of the exposure lag. The results fit reports on auditory-visual temporal recalibration and indicate that the brain adapts to temporal incongruencies between modalities in general.

Lexical information can bias categorization of an ambiguous phoneme and subsequently evoke a shift in the phonetic boundary. Here, we explored the extent to which this phenomenon is perceptual in nature. Listeners were asked to ignore auditory stimuli presented in a typical oddball sequence in which the standard was an ambiguous sound halfway between /t/ and /p/ embedded in a Dutch word normally ending in /t/ ('vloot', meaning 'fleet') or /p/ ('hoop', meaning 'hope'). As deviant served the non-ambiguous sound /t/ embedded in the same context. The amplitude of the MMN-response, indexing the perceptual difference between the ambiguous sound and unambiguous /t/ was bigger for the p-word 'hoop' than the t-word 'vloot'. This result is taken as an indication that lexical information actually reached down to early perceptual processing stages.

The ventriloquist illusion arises when sounds are mislocated towards a synchronous but spatially discrepant visual event. Here, we investigated the ventriloquist illusion at a neurophysiological level. The question was whether an illusory shift in sound location was reflected in the auditory mismatch negativity (MMN). An 'oddball' paradigm was used whereby simultaneously presented sounds and flashes coming from the same location served as standard. The deviant consisted of a sound originating from the same source as the standard together with a flash at 20 degrees spatial separation, which evoked an illusory sound shift. This illusory sound shift evoked an MMN closely resembling the MMN evoked by an actual sound shift. A visual-only control condition ruled out that the illusory-evoked MMN was confounded by the visual part of the audiovisual deviant. These results indicate that the crossmodal interaction on which the ventriloquist illusion is based takes place automatically at an early processing stage, within 200 ms after stimulus onset.

Judgement of the emotional tone of a spoken utterance is influenced by a simultaneously presented face expression. The time course of this integration was investigated by measuring the mismatch negativity (MMN). In one condition, the standard stimulus was an angry voice fragment combined with a (congruous) angry face expression. In the deviant pair, the voice expression was kept the same and only the face expression changed to an (incongruous) sad face. The pairs with a deviant visual item evoked a negative electric brain response showing the characteristics of the MMN, which is usually evoked only by auditory deviations. Similar results were obtained by employing incongruous standard and congruous deviant pairs. These findings provide compelling evidence of an early integration of face with voice information in the processing of affect.