Search

Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4-8 Hz) oscillations. Here, we investigated whether the theta modulation merely reflects mismatch processing or, alternatively, audiovisual integration of speech. We used electroencephalographic recordings from two previously published studies using audiovisual sine-wave speech (SWS), a spectrally degraded speech signal sounding nonsensical to naive perceivers but perceived as speech by informed subjects. Earlier studies have shown that informed, but not naive subjects integrate SWS phonetically with visual speech. In an N1/P2 event-related potential paradigm, we found a significant difference in theta-band activity between informed and naive perceivers of audiovisual speech, suggesting that audiovisual integration modulates induced theta-band oscillations. In a McGurk mismatch negativity paradigm (MMN) where infrequent McGurk stimuli were embedded in a sequence of frequent audio-visually congruent stimuli we found no difference between congruent and McGurk stimuli. The infrequent stimuli in this paradigm are violating both the general prediction of stimulus content, and that of audiovisual congruence. Hence, we found no support for the hypothesis that audiovisual mismatch modulates induced theta-band oscillations. We also did not find any effects of audiovisual integration in the MMN paradigm, possibly due to the experimental design.

During sentence production, linguistic information (semantics, syntax, phonology) of words is retrieved and assembled into a meaningful utterance. There is still debate on how we assemble single words into more complex syntactic structures such as noun phrases or sentences. In the present study, event-related potentials (ERPs) were used to investigate the time course of syntactic planning. Thirty-three volunteers described visually animated scenes using naming formats varying in syntactic complexity: from simple words ('W', e.g., "triangle", "red", "square", "green", "to fly towards"), to noun phrases ('NP', e.g., "the red triangle", "the green square", "to fly towards"), to a sentence ('S', e.g., "The red triangle flies towards the green square."). Behaviourally, we observed an increase in errors and corrections with increasing syntactic complexity, indicating a successful experimental manipulation. In the ERPs following scene onset, syntactic complexity variations were found in a P300-like component ('S'/'NP'>'W') and a fronto-central negativity (linear increase with syntactic complexity). In addition, the scene could display two actions - unpredictable for the participant, as the disambiguation occurred only later in the animation. Time-locked to the moment of visual disambiguation of the action and thus the verb, we observed another P300 component ('S'>'NP'/'W'). The data show for the first time evidence of sensitivity to syntactic planning within the P300 time window, time-locked to visual events critical of syntactic planning. We discuss the findings in the light of current syntactic planning views. © 2013 Timmers et al.

Background: Most people show a remarkable deficit to report the second of two targets when presented in close temporal succession, reflecting an attentional blink (AB). An aspect of the AB that is often ignored is that there are large individual differences in the magnitude of the effect. Here we exploit these individual differences to address a long-standing question: does attention to a visual target come at a cost for attention to an auditory target (and vice versa)? More specifically, the goal of the current study was to investigate a) whether individuals with a large within-modality AB also show a large cross-modal AB, and b) whether individual differences in AB magnitude within different modalities correlate or are completely separate.Methodology/Principal Findings: While minimizing differential task difficulty and chances for a task-switch to occur, a significant AB was observed when targets were both presented within the auditory or visual modality, and a positive correlation was found between individual within-modality AB magnitudes. However, neither a cross-modal AB nor a correlation between cross-modal and within-modality AB magnitudes was found.Conclusion/Significance: The results provide strong evidence that a major source of attentional restriction must lie in modality-specific sensory systems rather than a central amodal system, effectively settling a long-standing debate. Individuals with a large within-modality AB may be especially committed or focused in their processing of the first target, and to some extent that tendency to focus could cross modalities, reflected in the within-modality correlation. However, what they are focusing (resource allocation, blocking of processing) is strictly within-modality as it only affects the second target on within-modality trials. The findings show that individual differences in AB magnitude can provide important information about the modular structure of human cognition.