Search

During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively., (Copyright © 2017 Elsevier Inc. All rights reserved.)

Neuroimaging studies have shown that seeing others in pain activates brain regions that are involved in first-hand pain, suggesting that shared neuromolecular pathways support processing of first-hand and vicarious pain. We tested whether the dopamine and opioid neurotransmitter systems involved in nociceptive processing also contribute to vicarious pain experience. We used in vivo positron emission tomography to quantify type 2 dopamine and μ-opioid receptor (D2R and MOR, respectively) availabilities in brains of 35 subjects. During functional magnetic resonance imaging, the subjects watched short movie clips depicting persons in painful and painless situations. Painful scenes activated pain-responsive brain regions including anterior insulae, thalamus and secondary somatosensory cortices, as well as posterior superior temporal sulci. MOR availability correlated negatively with the haemodynamic responses during painful scenes in anterior and posterior insulae, thalamus, secondary and primary somatosensory cortices, primary motor cortex, and superior temporal sulci. MOR availability correlated positively with orbitofrontal haemodynamic responses during painful scenes. D2R availability was not correlated with the haemodynamic responses in any brain region. These results suggest that the opioid system contributes to neural processing of vicarious pain, and that interindividual differences in opioidergic system could explain why some individuals react more strongly than others to seeing pain., (© The Author 2017. Published by Oxford University Press.)

Spatial and non-spatial information of sound events is presumably processed in parallel auditory cortex (AC) "what" and "where" streams, which are modulated by inputs from the respective visual-cortex subsystems. How these parallel processes are integrated to perceptual objects that remain stable across time and the source agent's movements is unknown. We recorded magneto- and electroencephalography (MEG/EEG) data while subjects viewed animated video clips featuring two audiovisual objects, a black cat and a gray cat. Adaptor-probe events were either linked to the same object (the black cat meowed twice in a row in the same location) or included a visually conveyed identity change (the black and then the gray cat meowed with identical voices in the same location). In addition to effects in visual (including fusiform, middle temporal or MT areas) and frontoparietal association areas, the visually conveyed object-identity change was associated with a release from adaptation of early (50-150ms) activity in posterior ACs, spreading to left anterior ACs at 250-450ms in our combined MEG/EEG source estimates. Repetition of events belonging to the same object resulted in increased theta-band (4-8Hz) synchronization within the "what" and "where" pathways (e.g., between anterior AC and fusiform areas). In contrast, the visually conveyed identity changes resulted in distributed synchronization at higher frequencies (alpha and beta bands, 8-32Hz) across different auditory, visual, and association areas. The results suggest that sound events become initially linked to perceptual objects in posterior AC, followed by modulations of representations in anterior AC. Hierarchical what and where pathways seem to operate in parallel after repeating audiovisual associations, whereas the resetting of such associations engages a distributed network across auditory, visual, and multisensory areas., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Functional brain imaging has identified specialized neural systems supporting human body perception. Responses to nude vs. clothed bodies within this system are amplified. However, it remains unresolved whether nude and clothed bodies are processed by same cerebral networks or whether processing of nude bodies recruits additional affective and arousal processing areas. We recorded simultaneous MEG and EEG while participants viewed photographs of clothed and nude bodies. Global field power revealed a peak ∼145ms after stimulus onset to both clothed and nude bodies, and ∼205ms exclusively to nude bodies. Nude-body-sensitive responses were centered first (100-200ms) in the extrastriate and fusiform body areas, and subsequently (200-300ms) in affective-motivational areas including insula and anterior cingulate cortex. We conclude that visibility of sexual features facilitates early cortical processing of human bodies, the purpose of which is presumably to trigger sexual behavior and ultimately ensure reproduction., (Copyright © 2015 Elsevier B.V. All rights reserved.)

Understanding how the brain processes stimuli in a rich natural environment is a fundamental goal of neuroscience. Here, we showed a feature film to 10 healthy volunteers during functional magnetic resonance imaging (fMRI) of hemodynamic brain activity. We then annotated auditory and visual features of the motion picture to inform analysis of the hemodynamic data. The annotations were fitted to both voxel-wise data and brain network time courses extracted by independent component analysis (ICA). Auditory annotations correlated with two independent components (IC) disclosing two functional networks, one responding to variety of auditory stimulation and another responding preferentially to speech but parts of the network also responding to non-verbal communication. Visual feature annotations correlated with four ICs delineating visual areas according to their sensitivity to different visual stimulus features. In comparison, a separate voxel-wise general linear model based analysis disclosed brain areas preferentially responding to sound energy, speech, music, visual contrast edges, body motion and hand motion which largely overlapped the results revealed by ICA. Differences between the results of IC- and voxel-based analyses demonstrate that thorough analysis of voxel time courses is important for understanding the activity of specific sub-areas of the functional networks, while ICA is a valuable tool for revealing novel information about functional connectivity which need not be explained by the predefined model. Our results encourage the use of naturalistic stimuli and tasks in cognitive neuroimaging to study how the brain processes stimuli in rich natural environments.

Individuals with Asperger syndrome (AS) have problems in following conversation, especially in the situations where several people are talking. This might result from impairments in audiovisual speech perception, especially from difficulties in focusing attention to speech-relevant visual information and ignoring distracting information. We studied the effect of visual spatial attention on the audiovisual speech perception of adult individuals with AS and matched control participants. Two faces were presented side by side, one uttering /aka/ and the other /ata/, while an auditory stimulus of /apa/ was played. The participants fixated on a central cross and directed their attention to the face that an arrow pointed to, reporting which consonant they heard. We hypothesized that the adults with AS would be more distracted by a competing talking face than the controls. Instead, they were able to covertly attend to the talking face, and they were as distracted by a competing face as the controls. Independently of the attentional effect, there was a qualitative difference in audiovisual speech perception: when the visual articulation was /aka/, the control participants heard /aka/ almost exclusively, while the participants with AS heard frequently /ata/. This finding may relate to difficulties in face-to-face communication in AS.

Audiovisual speech perception has been considered to operate independent of sound location, since the McGurk effect (altered auditory speech perception caused by conflicting visual speech) has been shown to be unaffected by whether speech sounds are presented in the same or different location as a talking face. Here we show that sound location effects arise with manipulation of spatial attention. Sounds were presented from loudspeakers in five locations: the centre (location of the talking face) and 45°/90° to the left/right. Auditory spatial attention was focused on a location by presenting the majority (90%) of sounds from this location. In Experiment 1, the majority of sounds emanated from the centre, and the McGurk effect was enhanced there. In Experiment 2, the major location was 90° to the left, causing the McGurk effect to be stronger on the left and centre than on the right. Under control conditions, when sounds were presented with equal probability from all locations, the McGurk effect tended to be stronger for sounds emanating from the centre, but this tendency was not reliable. Additionally, reaction times were the shortest for a congruent audiovisual stimulus, and this was the case independent of location. Our main finding is that sound location can modulate audiovisual speech perception, and that spatial attention plays a role in this modulation.

The superior temporal sulcus has been suggested to play a significant role in the integration of auditory and visual sensory information. Here, we presented vowels and short video clips of the corresponding articulatory gestures to healthy adult humans with two auditory-visual stimulus intervals during sparse sampling 3-T functional magnetic resonance imaging to detect which brain areas are sensitive to synchrony of speech sounds and associated articulatory gestures. The upper bank of the left middle superior temporal sulcus showed stronger activation during naturally asynchronous stimulation than during simultaneous stimulus presentation. It is possible that this reflects sensitivity of the left middle superior temporal sulcus to temporal synchrony of audio-visual speech stimuli.

Emotion plays a significant role in goal-directed behavior, yet its neural basis is yet poorly understood. In several psychological models the cardinal dimensions that characterize the emotion space are considered to be valence and arousal. Here 3T functional magnetic resonance imaging (fMRI) was used to reveal brain areas that show valence- and arousal-dependent blood oxygen level dependent (BOLD) signal responses. Seventeen healthy adults viewed pictures from the International Affective Picture System (IAPS) for brief 100 ms periods in a block design paradigm. In many brain regions BOLD signals correlated significantly positively with valence ratings of unpleasant pictures. Interestingly, partly in the same regions but also in several other regions BOLD signals correlated negatively with valence ratings of pleasant pictures. Therefore, there were several areas where the correlation across all pictures was of inverted U-shape. Such correlations were found bilaterally in the dorsolateral prefrontal cortex (DLPFC), dorsomedial prefrontal cortex (DMPFC) extending to anterior cingulate cortex (ACC), and insula. Self-rated arousal of those pictures which were evaluated to be unpleasant correlated with BOLD signal in the ACC, whereas for pleasant pictures arousal correlated positively with the BOLD signal strength in the right substantia innominata. We interpret our results to suggest a major division of brain mechanisms underlying affective behavior to those evaluating stimuli to be pleasant or unpleasant. This is consistent with the basic division of behavior to approach and withdrawal, where differentiation of hostile and hospitable stimuli is crucial., ((c) 2009 Wiley-Liss, Inc.)

This paper assessed the neural systems involved in processing of dynamic facial expressions in adolescents. The processing of facial expressions changes as a function of age, and it is thus important to understand how healthy adolescent subjects process dynamic facial expressions prior to analyzing disease-related changes. We hypothesized that viewing of dynamic facial expressions with opposing valences (happy vs. fearful) induces differential activations and deactivations in the brain. 27 healthy adolescents (9 female, 18 male, mean age = 14.5 years; age range 11.6-17.3 years) were examined by using the ASSQ and K-SADS-PL and scanned with 1.5-T fMRI during viewing of dynamic facial expressions and mosaic control images. The stimuli activated the same areas as previously seen in dynamic facial expression in adults. Our results indicated that opposing-valence dynamic facial expressions had differential effects on many cortical structures but not on subcortical limbic structures. The mirror neuron system is activated more during viewing of fearful compared to happy expressions in bilateral inferior frontal gyrus (IFG) and superior temporal sulcus (STS) left dominantly. We also detected more deactivation in the ventral anterior cingulate gyrus (ACG), showing more automated attentional processing of fearful expressions during passive viewing. Females were found to deactivate the right frontal pole more than male adolescents during happy facial expressions, while there were no differences in fear processing between genders. No clear gender or age effects were detected. In conclusion fear induces stronger responses in attention and mirror neurons probably related to fear contagion.

Watching the lips of a speaker enhances speech perception. At the same time, the 100 ms response to speech sounds is suppressed in the observer's auditory cortex. Here, we used whole-scalp 306-channel magnetoencephalography (MEG) to study whether lipreading modulates human auditory processing already at the level of the most elementary sound features, i.e., pure tones. We further envisioned the temporal dynamics of the suppression to tell whether the effect is driven by top-down influences. Nineteen subjects were presented with 50 ms tones spanning six octaves (125-8000 Hz) (1) during "lipreading," i.e., when they watched video clips of silent articulations of Finnish vowels /a/, /i/, /o/, and /y/, and reacted to vowels presented twice in a row; (2) during a visual control task; (3) during a still-face passive control condition; and (4) in a separate experiment with a subset of nine subjects, during covert production of the same vowels. Auditory-cortex 100 ms responses (N100m) were equally suppressed in the lipreading and covert-speech-production tasks compared with the visual control and baseline tasks; the effects involved all frequencies and were most prominent in the left hemisphere. Responses to tones presented at different times with respect to the onset of the visual articulation showed significantly increased N100m suppression immediately after the articulatory gesture. These findings suggest that the lipreading-related suppression in the auditory cortex is caused by top-down influences, possibly by an efference copy from the speech-production system, generated during both own speech and lipreading.

Selective auditory attention powerfully modulates neural activity in the human auditory cortex (AC). In contrast, the role of attention in subcortical auditory processing is not well established. Here, we used functional MRI (fMRI) to examine activation of the human inferior colliculus (IC) during strictly controlled auditory attention tasks. The IC is an obligatory midbrain nucleus of the ascending auditory pathway with diverse internal and external connections. The IC also receives a massive descending projection from the AC, suggesting that cortical processes affect IC operations. In this study, 21 subjects selectively attended to left-ear or right-ear sounds and ignored sounds delivered to the other ear. IC activations depended on the direction of attention, indicating that auditory processing in the human IC is not only determined by acoustic input but also by the current behavioral goals.

We used whole-head magnetoencephalograpy (MEG) to record changes in neuromagnetic N100m responses generated in the left and right auditory cortex as a function of the match between visual and auditory speech signals. Stimuli were auditory-only (AO) and auditory-visual (AV) presentations of /pi/, /ti/ and /vi/. Three types of intensity matched auditory stimuli were used: intact speech (Normal), frequency band filtered speech (Band) and speech-shaped white noise (Noise). The behavioural task was to detect the /vi/ syllables which comprised 12% of stimuli. N100m responses were measured to averaged /pi/ and /ti/ stimuli. Behavioural data showed that identification of the stimuli was faster and more accurate for Normal than for Band stimuli, and for Band than for Noise stimuli. Reaction times were faster for AV than AO stimuli. MEG data showed that in the left hemisphere, N100m to both AO and AV stimuli was largest for the Normal, smaller for Band and smallest for Noise stimuli. In the right hemisphere, Normal and Band AO stimuli elicited N100m responses of quite similar amplitudes, but N100m amplitude to Noise was about half of that. There was a reduction in N100m for the AV compared to the AO conditions. The size of this reduction for each stimulus type was same in the left hemisphere but graded in the right (being largest to the Normal, smaller to the Band and smallest to the Noise stimuli). The N100m decrease for the Normal stimuli was significantly larger in the right than in the left hemisphere. We suggest that the effect of processing visual speech seen in the right hemisphere likely reflects suppression of the auditory response based on AV cues for place of articulation.

Musical training is known to modify cortical organization. Here, we show that such modifications extend to subcortical sensory structures and generalize to processing of speech. Musicians had earlier and larger brainstem responses than nonmusician controls to both speech and music stimuli presented in auditory and audiovisual conditions, evident as early as 10 ms after acoustic onset. Phase-locking to stimulus periodicity, which likely underlies perception of pitch, was enhanced in musicians and strongly correlated with length of musical practice. In addition, viewing videos of speech (lip-reading) and music (instrument being played) enhanced temporal and frequency encoding in the auditory brainstem, particularly in musicians. These findings demonstrate practice-related changes in the early sensory encoding of auditory and audiovisual information.

Coherent perception of objects in our environment often requires perceptual integration of auditory and visual information. Recent behavioral data suggest that audiovisual integration depends on attention. The current study investigated the neural basis of audiovisual integration using 3-Tesla functional magnetic resonance imaging (fMRI) in 12 healthy volunteers during attention to auditory or visual features, or audiovisual feature combinations of abstract stimuli (simultaneous harmonic sounds and colored circles). Audiovisual attention was found to modulate activity in the same frontal, temporal, parietal and occipital cortical regions as auditory and visual attention. In addition, attention to audiovisual feature combinations produced stronger activity in the superior temporal cortices than attention to only auditory or visual features. These modality-specific areas might be involved in attention-dependent perceptual binding of synchronous auditory and visual events into coherent audiovisual objects. Furthermore, the modality-specific temporal auditory and occipital visual cortical areas showed attention-related modulations during both auditory and visual attention tasks. This result supports the proposal that attention to stimuli in one modality can spread to encompass synchronously presented stimuli in another modality.

Observing a speaker's articulatory gestures can contribute considerably to auditory speech perception. At the level of neural events, seen articulatory gestures can modify auditory cortex responses to speech sounds and modulate auditory cortex activity also in the absence of heard speech. However, possible effects of attention on this modulation have remained unclear. To investigate the effect of attention on visual speech-induced auditory cortex activity, we scanned 10 healthy volunteers with functional magnetic resonance imaging (fMRI) at 3 T during simultaneous presentation of visual speech gestures and moving geometrical forms, with the instruction to either focus on or ignore the seen articulations. Secondary auditory cortex areas in the bilateral posterior superior temporal gyrus and planum temporale were active both when the articulatory gestures were ignored and when they were attended to. However, attention to visual speech gestures enhanced activity in the left planum temporale compared to the situation when the subjects saw identical stimuli but engaged in a nonspeech motion discrimination task. These findings suggest that attention to visually perceived speech gestures modulates auditory cortex function and that this modulation takes place at a hierarchically relatively early processing level., (Copyright 2005 Wiley-Liss, Inc.)

We presented phonetically matching and conflicting audiovisual vowels to 10 dyslexic and 10 fluent-reading young adults during "clustered volume acquisition" functional magnetic resonance imaging (fMRI) at 3 T. We further assessed co-variation between the dyslexic readers' phonological processing abilities, as indexed by neuropsychological test scores, and BOLD signal change within the visual cortex, auditory cortex, and Broca's area. Both dyslexic and fluent readers showed increased activation during observation of phonetically conflicting compared to matching vowels within the classical motor speech regions (Broca's area and the left premotor cortex), this activation difference being more extensive and bilateral in the dyslexic group. The between-group activation difference in conflicting > matching contrast reached significance in the motor speech regions and in the left inferior parietal lobule, with dyslexic readers exhibiting stronger activation than fluent readers. The dyslexic readers' BOLD signal change co-varied with their phonological processing abilities within the visual cortex and Broca's area, and to a lesser extent within the auditory cortex. We suggest these findings as reflecting dyslexic readers' greater use of motor-articulatory and visual strategies during phonetic processing of audiovisual speech, possibly to compensate for their difficulties in auditory speech perception.

Afferent auditory processing in the human brainstem is often assumed to be determined by acoustic stimulus features alone and immune to stimulation by other senses or cognitive factors. In contrast, we show that lipreading during speech perception influences early acoustic processing. Event-related brainstem potentials were recorded from ten healthy adults to concordant (acoustic-visual match), conflicting (acoustic-visual mismatch) and unimodal stimuli. Audiovisual (AV) interactions occurred as early as approximately 11 ms post-acoustic stimulation and persisted for the first 30 ms of the response. Furthermore, the magnitude of interaction depended on AV pairings. These findings indicate considerable plasticity in early auditory processing.

Maximum likelihood models of multisensory integration are theoretically attractive because the goals and assumptions of sensory information processing are explicitly stated in such optimal models. When subjects perceive stimuli categorically, as opposed to on a continuous scale, Maximum Likelihood Integration (MLI) can occur before or after categorization-early or late. We introduce early MLI and apply it to the audiovisual perception of rapid beeps and flashes. We compare it to late MLI and show that early MLI is a better fitting and more parsimonious model. We also show that early MLI is better able to account for the effects of information reliability, modality appropriateness and intermodal attention which affect multisensory perception.

We studied the modification of auditory perception in three different conditions in twenty subjects. Observing other person's discordant articulatory gestures deteriorated identification of acoustic speech stimuli and modified the auditory percept, causing a strong McGurk effect. A similar effect was found when the subjects watched their own silent articulation in a mirror and acoustic stimuli were simultaneously presented to their ears. Interestingly, a smaller but significant effect was even obtained when the subjects just silently articulated the syllables without visual feedback. On the other hand, observing other person's or one's own concordant articulation and silently articulating a concordant syllable improved identification of the acoustic stimuli. The modification of auditory percepts caused by visual observation of speech and silently articulating it are both suggested to be due to the alteration of activity in the auditory cortex. Our findings support the idea of a close relationship between speech perception and production.

Information processing in auditory and visual modalities interacts in many circumstances. Spatially and temporally coincident acoustic and visual information are often bound together to form multisensory percepts [B.E. Stein, M.A. Meredith, The Merging of the Senses, A Bradford Book, Cambridge, MA, (1993), 211 pp.; Psychol. Bull. 88 (1980) 638]. Shams et al. recently reported a multisensory fission illusion where a single flash is perceived as two flashes when two rapid tone beeps are presented concurrently [Nature 408 (2000) 788; Cogn. Brain Res. 14 (2002) 147]. The absence of a fusion illusion, where two flashes would fuse to one when accompanied by one beep, indicated a perceptual rather than cognitive nature of the illusion. Here we report both fusion and fission illusions using stimuli very similar to those used by Shams et al. By instructing subjects to count beeps rather than flashes and decreasing the sound intensity to near threshold, we also created a corresponding visually induced auditory illusion. We discuss our results in light of four hypotheses of multisensory integration, each advocating a condition for modality dominance. According to the discontinuity hypothesis [Cogn. Brain Res. 14 (2002) 147], the modality in which stimulation is discontinuous dominates. The modality appropriateness hypothesis [Psychol. Bull. 88 (1980) 638] states that the modality more appropriate for the task at hand dominates. The information reliability hypothesis [J.-L. Schwartz, J. Robert-Ribes, P. Escudier, Ten years after Summerfield: a taxonomy of models for audio-visual fusion in speech perception. In: R. Campbell (Ed.), Hearing by Eye: The Psychology of Lipreading, Lawrence Earlbaum Associates, Hove, UK, (1998), pp. 3-51] claims that the modality providing more reliable information dominates. In strong forms, none of these three hypotheses applies to our data. We re-state the hypotheses in weak forms so that discontinuity, modality appropriateness and information reliability are factors which increase a modality's tendency to dominate. All these factors are important in explaining our data. Finally, we interpret the effect of instructions in light of the directed attention hypothesis which states that the attended modality is dominant [Psychol. Bull. 88 (1980) 638].

During social interaction speech is perceived simultaneously by audition and vision. We studied interactions in the processing of auditory (A) and visual (V) speech signals in the human brain by comparing neuromagnetic responses to phonetically congruent audiovisual (AV) syllables with the arithmetic sum of responses to A and V syllables. Differences between AV and A+V responses were found bilaterally in the auditory cortices 150-200 ms and in the right superior temporal sulcus (STS) 250-600 ms after stimulus onset, showing that both sensory-specific and multisensory regions of the human temporal cortices are involved in AV speech processing. Importantly, our results suggest that AV interaction in the auditory cortex precedes that in the multisensory STS region.

Normal-learning children (NL) and children with learning disabilities (LD) reported their perceptions of unisensory (auditory or visual), concordant audiovisual (e.g. visual /apa/ and auditory /apa/) and conflicting (e.g. visual /aka/ and auditory /apa/) speech stimuli in quiet and noise (0 dB and -12 dB signal-to-noise ratio, SNR). In normal populations, watching such conflicting combinations typically changes auditory percepts ('McGurk effect'). NL and LD children identified unisensory auditory and congruent audiovisual stimuli similarly in all conditions. Despite being less accurate identifying unisensory visual stimuli, LD children were more likely than NL children to report hearing only the visual component of incongruent audiovisual stimuli at -12 dB SNR. Furthermore, LD children with brainstem timing deficits demonstrated a distinctive pattern of audiovisual perception. The results suggest that the perception of simultaneous auditory and visual speech differs between NL and LD children, perhaps reflecting variations in neural processing underlying multisensory integration.

Information from different sensory modalities is processed in different cortical regions. However, our daily perception is based on the overall impression resulting from the integration of information from multiple sensory modalities. At present it is not known how the human brain integrates information from different modalities into a unified percept. Using a robust phenomenon known as the McGurk effect it was shown in the present study that audio-visual synthesis takes place within a distributed and dynamic cortical networks with emergent properties. Various cortical sites within these networks interact with each other by means of so-called operational synchrony (Kaplan, Fingelkurts, Fingelkurts, & Darkhovsky, 1997). The temporal synchronization of cortical operations processing unimodal stimuli at different cortical sites reveals the importance of the temporal features of auditory and visual stimuli for audio-visual speech integration.

Visual awareness was studied in 11 subjects by using coherent and scrambled objects as stimuli. The stimuli were presented near the subjective perceptual threshold. Explicitly recognized stimuli elicited a specific negative ERP deflection peaking at 460 ms. Our reaction time experiment suggests that this visual awareness negativity (VAN) is similar to that found previously at a shorter latency. We propose that VAN might reflect the exact moment of becoming aware of the task-relevant stimulus properties and provide an online marker of the temporal dynamics of visual awareness.

Seeing a talker's articulatory gestures may affect the observer's auditory speech percept. Observing congruent articulatory gestures may enhance the recognition of speech sounds [J. Acoust. Soc. Am. 26 (1954) 212], whereas observing incongruent gestures may change the auditory percept phonetically as occurs in the McGurk effect [Nature 264 (1976) 746]. For example, simultaneous acoustic /ba/ and visual /ga/ are usually heard as /da/. We studied cortical processing of occasional changes in audiovisual and visual speech stimuli with magnetoencephalography. In the audiovisual experiment congruent (acoustic /iti/, visual /iti/) and incongruent (acoustic /ipi/, visual /iti/) audiovisual stimuli, which were both perceived as /iti/, were presented among congruent /ipi/ (acoustic /ipi/, visual /ipi/) stimuli. In the visual experiment only the visual components of these stimuli were presented. A visual change both in audiovisual and visual experiments activated supratemporal auditory cortices bilaterally. The auditory cortex activation to a visual change occurred later in the visual than in the audiovisual experiment, suggesting that interaction between modalities accelerates the detection of visual change in speech.

The present study examined hemispheric asymmetries in emotional processing. 40 right-handed women were presented sequences of Happy, Sad, and Disgusted facial expressions to their left and right visual fields followed by a Neutral face presented to ipsi- or contralateral visual field. After Happy sequences, Neutral faces were rated as significantly more pleasant than after Sad sequences. In respect to hemispheric asymmetries, the analyses gave a significant effect of visual field only for Sad sequences. The ratings of Neutral faces were significantly more positive after presentations of Sad sequences to the left than to the right visual field. These modulational effects are suggested to support recent findings of hemispheric asymmetries of human emotions.

The cerebral representation of language, deduced from observing patients with brain lesions and from stimulations and recordings performed during brain surgery, has been further clarified by recent positron emission tomography and functional magnetic resonance imaging measurements. We now expand this static view into the dynamics of cortical activation using the accurate spatiotemporal resolution of whole-head magnetoencephalography. During picture naming, the conversion from visual to symbolic representation progressed bilaterally from the occipital visual cortex towards temporal and frontal lobes. Overt naming elicited the most widespread cortical activation. Some language-related sites also reacted, though more weakly or after a longer delay, during covert naming and even passive viewing.

We have examined magnetic cortical responses of 15 healthy humans to 46 different pictures of faces. At least three areas outside the occipital visual cortex appeared to be involved in processing this input, 105-560 ms after the stimulus onset. The first active area was near the occipitotemporal junction, the second in the inferior parietal lobe, and the third in the middle temporal lobe. The source in the inferior parietal lobe was also activated by other simple and complex visual stimuli.

Effect of repetition of auditory and visual stimuli on sensory-evoked sustained potentials was studied. The stimuli were tones and flashes of 1 sec duration presented in trains of six stimuli with an inter-stimulus-interval of 1 sec. The repetition rate of the trains was 1 train/min. The EEG was recorded from electrodes located at Cz, Pz and Oz. Both auditory and visual stimuli elicited negative sustained potentials which during the first stimuli of the trains were maximal in amplitude and shortest in latency at Cz. Repetition of the stimulus resulted in almost complete disappearance of the sustained potentials; for auditory stimulation a small negative shift was recorded only at Cz, whereas for visual stimulation a small sustained potential was seen only at Pz and Oz. The results indicate that the slow potentials evoked by short tones or lights are composed of at least two components. One of these components may reflect neural processing associated with orienting reflex. The other component seems to be related to specific neural processing of the stimulus.