Search

In Autism Spectrum Disorders (ASD), impaired response inhibition and lack of adaptation are hypothesized to underlie core ASD symptoms, such as social communication and repetitive, stereotyped behavior. Thus, the aim of the present study was to compare neural correlates of inhibition, post-error adaptation, and reaction time variability in ASD and neuro-typical control (NTC) participants by investigating possible differences in error-related changes of oscillatory MEG activity. Twelve male NTC (mean age 20.3 ± 3.7) and fourteen male patients with ASD (mean age 17.8 ± 2.9) were included in the analysis. Subjects with ASD showed increased error-related reaction time variability. MEG analysis revealed decreased beta power in the ASD group in comparison to the NTC group over the centro-parietal channels in both, the pre-stimulus and post-response interval. In the ASD group, mean centro-parietal beta power negatively correlated with dimensional autism symptoms. In both groups, false alarms were followed by an early increase in temporo-frontal theta to alpha power; and by a later decrease in alpha to beta power at central and posterior sensors. Single trial correlations were additionally studied in the ASD group, who showed a positive correlation of pre-stimulus beta power with post-response theta, alpha, and beta power, particularly after hit trials. On a broader scale, the results deliver important insights into top-down control deficits that may relate to core symptoms observed in ASD.

Objective: To determine whether the performance of multiple sclerosis (MS) patients in the sound-induced flash illusion (SiFi), a multisensory perceptual illusion, would reflect their cognitive impairment. Methods: We performed the SiFi task as well as an extensive neuropsychological testing in 95 subjects [39 patients with relapse-remitting MS (RRMS), 16 subjects with progressive multiple sclerosis (PMS) and 40 healthy control subjects (HC)]. Results: MS patients reported more frequently the multisensory SiFi than HC. In contrast, there were no group differences in the control conditions. Essentially, patients with progressive type of MS continued to perceive the illusion at stimulus onset asynchronies (SOA) that were more than three times longer than the SOA at which the illusion was already disrupted for healthy controls. Furthermore, MS patients' degree of cognitive impairment measured with a broad neuropsychological battery encompassing tests for memory, attention, executive functions, and fluency was predicted by their performance in the SiFi task for the longest SOA of 500 ms. Conclusions: These findings support the notion that MS patients exhibit an altered multisensory perception in the SiFi task and that their susceptibility to the perceptual illusion is negatively correlated with their neuropsychological test performance. Since MS lesions affect white matter tracts and cortical regions which seem to be involved in the transfer and processing of both crossmodal and cognitive information, this might be one possible explanation for our findings. SiFi might be considered as a brief, non-expensive, language- and education-independent screening test for cognitive deficits in MS patients.

Multisensory integration strongly depends on the temporal proximity between two inputs. In the audio-visual domain, stimulus pairs with delays up to a few hundred milliseconds can be perceived as simultaneous and integrated into a unified percept. Previous research has shown that the size of this temporal window of integration can be narrowed by feedback-guided training on an audio-visual simultaneity judgment task. Yet, it has remained uncertain how the neural network that processes audio-visual asynchronies is affected by the training. In the present study, participants were trained on a 2-interval forced choice audio-visual simultaneity judgment task. We recorded their neural activity with magnetoencephalography in response to three different stimulus onset asynchronies (0 ms, each participant’s individual binding window, 300 ms) before, and one day following training. The Individual Window stimulus onset asynchrony condition was derived by assessing each participant’s point of subjective simultaneity. Training improved performance in both asynchronous stimulus onset conditions (300 ms, Individual Window). Furthermore, beta-band amplitude (12–30 Hz) increased from pre-compared to post-training sessions. This increase moved across central, parietal, and temporal sensors during the time window of 80–410 ms post-stimulus onset. Considering the putative role of beta oscillations in carrying feedback from higher to lower cortical areas, these findings suggest that enhanced top-down modulation of sensory processing is responsible for the improved temporal acuity after training. As beta oscillations can be assumed to also preferentially support neural communication over longer conduction delays, the widespread topography of our effect could indicate that training modulates not only processing within primary sensory cortex, but rather the communication within a large-scale network.

The neurophysiological underpinnings of the nonsocial symptoms of autism spectrum disorder (ASD) which include sensory and perceptual atypicalities remain poorly understood. Well-known accounts of less dominant top-down influences and more dominant bottom-up processes compete to explain these characteristics. These accounts have been recently embedded in the popular framework of predictive coding theory. To differentiate between competing accounts, we studied altered information dynamics in ASD by quantifying predictable information in neural signals. Predictable information in neural signals measures the amount of stored information that is used for the next time step of a neural process. Thus, predictable information limits the (prior) information which might be available for other brain areas, for example, to build predictions for upcoming sensory information. We studied predictable information in neural signals based on resting-state magnetoencephalography (MEG) recordings of 19 ASD patients and 19 neurotypical controls aged between 14 and 27 years. Using whole-brain beamformer source analysis, we found reduced predictable information in ASD patients across the whole brain, but in particular in posterior regions of the default mode network. In these regions, epoch-by-epoch predictable information was positively correlated with source power in the alpha and beta frequency range as well as autocorrelation decay time. Predictable information in precuneus and cerebellum was negatively associated with nonsocial symptom severity, indicating a relevance of the analysis of predictable information for clinical research in ASD. Our findings are compatible with the assumption that use or precision of prior knowledge is reduced in ASD patients.

Aging is accompanied by unisensory decline; but to compensate for this, two complementary strategies are potentially relied upon increasingly: first, older adults integrate more information from different sensory organs. Second, according to predictive coding (PC) we form ‘templates’ (internal models or ‘priors’) of the environment through our experiences. It is through increased life experience that older adults may rely more on these templates compared to younger adults. Multisensory integration and predictive coding would be effective strategies for the perception of near-threshold stimuli, but they come at the cost of integrating irrelevant information. Their role can be studied in multisensory illusions because these require the integration of different sensory information, as well as an internal model of the world that can take precedence over sensory input. Here, we elicited a classic multisensory illusion, the sound-induced flash illusion, in younger (mean: 27 yrs) and older (mean: 67 yrs) adult participants while recording the magnetoencephalogram. Older adults perceived more illusions than younger adults. Older adults had increased pre-stimulus beta(β)-band activity compared to younger adults as predicted by microcircuit theories of predictive coding, which suggest priors and predictions are linked to β-band activity. In line with our hypothesis, transfer entropy analysis and dynamic causal models of pre-stimulus MEG data revealed a stronger illusion-related modulation of cross-modal connectivity from auditory to visual cortices in older compared to younger adults. We interpret this as the neural correlate of increased reliance on a cross-modal predictive template in older adults that is leading to the illusory percept.

Human neuroimaging studies suggest that neural cue reactivity is strongly associated with indices of drug use, including addiction severity and treatment success. However, little is known about factors that modulate cue reactivity. The goal of this review, in which we survey published fMRI and PET studies on drug cue reactivity in cocaine, alcohol, and tobacco cigarette users, is to highlight major factors that modulate brain reactivity to drug cues. First, we describe cue reactivity paradigms used in neuroimaging research and outline the brain circuits that underlie cue reactivity. We then discuss major factors that have been shown to modulate cue reactivity and review specific evidence as well as outstanding questions related to each factor. Building on previous model-building reviews on the topic, we then outline a simplified model that includes the key modulatory factors and a tentative ranking of their relative impact. We conclude with a discussion of outstanding challenges and future research directions, which can inform future neuroimaging studies as well as the design of treatment and prevention programs.

The most common form of synaesthesia is grapheme–colour synaesthesia. However, rarer forms of synaesthesia also exist, such as word–gustatory and olfactory–gustatory synaesthesia, whereby a word or smell will induce a specific. In this study we describe a single individual (LJ) who experiences a concurrent olfactory stimulus when presented with congruent visual images. For some visual stimuli, he perceives a strong and automatic olfactory percept, which has existed throughout his life. In this study, we explore whether his experiences are a new form of synaesthesia or simply vivid imagery. Unlike other forms of synaesthesia, the concurrent odour is congruent to the visual inducer. For example, a photograph of dress shoes will elicit the smell of leather. We presented LJ and several control participants with 75 images of everyday objects. Their task was to indicate the strength of any perceived odours induced by the visual images. LJ rated several of the images as inducing a concurrent odour, while controls did not have any such percept. Images that LJ reported as inducing the strongest odours were used, along with colour-matched control images, in the context of an fMRI experiment. Participants were given a one-back task to maintain attention. A block-design odour localizer was presented to localize the piriform cortex (primary olfactory cortex). We found an increased BOLD response in the piriform cortex for the odour-inducing images compared to the control images in LJ. There was no difference in BOLD response between these two stimulus types in the control participants. A subsequent olfactory imagery task did not elicit enhanced activity in the piriform cortex in LJ, suggesting his perceptual experiences may not be based on olfactory imagery.

Addiction is a significant health burden with a major impact on society. Its neural underpinnings have been extensively studied in animal experiments and human neuroimaging studies, which have demonstrated that during addiction various reorganizational processes hijack the reward and habit systems. Thus, subcortical and cortical brain regions become involved in pathological learning and maintenance of incentive values and automatized actions, which consequently contribute to the emergence and persistence of addiction. Reward-related processes that are affected predominantly during early stages of addiction comprise the ventral tegmental area, ventral and dorsomedial striatum, amygdala, hippocampus, orbitofrontal/medial prefrontal cortex, anterior cingulate cortex, and insula. Habit-related mechanisms are under the control of a more complex network including interconnections between the ventral tegmental area, ventral and dorsolateral striatum, parts of the amygdala, and cortical regions such as premotor and motor cortex, parietal and inferior/middle temporal cortex, as well as the cerebellum.

Behavioral experiments have demonstrated that the sensory modality of presentation modulates drug cue reactivity. The present study on nicotine addiction tested whether neural responses to smoking cues are modulated by the sensory modality of stimulus presentation. We measured brain activation using functional magnetic resonance imaging (fMRI) in 15 smokers and 15 nonsmokers while they viewed images of smoking paraphernalia and control objects and while they touched the same objects without seeing them. Haptically presented, smoking-related stimuli induced more pronounced neural cue reactivity than visual cues in the left dorsal striatum in smokers compared to nonsmokers. The severity of nicotine dependence correlated positively with the preference for haptically explored smoking cues in the left inferior parietal lobule/somatosensory cortex, right fusiform gyrus/inferior temporal cortex/cerebellum, hippocampus/parahippocampal gyrus, posterior cingulate cortex, and supplementary motor area. These observations are in line with the hypothesized role of the dorsal striatum for the expression of drug habits and the well-established concept of drug-related automatized schemata, since haptic perception is more closely linked to the corresponding object-specific action pattern than visual perception. Moreover, our findings demonstrate that with the growing severity of nicotine dependence, brain regions involved in object perception, memory, self-processing, and motor control exhibit an increasing preference for haptic over visual smoking cues. This difference was not found for control stimuli. Considering the sensory modality of the presented cues could serve to develop more reliable fMRI-specific biomarkers, more ecologically valid experimental designs, and more effective cue-exposure therapies of addiction.

Zusammenfassung: Obwohl unser Verständnis der Rolle neurobiologischer und kognitiver Mechanismen für die Entstehung und Aufrechterhaltung von Sucht ständig zunimmt, wird dieses Wissen noch wenig in die ärztliche oder psychotherapeutische Ausbildung und Praxis integriert. Dieser Artikel gibt einen Überblick über diejenigen neuronalen und kognitiven Prozesse, die den verschiedenen Symptomen der Sucht zugrunde liegen. Dazu gehören das Motivations- und Gedächtnissystem sowie automatisiertes Verhalten und exekutive Kontrollfunktionen. Besonders betont wird die Relevanz neurowissenschaftlicher Forschungsergebnisse für ein besseres Verständnis der neurophysiologischen Hintergründe der diagnostischen Kriterien des Abhängigkeitssyndroms.

Neuroimaging studies on cue reactivity have substantially contributed to the understanding of addiction. In the majority of studies drug cues were presented in the visual modality. However, exposure to conditioned cues in real life occurs often simultaneously in more than one sensory modality. Therefore, multisensory cues should elicit cue reactivity more consistently than unisensory stimuli and increase the ecological validity and the reliability of brain activation measurements. This review includes the data from 44 whole-brain functional neuroimaging studies with a total of 1168 subjects (812 patients and 356 controls). Correlations between neural cue reactivity and clinical covariates such as craving have been reported significantly more often for multisensory than unisensory cues in the motor cortex, insula and posterior cingulate cortex. Thus, multisensory drug cues are particularly effective in revealing brain-behavior relationships in neurocircuits of addiction responsible for motivation, craving awareness and self-related processing.

The new DSM-5 diagnostic criteria for autism spectrum disorders (ASDs) include sensory disturbances in addition to the well-established language, communication, and social deficits. One sensory disturbance seen in ASD is an impaired ability to integrate multisensory information into a unified percept. This may arise from an underlying impairment in which individuals with ASD have difficulty perceiving the temporal relationship between cross-modal inputs, an important cue for multisensory integration. Such impairments in multisensory processing may cascade into higher-level deficits, impairing day-to-day functioning on tasks, such as speech perception. To investigate multisensory temporal processing deficits in ASD and their links to speech processing, the current study mapped performance on a number of multisensory temporal tasks (with both simple and complex stimuli) onto the ability of individuals with ASD to perceptually bind audiovisual speech signals. High-functioning children with ASD were compared with a group of typically developing children. Performance on the multisensory temporal tasks varied with stimulus complexity for both groups; less precise temporal processing was observed with increasing stimulus complexity. Notably, individuals with ASD showed a speech-specific deficit in multisensory temporal processing. Most importantly, the strength of perceptual binding of audiovisual speech observed in individuals with ASD was strongly related to their low-level multisensory temporal processing abilities. Collectively, the results represent the first to illustrate links between multisensory temporal function and speech processing in ASD, strongly suggesting that deficits in low-level sensory processing may cascade into higher-order domains, such as language and communication.

The processing of visual and haptic inputs, occurring either separately or jointly, is crucial for everyday-life object recognition, and has been a focus of recent neuroimaging research. Previously, visuohaptic convergence has been mostly investigated with matching-task paradigms. However, much less is known about visuohaptic convergence in the absence of additional task demands. We conducted two functional magnetic resonance imaging experiments in which subjects actively touched and/or viewed unfamiliar object stimuli without any additional task demands. In addition, we performed two control experiments with audiovisual and audiohaptic stimulation to examine the specificity of the observed visuohaptic convergence effects. We found robust visuohaptic convergence in bilateral lateral occipital cortex and anterior cerebellum. In contrast, neither the anterior cerebellum nor the lateral occipital cortex showed any involvement in audiovisual or audiohaptic convergence, indicating that multisensory convergence in these regions is specifically geared to visual and haptic inputs. These data suggest that in humans the lateral occipital cortex and the anterior cerebellum play an important role in visuohaptic processing even in the absence of additional task demands.

Research on the psychological and neuronal underpinnings of addiction has concentrated mostly on affective, motivational, learning, and executive processes and the brain regions subserving these functions. In contrast, sensory and motor aspects of addiction have largely been neglected even though they may be highly relevant for the development and preservation of addiction. The aim of the present review is to emphasize the significance of sensory and motor processes for the better understanding and treatment of addiction. We focus on research investigating substance-related addictions and on brain imaging studies in humans that have assessed the contribution of cortical and cerebellar systems to sensory and motor mechanisms of addiction. Activations of sensory and motor brain regions in response to drug-associated cues can predict relapse and correlate with craving, severity of dependence and automatized behavioral reactions towards drug-related stimuli. We propose a model of how sensory and motor processes might be involved in the different stages of addiction.

Several regions in human temporal and frontal cortex are known to integrate visual and auditory object features. The processing of audio–visual (AV) associations in these regions has been found to be modulated by object familiarity. The aim of the present study was to explore training-induced plasticity in human cortical AV integration. We used functional magnetic resonance imaging to analyze the neural correlates of AV integration for unfamiliar artificial object sounds and images in naïve subjects (PRE training) and after a behavioral training session in which subjects acquired associations between some of these sounds and images (POST-training). In the PRE-training session, unfamiliar artificial object sounds and images were mainly integrated in right inferior frontal cortex (IFC). The POST-training results showed extended integration-related IFC activations bilaterally, and a recruitment of additional regions in bilateral superior temporal gyrus/sulcus and intraparietal sulcus. Furthermore, training-induced differential response patterns to mismatching compared with matching (i.e., associated) artificial AV stimuli were most pronounced in left IFC. These effects were accompanied by complementary training-induced congruency effects in right posterior middle temporal gyrus and fusiform gyrus. Together, these findings demonstrate that short-term cross-modal association learning was sufficient to induce plastic changes of both AV integration of object stimuli and mechanisms of AV congruency processing.

The cortical integration of auditory and visual features is crucial for efficient object recognition. Previous studies have shown that audiovisual (AV) integration is affected by where and when auditory and visual features occur. However, because relatively little is known about the impact of what is integrated, we here investigated the impact of semantic congruency and object familiarity on the neural correlates of AV integration. We used functional magnetic resonance imaging to identify regions involved in the integration of both (congruent and incongruent) familiar animal sounds and images and of arbitrary combinations of unfamiliar artificial sounds and object images. Unfamiliar object images and sounds were integrated in the inferior frontal cortex (IFC), possibly reflecting learning of novel AV associations. Integration of familiar, but semantically incongruent combinations also correlated with IFC activation and additionally involved the posterior superior temporal sulcus (pSTS). For highly familiar semantically congruent AV pairings, we again found AV integration effects in pSTS and additionally in superior temporal gyrus. These findings demonstrate that the neural correlates of object-related AV integration reflect both semantic congruency and familiarity of the integrated sounds and images.

The successful integration of visual and auditory stimuli requires information about whether visual and auditory signals originate from corresponding places in the external world. Here we report crossmodal effects of spatially congruent and incongruent audio-visual (AV) stimulation. Visual and auditory stimuli were presented from one of four horizontal locations in external space. Seven healthy human subjects had to assess the spatial fit of a visual stimulus (i.e. a gray-scaled picture of a cartoon dog) and a simultaneously presented auditory stimulus (i.e. a barking sound). Functional magnetic resonance imaging (fMRI) revealed two distinct networks of cortical regions that processed preferentially either spatially congruent or spatially incongruent AV stimuli. Whereas earlier visual areas responded preferentially to incongruent AV stimulation, higher visual areas of the temporal and parietal cortex (left inferior temporal gyrus [ITG], right posterior superior temporal gyrus/sulcus [pSTG/STS], left intra-parietal sulcus [IPS]) and frontal regions (left pre-central gyrus [PreCG], left dorsolateral pre-frontal cortex [DLPFC]) responded preferentially to congruent AV stimulation. A position-resolved analysis revealed three robust cortical representations for each of the four visual stimulus locations in retinotopic visual regions corresponding to the representation of the horizontal meridian in area V1 and at the dorsal and ventral borders between areas V2 and V3. While these regions of interest (ROIs) did not show any significant effect of spatial congruency, we found subregions within ROIs in the right hemisphere that showed an incongruency effect (i.e. an increased fMRI signal during spatially incongruent compared to congruent AV stimulation). We interpret this finding as a correlate of spatially distributed recurrent feedback during mismatch processing: whenever a spatial mismatch is detected in multisensory regions (such as the IPS), processing resources are re-directed to low-level visual areas.

The perception of objects is a cognitive function of prime importance. In everyday life, object perception benefits from the coordinated interplay of vision, audition, and touch. The different sensory modalities provide both complementary and redundant information about objects, which may improve recognition speed and accuracy in many circumstances. We review crossmodal studies of object recognition in humans that mainly employed functional magnetic resonance imaging (fMRI). These studies show that visual, tactile, and auditory information about objects can activate cortical association areas that were once believed to be modality-specific. Processing converges either in multisensory zones or via direct crossmodal interaction of modality-specific cortices without relay through multisensory regions. We integrate these findings with existing theories about semantic processing and propose a general mechanism for crossmodal object recognition: The recruitment and location of multisensory convergence zones varies depending on the information content and the dominant modality.

Visuo-haptic integration contributes essentially to object shape recognition. Although there has been a considerable advance in elucidating the neural underpinnings of multisensory perception, it is still unclear whether seeing an object and exploring it with the dominant hand elicits the same brain response as compared to the non-dominant hand. Using fMRI to measure brain activation in right-handed participants, we found that for both left- and right-hand stimulation the left lateral occipital complex (LOC) and anterior cerebellum (aCER) were involved in visuo-haptic integration of familiar objects. These two brain regions were then further investigated in another study, where unfamiliar, novel objects were presented to a different group of right-handers. Here the left LOC and aCER were more strongly activated by bimodal than unimodal stimuli only when the left but not the right hand was used. A direct comparison indicated that the multisensory gain of the fMRI activation was significantly higher for the left than the right hand. These findings are in line with the principle of "inverse effectiveness", implying that processing of bimodally presented stimuli is particularly enhanced when the unimodal stimuli are weak. This applies also when right-handed subjects see and simultaneously touch unfamiliar objects with their non-dominant left hand. Thus, the fMRI signal in the left LOC and aCER induced by visuo-haptic stimulation is dependent on which hand was employed for haptic exploration.

Previous studies investigating mild cognitive impairment (MCI) have focused primarily on cognitive, memory, attention, and executive function deficits. There has been relatively little research on the perceptual deficits people with MCI may exhibit. This is surprising given that it has been suggested that sensory and cognitive functions share a common cortical framework [1]. In the following study, we presented the sound-induced flash illusion (SiFi) to a group of participants with mild cognitive impairment (MCI) and healthy controls (HC). The SiFi is an audio-visual illusion whereby two-beeps and one-flash are presented. Participants tend to perceive two flashes when the time-interval between the auditory beeps is small [2, 3]. Participants with MCI perceived significantly more illusions compared to HC over longer auditory time-intervals. This suggests that MCIs integrate more (arguably irrelevant) audiovisual information compared to HCs. By incorporating perceptual tasks into a clinical diagnosis it may be possible to gain a more comprehensive understanding into the disease, as well as provide a more accurate diagnose to those who may have a language impairment.

The car dependence of people living in contemporary cities is a major concern for policy makers, who often find it difficult to persuade people into more sustainable transport modes. By contrast, recent insights from neuroscience have shown that a broad spectrum of behaviors can become habitual and, thus, resistant to change. Here, we outline the potential of collaboration between neuroscience and human geography aiming at a better understanding of habits that determine everyday commuting routines.

Dual-coding theory (Paivio, 1986) postulates that the human mind represents objects not just with an analogous, or semantic code, but with a perceptual representation as well. Previous studies (eg, Fiebach & Friederici, 2004) indicated that the modality of this representation is not necessarily the one that triggers the representation. The human visual cortex contains several regions, such as the Lateral Occipital Complex (LOC), that respond specifically to object stimuli. To investigate whether these principally visual representations regions are also recruited for auditory stimuli, we presented subjects with spoken words with specific, concrete meanings (‘car’) as well as words with abstract meanings (‘hope’). Their brain activity was measured with functional magnetic resonance imaging. Whole-brain contrasts showed overlap between regions differentially activated by words for concrete objects compared to words for abstract concepts with visual regions activated by a contrast of object versus non-object visual stimuli. We functionally localized LOC for individual subjects and a preliminary analysis showed a trend for a concreteness effect in this region-of-interest on the group level. Appropriate further analysis might include connectivity and classification measures. These results can shed light on the role of crossmodal representations in cognition.

Primate multisensory object perception involves distributed brain regions. To investigate the network character of these regions of the human brain, we applied data-driven group spatial independent component analysis (ICA) to a functional magnetic resonance imaging (fMRI) data set acquired during a passive audio-visual (AV) experiment with common object stimuli. We labeled three group-level independent component (IC) maps as auditory (A), visual (V), and AV, based on their spatial layouts and activation time courses. The overlap between these IC maps served as definition of a distributed network of multisensory candidate regions including superior temporal, ventral occipito-temporal, posterior parietal and prefrontal regions. During an independent second fMRI experiment, we explicitly tested their involvement in AV integration. Activations in nine out of these twelve regions met the max-criterion (A V) for multisensory integration. Comparison of this approach with a general linear model-based region-of-interest definition revealed its complementary value for multisensory neuroimaging. In conclusion, we estimated functional networks of uni- and multisensory functional connectivity from one dataset and validated their functional roles in an independent dataset. These findings demonstrate the particular value of ICA for multisensory neuroimaging research and using independent datasets to test hypotheses generated from a data-driven analysis. Electronic supplementary material The online version of this article (doi:10.1007/s00221-011-2669-0) contains supplementary material, which is available to authorized users.

The study of Wagner et al. ( J Neurosci 31: 894–898, 2011) reveals the neural correlates of spontaneously activated action representations in smokers when subjects watch movie characters smoke. We stress the importance of differentiating how these representations are activated: while the anterior intraparietal sulcus and inferior frontal gyrus are part of the mirror neuron system of smokers, the middle frontal gyrus, premotor cortex, and superior parietal lobule represent the smoking-related tool use skills and action knowledge activated by smoking paraphernalia.

Information integration across different sensory modalities contributes to object recognition, the generation of associations and long-term memory representations. Here, we used functional magnetic resonance imaging adaptation to investigate the presence of sensory integrative effects at cortical levels as early as nonprimary auditory and extrastriate visual cortices, which are implicated in intermediate stages of object processing. Stimulation consisted of an adapting audiovisual stimulus S1and a subsequent stimulus S2from the same basic-level category (e.g., cat). The stimuli were carefully balanced with respect to stimulus complexity and semantic congruency and presented in four experimental conditions: (1) the same image and vocalization for S1and S2, (2) the same image and a different vocalization, (3) different images and the same vocalization, or (4) different images and vocalizations. This two-by-two factorial design allowed us to assess the contributions of auditory and visual stimulus repetitions and changes in a statistically orthogonal manner. Responses in visual regions of right fusiform gyrus and right lateral occipital cortex were reduced for repeated visual stimuli (repetition suppression). Surprisingly, left lateral occipital cortex showed stronger responses to repeated auditory stimuli (repetition enhancement). Similarly, auditory regions of interest of the right middle superior temporal gyrus and sulcus exhibited repetition suppression to auditory repetitions and repetition enhancement to visual repetitions. Our findings of crossmodal repetition-related effects in cortices of the respective other sensory modality add to the emerging view that in human subjects sensory integrative mechanisms operate on earlier cortical processing levels than previously assumed.

A. Methodological considerations Charles E. Schroeder, Nathan S. Kline Institute, Orangeburg, NY 'Invasive electrophysiology of multisensory processing in monkeys' Alexandre Pouget, University of Rochester, Rochester, NY 'A computational review of multisensory perception' Marie-Helene Giard, INSERM, Lyon, France 'Electrophysiology of multisensory interactions in humans' Marcus J. Naumer, Jochen Kaiser et al., Goethe-Univ., Frankfurt, Germany 'Statistical criteria for functional imaging studies of multisensory integration' B. Audio-visual integration Charles Spence, Oxford Univ., UK 'Audio-visual object processing' Christoph Kayser, Max Planck Institute, Tuebingen, Germany 'Multisensory integration in early sensory cortices' Asif Ghazanfar, Princeton Univ., Princeton, NJ 'Multisensory integration in the rhesus monkey temporal lobe' Lizabeth M. Romanski, Univ. of Rochester, Rochester, NY 'Multisensory integration in primate prefrontal cortex' John J. Foxe, Nathan S. Kline Institute, Orangeburg, NY 'Electrophysiological correlates of audio-visual object perception' Marcus J. Naumer, Jochen Kaiser et al., Goethe-Univ., Frankfurt, Germany 'Audio-visual integration of familiar natural objects' Micah M. Murray, Universite de Lausanne, Lausanne, Switzerland 'Audio-visual object memory' C. Visuo-tactile integration Roberta L. Klatzky, Carnegie Mellon University, Pittsburgh, PA 'Multisensory texture perception' Krish Sathian, Emory Univ., Atlanta, GA 'Visuo-tactileintegration of texture and shape and its plasticity' Michael S. Beauchamp, University of Texas, Houston, TX 'Grounding object concepts in perception and action: evidence from fMRI studies of tools' Fiona N. Newell, Trinity College, Dublin, Ireland 'Visuo-haptic perception of objects and scenes' Susan J. Lederman, Queen's University, Kingston, Ontario, CA 'Visuo-haptic face processing' D. Plasticity David J. Lewkowicz, Florida Atlantic University, Boca Raton, FL 'The ontogeny of human multisensory perception' Mark T. Wallace, Vanderbilt Univ., Nashville, TN 'The development of multisensory object representations' Amir Amedi, Hebrew Univ., Jerusalem, Israel 'Developmental versus adult plasticity and the use of sensory substitution devices'

Oscillatory activity in the higher (gamma) frequency range in electro- or magnetoencephalogram (EEG or MEG, respectively) has been proposed as a correlate of cortical network synchronization. This chapter reviews the evidence for an involvement of gamma-band activity in multisensory integration. The relevant studies are grouped by the level at which oscillatory EEG or MEG data were analyzed. The first section describes reports of topographically unspecific or widespread spectral activity. The second part focuses on studies demonstrating topographically local activity both at the electrode/sensor level and at the cortical source level. Finally, reports of cortico-cortical interactions as suggested by EEG/MEG coherence or phase synchrony are summarized. Gamma-band activity increases have been reported for temporally, spatially, or semantically matching bimodal stimulation. Some studies, however, have reported enhanced oscillatory signals also to incongruent stimuli, suggesting an important role of task requirements and attentional factors.

Since its invention almost two decades ago functional magnetic resonance imaging (fMRI) has become the prime research methodology in human neuroscience. Its capabilities continue to evolve based on combined improvements of scanner hardware, experimental designs, and data analysis tools. Within the rapidly growing field of multisensory research the use of noninvasive neuroimaging techniques in general and fMRI in particular is also of increasing relevance. For several years, discussion in the multisensory fMRI community has mainly focused on principles of and statistical criteria for multisensory integration. The recent availability of more sophisticated experimental designs and increasingly sensitive (multivariate) analysis tools allows multisensory researchers to (noninvasively) differentiate between regional and neuronal convergence and to reveal the connectional basis of human multisensory integration.

In mammals smooth retinotopic maps of the visual field are formed along the visual processing pathway whereby the left visual field is represented in the right hemisphere and vice versa. The reorganization of retinotopic maps in the lateral geniculate nucleus (LGN) of the thalamus and early visual areas (V1–V3) is studied in a patient who was born with only one cerebral hemisphere. Before the seventh week of embryonic gestation, the development of the patient's right cerebral hemisphere terminated. Despite the complete loss of her right hemisphere (di- and telencephalon) at birth, the patient's remaining hemisphere has not only developed maps of the contralateral (right) visual hemifield but, surprisingly, also maps of the ipsilateral (left) visual hemifield. Retinal ganglion-cells changed their predetermined crossing pattern in the optic chiasm and grew to the ipsilateral LGN. In the visual cortex, islands of ipsilateral visual field representations were located along the representations of the vertical meridian. In V1, smooth and continuous maps from contra- and ipsilateral hemifield overlap each other, whereas in ventral V2 and V3 ipsilateral quarter field representations invaded small distinct cortical patches. This reveals a surprising flexibility of the self-organizing developmental mechanisms responsible for map formation.

In addition to reward- and craving-related processes, habitual mechanisms play an important role in addiction. While the dorsal striatum has been proposed to code for the motivational state of habitual drug-seeking actions, the neural underpinnings of the corresponding drug-taking skills and action knowledge remain poorly understood. We used functional magnetic resonance imaging (fMRI) and a behavioral orientation affordance paradigm to investigate the neural and behavioral correlates of automatized drug-taking actions in nicotine dependence. Smokers exhibited higher fMRI activations than nonsmokers when viewing smoking-related but not when viewing control images. These group differences in fMRI activations were located not only in brain regions associated with craving and habitual learning (left ventral and dorsal striatum, dorsolateral prefrontal cortex, insula, uncus, medial frontal gyrus, right subcallosal gyrus, and bilateral parahippocampal gyrus), but also in a network of brain regions which has been strongly implicated in the encoding of action knowledge and tool use skills (bilateral premotor cortex, left superior parietal lobule, and right lateral cerebellum). A behavioral affordance reaction-time task indicated that smokers, but not nonsmokers, showed an automatized responsiveness to smoking paraphernalia similar to everyday objects. Moreover, smokers showed strong intercorrelations between fMRI activations in tool use-related brain regions, behavioral responsiveness to smoking-related cues, and severity of nicotine dependence. Apparently smoking-related action representations in smokers are stored in brain regions typically representing tool use skills and action knowledge. Most importantly, cortical and behavioral correlates of the respective drug-taking skills vary with the individual degree of nicotine dependence.

In complex natural environments, auditory and visual information often have to be processed simultaneously. Previous functional magnetic resonance imaging (fMRI) studies focused on the spatial localization of brain areas involved in audiovisual (AV) information processing, but the temporal characteristics of AV information flow in these regions remained unclear. In this study, we used fMRI and a novel information–theoretic approach to study the flow of AV sensory information. Subjects passively perceived sounds and images of objects presented either alone or simultaneously. Applying the measure of mutual information, we computed for each voxel the latency in which the blood oxygenation level-dependent signal had the highest information content about the preceding stimulus. The results indicate that, after AV stimulation, the earliest informative activity occurs in right Heschl's gyrus, left primary visual cortex, and the posterior portion of the superior temporal gyrus, which is known as a region involved in object-related AV integration. Informative activity in the anterior portion of superior temporal gyrus, middle temporal gyrus, right occipital cortex, and inferior frontal cortex was found at a later latency. Moreover, AV presentation resulted in shorter latencies in multiple cortical areas compared with isolated auditory or visual presentation. The results provide evidence for bottom-up processing from primary sensory areas into higher association areas during AV integration in humans and suggest that AV presentation shortens processing time in early sensory cortices.

Earlier studies reported evidence suggesting distinct category-related auditory representations for environmental sounds such as animal vocalizations and tool sounds in superior and middle temporal regions of the temporal lobe. However, the degree of selectivity of these representations remains to be determined. The present study combined functional magnetic resonance imaging (fMRI) adaptation with a silent acquisition protocol to further investigate category-related auditory processing of environmental sounds. To this end, we consecutively presented pairs of sounds taken from the categories 'tool sounds' or 'animal vocalizations' with either the same or different identity/category. We examined the degree of selectivity as evidenced by adaptation effects to both or only one sound category in the course of whole-brain as well as functionally and anatomically constrained region of interest analyses. While most regions predominately in the temporal cortex showed an adaptation to both sound categories, particularly the left superior temporal gyrus (STG) and the left posterior middle temporal gyrus (pMTG) selectively adapted to animal vocalizations and tool sounds, respectively. However, the activation profiles of these regions differed with respect to the general responsiveness to sounds. While tool sounds still produced fMRI signals significantly different from fixation baseline in the STG, this was not the case for animal vocalizations in pMTG. Consistent with the interpretation of STG as an intermediate auditory processing stage, this region might differentiate auditory stimuli into categories based on variations of physical stimulus properties. However, processing in left pMTG seems to be even more restricted to action-related sounds of man-made objects.

By using meaningful stimuli, multisensory research has recently started to investigate the impact of stimulus content on crossmodal integration. Variations in this respect have often been termed as "semantic". In this paper we will review work related to the question for which tasks the influence of semantic factors has been found and which cortical networks are most likely to mediate these effects. More specifically, the focus of this paper will be on processing of object stimuli presented in the auditory and visual sensory modalities. Furthermore, we will investigate which cortical regions are particularly responsive to experimental variations of content by comparing semantically matching ("congruent") and mismatching ("incongruent") experimental conditions. In this context, recent neuroimaging studies point toward a possible functional differentiation of temporal and frontal cortical regions, with the former being more responsive to semantically congruent and the latter to semantically incongruent audio-visual (AV) stimulation. To account for these differential effects, we will suggest in the final section of this paper a possible synthesis of these data on semantic modulation of AV integration with findings from neuroimaging studies and theoretical accounts of semantic memory.

The McGurk-effect (McGurk and MacDonald, 1976) is a robust illusion which is broadly studied in the context of audiovisual integration. In the illusion, auditory speech perception is modified by discrepant visual lip-movements when presented synchronously, leading to a third, not physically present percept. There is an ongoing debate as to whether the percept is integrated early in the sensory system or if it is susceptible to cognitive intervention. Here, the McGurk-effect was studied using gender congruency (face/voice stimuli) as the cognitive intervention to address the question of integration. We first investigated changes in prevalence and reaction times due to perceived gender discrepancy during the McGurk-percept. In the second experiment, neural correlates of the gender discrepant McGurk-percept were studied using fMRI. We did not find differences in reaction times or accuracy between gender congruent or incongruent stimuli. Thus it appears the McGurk-percept itself is unaffected by gender incongruent stimuli on a behavioural level. In the fMRI experiment, we found the superior temporal gyrus to reveal a significantly increased activity when the McGurk-effect is perceived as compared to non-McGurk trials, but no effect in this area was found for gender congruency. However, a 2 × 2 whole brain ANOVA revealed a significant interaction in the face identity processing area (fusiform gyrus) as well as inferior parietal gyrus and superior colliculus. We suggest a mechanism, allowing a stable illusory percept in the gender discrepant McGurk-percept to be facilitated. To our knowledge this is the first study to demonstrate on a neural level that gender congruency affects processing of the McGurk-effect.

To date, noninvasive neuroimaging research on multisensory perception has focused on cortical activations. In a series of elegant functional magnetic resonance imaging experiments, Beauchamp and Ro recently investigated altered cortical activations associated with acquired sound–touch synesthesia resulting from a thalamic lesion. Their findings highlight the important role of intact thalamocortical projections for preventing illusory crossmodal perception and for underlying reliable multisensory integration.

The processing of audiovisual information is an important aspect of object recognition since many objects are characterized by visual and auditory features. The images and sounds of everyday life objects are often associated with positive or negative emotions. There is evidence that emotional valence modulates the neural processing of object images and sounds presented in isolation. However, the impact of emotional valence on the neural processing of audiovisual objects is unknown. The aim of the present study was to investigate the influence of evaluative conditioning on cortical plasticity in human audiovisual object processing. We used functional magnetic resonance imaging (fMRI) to analyze changes in cortical audiovisual processing of unfamiliar artificial objects and sounds in naive subjects (Pre fMRI) and after a behavioral evaluative conditioning session in which subjects acquired positive and negative emotional associations with the artificial stimuli (Post fMRI). Results from the Post fMRI showed that additional brain regions were recruited for the processing of positively and negatively conditioned stimuli compared to the Pre fMRI data. The processing of positive stimuli additionally involved the left caudate nucleus and the right medial prefrontal cortex whereas the processing of negative stimuli additionally recruited the left insula. Furthermore, a congruence effect for matching emotional stimulus pairs was found in the right transversal temporal gyrus. These results indicate that emotional valence modulates object-related audiovisual processing. Moreover, we were able to show that emotional valence can be induced by a behavioral evaluative conditioning procedure leading to plastic changes in the processing of audiovisual stimuli.

Moving dots can evoke a percept of the spatial structure of a three-dimensional object in the absence of other visual cues. This phenomenon, called structure from motion (SFM), suggests that the motion flowfield represented in the dorsal stream can form the basis of object recognition performed in the ventral stream. SFM processing is likely to contribute to object perception whenever there is relative motion between the observer and the object viewed. Here we investigate the motion flowfield component of object recognition with functional magnetic resonance imaging. Our SFM stimuli encoded face surfaces and random three-dimensional control shapes with matched curvature properties. We used two different types of an SFM stimulus with the dots either fixed to the surface of the object or moving on it. Despite the radically different encoding of surface structure in the two types of SFM, both elicited strong surface percepts and involved the same network of cortical regions. From early visual areas, this network extends dorsally into the human motion complex and parietal regions and ventrally into object-related cortex. The SFM stimuli elicited a face-selective response in the fusiform face area. The human motion complex appears to have a central role in SFM object recognition, not merely representing the motion flowfield but also the surface structure of the motion-defined object. The motion complex and a region in the intraparietal sulcus reflected the motion state of the SFM-implicit object, responding more strongly when the implicit object was in motion than when it was stationary.