Search

Introduction: The prevailing theories of consciousness consider the integration of different sensory stimuli as a key component for this phenomenon to rise on the brain level. Despite many theories and models have been proposed for multisensory integration between supraliminal stimuli (e.g., the optimal integration model), we do not know if multisensory integration occurs also for subliminal stimuli and what psychophysical mechanisms it follows. Methods: To investigate this, subjects were exposed to visual (Virtual Reality) and/or haptic stimuli (Electro-Cutaneous Stimulation) above or below their perceptual threshold. They had to discriminate, in a two-Alternative Forced Choice Task, the intensity of unimodal and/or bimodal stimuli. They were then asked to discriminate the sensory modality while recording their EEG responses. Results: We found evidence of multisensory integration for supraliminal condition, following the classical optimal model. Importantly, even for subliminal trials participant's performances in the bimodal condition were significantly more accurate when discriminating the intensity of the stimulation. Moreover, significant differences emerged between unimodal and bimodal activity templates in parietotemporal areas known for their integrative role. Discussion: These converging evidences - even if preliminary and needing confirmation from the collection of further data - suggest that subliminal multimodal stimuli can be integrated, thus filling a meaningful gap in the debate about the relationship between consciousness and multisensory integration. [ABSTRACT FROM AUTHOR]

IntroductionThe prevailing theories of consciousness consider the integration of different sensory stimuli as a key component for this phenomenon to rise on the brain level. Despite many theories and models have been proposed for multisensory integration between supraliminal stimuli (e.g., the optimal integration model), we do not know if multisensory integration occurs also for subliminal stimuli and what psychophysical mechanisms it follows.MethodsTo investigate this, subjects were exposed to visual (Virtual Reality) and/or haptic stimuli (Electro-Cutaneous Stimulation) above or below their perceptual threshold. They had to discriminate, in a two-Alternative Forced Choice Task, the intensity of unimodal and/or bimodal stimuli. They were then asked to discriminate the sensory modality while recording their EEG responses.ResultsWe found evidence of multisensory integration for supraliminal condition, following the classical optimal model. Importantly, even for subliminal trials participant’s performances in the bimodal condition were significantly more accurate when discriminating the intensity of the stimulation. Moreover, significant differences emerged between unimodal and bimodal activity templates in parieto-temporal areas known for their integrative role.DiscussionThese converging evidences - even if preliminary and needing confirmation from the collection of further data - suggest that subliminal multimodal stimuli can be integrated, thus filling a meaningful gap in the debate about the relationship between consciousness and multisensory integration.

Abstract Individuals with Anorexia Nervosa and Bulimia Nervosa present alterations in the way they experience their bodies. Body experience results from a multisensory integration process in which information from different sensory domains and spatial reference frames is combined into a coherent percept. Given the critical role of the body in the onset and maintenance of both Anorexia Nervosa and Bulimia Nervosa, we conducted a systematic review to examine multisensory integration abilities of individuals affected by these two conditions and investigate whether they exhibit impairments in crossmodal integration. We searched for studies evaluating crossmodal integration in individuals with a current diagnosis of Anorexia Nervosa and Bulimia Nervosa as compared to healthy individuals from both behavioral and neurobiological perspectives. A search of PubMed, PsycINFO, and Web of Sciences databases was performed to extract relevant articles. Of the 2348 studies retrieved, 911 were unique articles. After the screening, 13 articles were included. Studies revealed multisensory integration abnormalities in patients affected by Anorexia Nervosa; only one included individuals with Bulimia Nervosa and observed less severe impairments compared to healthy controls. Overall, results seemed to support the presence of multisensory deficits in Anorexia Nervosa, especially when integrating interoceptive and exteroceptive information. We proposed the Predictive Coding framework for understanding our findings and suggested future lines of investigation.

Individuals with Anorexia Nervosa and Bulimia Nervosa present alterations in the way they experience their bodies. Body experience results from a multisensory integration process in which information from different sensory domains and spatial reference frames is combined into a coherent percept. Given the critical role of the body in the onset and maintenance of both Anorexia Nervosa and Bulimia Nervosa, we conducted a systematic review to examine multisensory integration abilities of individuals affected by these two conditions and investigate whether they exhibit impairments in crossmodal integration. We searched for studies evaluating crossmodal integration in individuals with a current diagnosis of Anorexia Nervosa and Bulimia Nervosa as compared to healthy individuals from both behavioral and neurobiological perspectives. A search of PubMed, PsycINFO, and Web of Sciences databases was performed to extract relevant articles. Of the 2348 studies retrieved, 911 were unique articles. After the screening, 13 articles were included. Studies revealed multisensory integration abnormalities in patients affected by Anorexia Nervosa; only one included individuals with Bulimia Nervosa and observed less severe impairments compared to healthy controls. Overall, results seemed to support the presence of multisensory deficits in Anorexia Nervosa, especially when integrating interoceptive and exteroceptive information. We proposed the Predictive Coding framework for understanding our findings and suggested future lines of investigation. Plain English summary: Diagnoses of Anorexia Nervosa and Bulimia Nervosa have increased dramatically in recent years, taking on the characteristics of a mental health emergency. More research is therefore needed to better understand these conditions, especially given their complex and multifaceted nature. Patients affected by these conditions report significant alterations in body-self experience. Body experience results from a cross-modal integration process in which information from different sensory modalities and spatial frames is combined. Therefore, we systematically reviewed studies that focused on multisensory integration in patients affected by Anorexia Nervosa and Bulimia Nervosa, as it may play a key role in the onset and maintenance of these pathologies. Studies in this review found multisensory integration difficulties in patients with Anorexia Nervosa, but not enough studies were retrieved to draw conclusions regarding Bulimia Nervosa. We discussed findings trying to link behavioral, neuropsychological, and neuroscientific evidence in light of the predictive coding framework to provide a different perspective on patients' distorted body experiences. This may lead to new insights to refine our understanding of these complex and poorly understood disorders. [ABSTRACT FROM AUTHOR]

Processing ambiguous situations is a constant challenge in everyday life and sensory input from different modalities needs to be integrated to form a coherent mental representation on the environment. The bouncing/streaming illusion can be studied to provide insights into the ambiguous perception and processing of multi-modal environments. In short, the likelihood of reporting bouncing rather than streaming impressions increases when a sound coincides with the moment of overlap between two moving disks. Neuroimaging studies revealed that the right posterior parietal cortex is crucial in cross-modal integration and is active during the bouncing/streaming illusion. Consequently, in the present study, we used transcranial direct current stimulation to stimulate this brain area. In the active stimulation conditions, a 9 cm2 electrode was positioned over the P4-EEG position and the 35 cm2 reference positioned over the left upper arm. The stimulation lasted 15 min. Each participant did the bouncing/streaming task three times: before, during and after anodal or sham stimulation. In a sample of N = 60 healthy, young adults, we found no influence of anodal tDCS. Bayesian analysis showed strong evidence against tDCS effects. There are two possible explanations for the finding that anodal tDCS over perceptual areas did not modulate multimodal integration. First, upregulation of multimodal integration is not possible using tDCS over the PPC as the integration process already functions at maximum capacity. Second, prefrontal decision-making areas may have overruled any modulated input from the PPC as it may not have matched their decision-making criterion and compensated for the modulation. [ABSTRACT FROM AUTHOR]

Timbre is an integral dimension of musical sound quality, and people accumulate knowledge about timbre of sounds generated by various musical instruments throughout their life. Recent studies have proposed the possibility that musical sound is crossmodally integrated with visual information related to the sound. However, little is known about the influence of visual information on musical timbre perception. The present study investigated the automaticity of crossmodal integration between musical timbre and visual image of hands playing musical instruments. In the experiment, an image of hands playing piano or violin, or a control scrambled image was presented to participants unconsciously. Simultaneously, participants heard intermediate sounds synthesised by morphing piano and violin sounds with the same note. The participants answered whether the musical tone sounded like piano or violin. The results revealed that participants were more likely to perceive violin sound when an image of a violin was presented unconsciously than when playing piano was presented. This finding indicates that timbral perception of musical sound is influenced by visual information of musical performance without conscious awareness, supporting the automaticity of crossmodal integration in musical timbre perception. [ABSTRACT FROM AUTHOR]

Although emotional audiovisual integration has been investigated previously, whether emotional audiovisual integration is affected by the spatial allocation of visual attention is currently unknown. To examine this question, a variant of the exogenous spatial cueing paradigm was adopted, in which stimuli varying by facial expressions and nonverbal affective prosody were used to express six basic emotions (happiness, anger, disgust, sadness, fear, surprise) via a visual, an auditory, or an audiovisual modality. The emotional stimuli were preceded by an unpredictive cue that was used to attract participants' visual attention. The results showed significantly higher accuracy and quicker response times in response to bimodal audiovisual stimuli than to unimodal visual or auditory stimuli for emotional perception under both valid and invalid cue conditions. The auditory facilitation effect was stronger than the visual facilitation effect under exogenous attention for the six emotions tested. Larger auditory enhancement was induced when the target was presented at the expected location than at the unexpected location. For emotional perception, happiness shared the biggest auditory enhancement among all six emotions. However, the influence of exogenous cueing effect on emotional perception seemed to be absent. [ABSTRACT FROM AUTHOR]

Human infants are able to acquire natural language seemingly easily at an early age. Their language learning seems to occur simultaneously with learning other cognitive functions as well as with playful interactions with the environment and caregivers. From a neuroscientific perspective, natural language is embodied, grounded in most, if not all, sensory and sensorimotor modalities, and acquired by means of crossmodal integration. However, characterizing the underlying mechanisms in the brain is difficult and explaining the grounding of language in crossmodal perception and action remains challenging. In this paper, we present a neurocognitive model for language grounding which reflects bio-inspired mechanisms such as an implicit adaptation of timescales as well as end-to-end multimodal abstraction. It addresses developmental robotic interaction and extends its learning capabilities using larger-scale knowledge-based data. In our scenario, we utilize the humanoid robot NICO in obtaining the EMIL data collection, in which the cognitive robot interacts with objects in a children's playground environment while receiving linguistic labels from a caregiver. The model analysis shows that crossmodally integrated representations are sufficient for acquiring language merely from sensory input through interaction with objects in an environment. The representations self-organize hierarchically and embed temporal and spatial information through composition and decomposition. This model can also provide the basis for further crossmodal integration of perceptually grounded cognitive representations. [ABSTRACT FROM AUTHOR]

A single flash accompanied by two auditory beeps tends to be perceived as two flashes (Shams et al. Nature 408:788, 2000, Cogn Brain Res 14:147–152, 2002). This phenomenon is known as ‘sound-induced flash illusion.’ Previous neuroimaging studies have shown that this illusion is correlated with modulation of activity in early visual cortical areas (Arden et al. Vision Res 43(23):2469–2478, 2003; Bhattacharya et al. NeuroReport 13:1727–1730, 2002; Shams et al. NeuroReport 12(17):3849–3852, 2001, Neurosci Lett 378(2):76–81, 2005; Watkins et al. Neuroimage 31:1247–1256, 2006, Neuroimage 37:572–578, 2007; Mishra et al. J Neurosci 27(15):4120–4131, 2007). We examined how robust the illusion is by testing whether the frequency of the illusion can be reduced by providing feedback. We found that the sound-induced flash illusion was resistant to feedback training, except when the amount of monetary reward was made dependent on accuracy in performance. However, even in the latter case the participants reported that they still perceived illusory two flashes even though they correctly reported single flash. Moreover, the feedback training effect seemed to disappear once the participants were no longer provided with feedback suggesting a short-lived refinement of discrimination between illusory and physical double flashes rather than vanishing of the illusory percept. These findings indicate that the effect of sound on the perceptual representation of visual stimuli is strong and robust to feedback training, and provide further evidence against decision factors accounting for the sound-induced flash illusion.

Phonological awareness skills in children with reading difficulty (RD) may reflect impaired automatic integration of orthographic and phonological representations. However, little is known about the underlying neural mechanisms involved in phonological awareness for children with RD. Eighteen children with RD, ages 9–13, participated in a functional magnetic resonance imaging (fMRI) study designed to assess the relationship of two constructs of phonological awareness, phoneme synthesis, and phoneme analysis, with crossmodal rhyme judgment. Participants completed a rhyme judgment task presented in two modality conditions; unimodal auditory only and crossmodal audiovisual. Measures of phonological awareness were correlated with unimodal, but not crossmodal, lexical processing. Moreover, these relationships were found only in unisensory brain regions, and not in multisensory brain areas. The results of this study suggest that children with RD rely on unimodal representations and unisensory brain areas, and provide insight into the role of phonemic awareness in mapping between auditory and visual modalities during literacy acquisition.

Phonological awareness skills in children with reading difficulty (RD) may reflect impaired automatic integration of orthographic and phonological representations. However, little is known about the underlying neural mechanisms involved in phonological awareness for children with RD. Eighteen children with RD, ages 9–13, participated in a functional magnetic resonance imaging (fMRI) study designed to assess the relationship of two constructs of phonological awareness, phoneme synthesis, and phoneme analysis, with crossmodal rhyme judgment. Participants completed a rhyme judgment task presented in two modality conditions; unimodal auditory only and crossmodal audiovisual. Measures of phonological awareness were correlated with unimodal, but not crossmodal, lexical processing. Moreover, these relationships were found only in unisensory brain regions, and not in multisensory brain areas. The results of this study suggest that children with RD rely on unimodal representations and unisensory brain areas, and provide insight into the role of phonemic awareness in mapping between auditory and visual modalities during literacy acquisition. [ABSTRACT FROM AUTHOR]

Bayesian cognitive modeling has become a prominent tool for the cognitive sciences aiming at a deeper understanding of the human mind and applications in cognitive systems, e.g., humanoid or wearable robotics. Such approaches can capture human behavior adequately with a focus on the crossmodal processing of sensory information. The rubber foot illusion is a paradigm in which such integration is relevant. After experimental stimulation, many participants perceive their real limb closer to an artificial replicate than it actually is. A measurable effect of this recalibration on localization is called the proprioceptive drift. We investigate whether the Bayesian causal inference model can estimate the proprioceptive drift observed in empirical studies. Moreover, we juxtapose two models employing informed prior distributions on limb location against an existing model assuming uniform prior distribution. The model involving empirically informed prior information yields better predictions of the proprioceptive drift regarding the rubber foot illusion when evaluated with separate experimental data. Contrary, the uniform model produces implausibly narrow position estimates that seem due to the precision ratio between the contributing sensory channels. We conclude that an informed prior on limb localization is a plausible and necessary modification to the Bayesian causal inference model when applied to limb illusions. Future research could overcome the remaining discrepancy between model predictions and empirical observation by investigating the changes in sensory precision as a function of distance between the eyes and respective limbs. [ABSTRACT FROM AUTHOR]

Recent research on the crossmodal integration of visual and auditory perception suggests that evaluations of emotional information in one sensory modality may tend toward the emotional value generated in another sensory modality. This implies that the emotions elicited by musical stimuli can influence the perception of emotional stimuli presented in other sensory modalities, through a top-down process. The aim of this work was to investigate how crossmodal perceptual processing influences emotional face recognition and how potential modulation of this processing induced by music could be influenced by the subject's musical competence. We investigated how emotional face recognition processing could be modulated by listening to music and how this modulation varies according to the subjective emotional salience of the music and the listener's musical competence. The sample consisted of 24 participants: 12 professional musicians and 12 university students (non-musicians). Participants performed an emotional go/no-go task whilst listening to music by Albeniz, Chopin, or Mozart. The target stimuli were emotionally neutral facial expressions. We examined the N170 Event-Related Potential (ERP) and behavioral responses (i.e., motor reaction time to target recognition and musical emotional judgment). A linear mixed-effects model and a decision-tree learning technique were applied to N170 amplitudes and latencies. The main findings of the study were that musicians' behavioral responses and N170 is more affected by the emotional value of music administered in the emotional go/no-go task and this bias is also apparent in responses to the non-target emotional face. This suggests that emotional information, coming from multiple sensory channels, activates a crossmodal integration process that depends upon the stimuli emotional salience and the listener's appraisal.

Introduction: Decoding emotional information from faces and voices is crucial for efficient interpersonal communication. Emotional decoding deficits have been found in alcohol-dependence (ALC), particularly in crossmodal situations (with simultaneous stimulations from different modalities), but are still underexplored in Korsakoff syndrome (KS). The aim of this study is to determine whether the continuity hypothesis, postulating a gradual worsening of cognitive and brain impairments from ALC to KS, is valid for emotional crossmodal processing. Methods: Sixteen KS, 17 ALC and 19 matched healthy controls (CP) had to detect the emotion (anger or happiness) displayed by auditory, visual or crossmodal auditory-visual stimuli. Crossmodal stimuli were either emotionally congruent (leading to a facilitation effect, i.e. enhanced performance for crossmodal condition compared to unimodal ones) or incongruent (leading to an interference effect, i.e. decreased performance for crossmodal condition due to discordant information across modalities). Reaction times and accuracy were recorded. Results: Crossmodal integration for congruent information was dampened only in ALC, while both ALC and KS demonstrated, compared to CP, decreased performance for decoding emotional facial expressions in the incongruent condition. Conclusions: The crossmodal integration appears impaired in ALC but preserved in KS. Both alcohol-related disorders present an increased interference effect. These results show the interest of more ecological designs, using crossmodal stimuli, to explore emotional decoding in alcohol-related disorders. They also suggest that the continuum hypothesis cannot be generalised to emotional decoding abilities. [ABSTRACT FROM AUTHOR]

Recent research on the crossmodal integration of visual and auditory perception suggests that evaluations of emotional information in one sensory modality may tend toward the emotional value generated in another sensory modality. This implies that the emotions elicited by musical stimuli can influence the perception of emotional stimuli presented in other sensory modalities, through a top-down process. The aim of this work was to investigate how crossmodal perceptual processing influences emotional face recognition and how potential modulation of this processing induced by music could be influenced by the subject's musical competence. We investigated how emotional face recognition processing could be modulated by listening to music and how this modulation varies according to the subjective emotional salience of the music and the listener's musical competence. The sample consisted of 24 participants: 12 professional musicians and 12 university students (non-musicians). Participants performed an emotional go/no-go task whilst listening to music by Albeniz, Chopin, or Mozart. The target stimuli were emotionally neutral facial expressions. We examined the N170 Event-Related Potential (ERP) and behavioral responses (i.e., motor reaction time to target recognition and musical emotional judgment). A linear mixed-effects model and a decision-tree learning technique were applied to N170 amplitudes and latencies. The main findings of the study were that musicians' behavioral responses and N170 is more affected by the emotional value of music administered in the emotional go/no-go task and this bias is also apparent in responses to the non-target emotional face. This suggests that emotional information, coming from multiple sensory channels, activates a crossmodal integration process that depends upon the stimuli emotional salience and the listener's appraisal. [ABSTRACT FROM AUTHOR]

The integration of information has been considered a hallmark of human consciousness, as it requires information being globally available via widespread neural interactions. Yet the complex interdependencies between multisensory integration and perceptual awareness, or consciousness, remain to be defined. While perceptual awareness has traditionally been studied in a single sense, in recent years we have witnessed a surge of interest in the role of multisensory integration in perceptual awareness. Based on a recent IMRF symposium on multisensory awareness, this review discusses three key questions from conceptual, methodological and experimental perspectives: (1) What do we study when we study multisensory awareness? (2) What is the relationship between multisensory integration and perceptual awareness? (3) Which experimental approaches are most promising to characterize multisensory awareness? We hope that this review paper will provoke lively discussions, novel experiments, and conceptual considerations to advance our understanding of the multifaceted interplay between multisensory integration and consciousness. [ABSTRACT FROM AUTHOR]

Metacognition – the ability to monitor one's own decisions and representations, their accuracy and uncertainty – is considered a hallmark of intelligent behavior. Little is known about metacognition in our natural multisensory environment. To form a coherent percept, the brain should integrate signals from a common cause but segregate those from independent causes. Multisensory perception thus relies on inferring the world's causal structure, raising new challenges for metacognition. We discuss the extent to which observers can monitor their uncertainties not only about their final integrated percept but also about the individual sensory signals and the world's causal structure. The latter causal metacognition highlights fundamental links between perception and other cognitive domains such as social and abstract reasoning. [ABSTRACT FROM AUTHOR]

This study investigated whether samples of pleasant, neutral and unpleasant music can impact food perception. To this end, the pleasantness of three different types of chocolate gelati (milk chocolate, dark chocolate and bittersweet chocolate) was determined using 50 participants exposed to silence (the reference condition) and three music samples differing in self-rated preference. To measure hedonic responses to the gelati samples, the Time Intensity method was utilized to derive the maximum intensity of pleasantness and the area under the Time Intensity pleasantness curve. The presence of non-preferred music significantly decreased the pleasantness ratings of all three types of chocolate gelati tested, while preferred music increased perceived pleasantness ratings of dark and bittersweet chocolate gelati, but not milk chocolate gelato. Time Intensity parameters for pleasantness ratings did not differ significantly across the three different types of chocolate gelati in the silent condition, suggesting that listening to the music influenced gelati pleasantness ratings. This study demonstrated that the pleasantness of gelati changed with music valence. The findings echo previous studies emphasizing the importance of crossmodal effects between music and food perception. [ABSTRACT FROM AUTHOR]

Many human behaviors are known to benefit from audiovisual integration, including language and communication, recognizing individuals, social decision making, and memory. Exceptionally little is known about the contributions of audiovisual integration to behavior in other primates. The current experiment investigated whether short-term memory in nonhuman primates is facilitated by the audiovisual presentation format. Three macaque monkeys that had previously learned an auditory delayed matching-to-sample (DMS) task were trained to perform a similar visual task, after which they were tested with a concurrent audiovisual DMS task with equal proportions of auditory, visual, and audiovisual trials. Parallel to outcomes in human studies, accuracy was higher and response times were faster on audiovisual trials than either unisensory trial type. Unexpectedly, two subjects exhibited superior unimodal performance on auditory trials, a finding that contrasts with previous studies, but likely reflects their training history. Our results provide the first demonstration of a bimodal memory advantage in nonhuman primates, lending further validation to their use as a model for understanding audiovisual integration and memory processing in humans. [ABSTRACT FROM AUTHOR]

The current research uses eye-tracking technology in a consumer context to explore the interactive effects of olfactory and visual cues on consumers' eye gaze patterns. We manipulate the semantic correspondence between pictorial objects depicted in print advertisements and odors smelled (or not) while looking at the ads. The results indicate that smelling a scent that shares learned semantic associations with an object in the advertisement diverts consumers' eye gazes to the semantically related object in the ad, with positive downstream effects on advertising recall and purchase intent. This is the first study we are aware of demonstrating multisensory integration of odors and pictures on consumer eye gaze patterns with clear implications for consumer choice. Copyright © 2015 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Suppose that you are at a live jazz show. The drummer begins a solo. You see the cymbal jolt and you hear the clang. But in addition seeing the cymbal jolt and hearing the clang, you are also aware that the jolt and the clang are part of the same event. Casey O’Callaghan (forthcoming) calls this awareness intermodal feature binding awareness. Psychologists have long assumed that multimodal perceptions such as this one are the result of a subpersonal feature binding mechanism (see Vatakis and Spence, 2007, Kubovy and Schutz, 2010, Pourtois et al., 2000, and Navarra et al., 2012). I present new evidence against this. I argue that there is no automatic feature binding mechanism that couples features like the jolt and the clang together. Instead, when you experience the jolt and the clang as part of the same event, this is the result of an associative learning process. The cymbal’s jolt and the clang are best understood as a single learned perceptual unit, rather than as automatically bound. I outline the specific learning process in perception called unitization, whereby we come to chunk the world into multimodal units. Unitization has never before been applied to multimodal cases. Yet I argue that this learning process can do the same work that intermodal binding would do, and that this issue has important philosophical implications. Specifically, whether we take multimodal cases to involve a binding mechanism or an associative process will have impact on philosophical issues from Molyneux’s question to the question of how active or passive we consider perception to be.

Capacity limitations of attentional resources allow only a fraction of sensory inputs to enter our awareness. Most prominently, in the attentional blink the observer often fails to detect the second of two rapidly successive targets that are presented in a sequence of distractor items. To investigate how auditory inputs enable a visual target to escape the attentional blink, this study presented the visual letter targets T1 and T2 together with phonologically congruent or incongruent spoken letter names. First, a congruent relative to an incongruent sound at T2 rendered visual T2 more visible. Second, this T2 congruency effect was amplified when the sound was congruent at T1 as indicated by a T1 congruency x T2 congruency interaction. Critically, these effects were observed both when the sounds were presented in synchrony with and prior to the visual target letters suggesting that the sounds may increase visual target identification via multiple mechanisms such as audiovisual priming or decisional interactions.Our results demonstrate that a sound around the time of T2 increases subjects’ awareness of the visual target as a function of T1 and T2 congruency. Consistent with Bayesian causal inference, the brain may thus combine (1) prior congruency expectations based on T1 congruency and (2) phonological congruency cues provided by the audiovisual inputs at T2 to infer whether auditory and visual signals emanate from a common source and should hence be integrated for perceptual decisions.

The brain generates a representation of our environment by integrating signals from a common source, but segregating signals from different sources. This fMRI study investigated how the brain arbitrates between perceptual integration and segregation based on top-down congruency expectations and bottom-up stimulus-bound congruency cues. Participants were presented audiovisual movies of phonologically congruent, incongruent or McGurk syllables that can be integrated into an illusory percept (e.g. “ti” percept for visual «ki» with auditory /pi/). They reported the syllable they perceived. Critically, we manipulated participants' top-down congruency expectations by presenting McGurk stimuli embedded in blocks of congruent or incongruent syllables. Behaviorally, participants were more likely to fuse audiovisual signals into an illusory McGurk percept in congruent than incongruent contexts. At the neural level, the left inferior frontal sulcus (lIFS) showed increased activations for bottom-up incongruent relative to congruent inputs. Moreover, lIFS activations were increased for physically identical McGurk stimuli, when participants segregated the audiovisual signals and reported their auditory percept. Critically, this activation increase for perceptual segregation was amplified when participants expected audiovisually incongruent signals based on prior sensory experience. Collectively, our results demonstrate that the lIFS combines top-down prior (in)congruency expectations with bottom-up (in)congruency cues to arbitrate between multisensory integration and segregation. [ABSTRACT FROM AUTHOR]

Voices provide a rich source of information that is important for identifying individuals and for social interaction. During search for a face in a crowd, voices often accompany visual information, and they facilitate localization of the sought-after individual. However, it is unclear whether this facilitation occurs primarily because the voice cues the location of the face or because it also increases the salience of the associated face. Here we demonstrate that a voice that provides no location information nonetheless facilitates visual search for an associated face. We trained novel face-voice associations and verified learning using a two-alternative forced choice task in which participants had to correctly match a presented voice to the associated face. Following training, participants searched for a previously learned target face among other faces while hearing one of the following sounds (localized at the center of the display): a congruent learned voice, an incongruent but familiar voice, an unlearned and unfamiliar voice, or a time-reversed voice. Only the congruent learned voice speeded visual search for the associated face. This result suggests that voices facilitate the visual detection of associated faces, potentially by increasing their visual salience, and that the underlying crossmodal associations can be established through brief training. [ABSTRACT FROM AUTHOR]

Capacity limitations of attentional resources allow only a fraction of sensory inputs to enter our awareness. Most prominently, in the attentional blink the observer often fails to detect the second of two rapidly successive targets that are presented in a sequence of distractor items. To investigate how auditory inputs enable a visual target to escape the attentional blink, this study presented the visual letter targets T1 and T2 together with phonologically congruent or incongruent spoken letter names. First, a congruent relative to an incongruent sound at T2 rendered visual T2 more visible. Second, this T2 congruency effect was amplified when the sound was congruent at T1 as indicated by a T1 congruency × T2 congruency interaction. Critically, these effects were observed both when the sounds were presented in synchrony with and prior to the visual target letters suggesting that the sounds may increase visual target identification via multiple mechanisms such as audiovisual priming or decisional interactions. Our results demonstrate that a sound around the time of T2 increases subjects' awareness of the visual target as a function of T1 and T2 congruency. Consistent with Bayesian causal inference, the brain may thus combine (1) prior congruency expectations based on T1 congruency and (2) phonological congruency cues provided by the audiovisual inputs at T2 to infer whether auditory and visual signals emanate from a common source and should hence be integrated for perceptual decisions. [ABSTRACT FROM AUTHOR]

Highlights: [•] The “Bouba–Kiki” effect is thought to be universal for visual/auditory associations. [•] This study shows a robust effect for haptic/auditory associations in sighted people. [•] The effect was significantly weaker in blind and partially-sighted participants. [•] It suggests that a haptic/auditory “Bouba–Kiki” effect is not independent of vision. [•] Even in non-visual modalities, visual imagery plays a role in crossmodal integration. [ABSTRACT FROM AUTHOR]

Visual and auditory temporal processing and crossmodal integration are crucial factors in the word decoding process. The speed of processing (SOP) gap (Asynchrony) between these two modalities, which has been suggested as related to the dyslexia phenomenon, is the focus of the current study. Nineteen dyslexic and 17 non-impaired University adult readers were given stimuli in a reaction time (RT) procedure where participants were asked to identify whether the stimulus type was only visual, only auditory or crossmodally integrated. Accuracy, RT, and Event Related Potential (ERP) measures were obtained for each of the three conditions. An algorithm to measure the contribution of the temporal SOP of each modality to the crossmodal integration in each group of participants was developed. Results obtained using this model for the analysis of the current study data, indicated that in the crossmodal integration condition the presence of the auditory modality at the pre-response time frame (between 170 and 240ms after stimulus presentation), increased processing speed in the visual modality among the non-impaired readers, but not in the dyslexic group. The differences between the temporal SOP of the modalities among the dyslexics and the non-impaired readers give additional support to the theory that an asynchrony between the visual and auditory modalities is a cause of dyslexia. [ABSTRACT FROM AUTHOR]

Abstract: The ability to localize nociceptive stimuli on the body surface is essential for an organism to respond appropriately to potential physical threats. This ability not only requires a representation of the space of the observer׳s body, but also of the external space with respect to their body. Therefore, localizing nociceptive stimuli requires coordinating multiple senses into an integrated frame of reference. The peripersonal frame of reference allows for the coding of the position of somatosensory stimuli on the body surface and the position of stimuli occurring close to the body (e.g., visual stimuli). Intensively studied for touch, this topic has been largely ignored when it comes to nociception. Here, we investigated, using a temporal order judgment task, whether the spatial perception of nociceptive stimuli is coordinated with that of proximal visual stimuli into an integrated representation of peripersonal space. Participants judged which of two nociceptive stimuli, one presented to either hand, had been presented first. Each pair of nociceptive stimuli was preceded by lateralized visual cues presented either unilaterally or bilaterally, and either close to, or far from, the participant׳s body. The perception of nociceptive stimuli was biased in favor of the stimulus delivered on the hand adjacent to the unilateral visual cue, especially when the cue was presented near the participant׳s hand. These results therefore suggest that a peripersonal frame of reference is used to map the position of nociceptive stimuli in multisensory space. We propose that peripersonal space constitutes a kind of margin of safety around the body to alert an organism to possible threats. [Copyright &y& Elsevier]

How do different sources of information arising from different modalities interact to control where we look? To answer this question with respect to real-world operational conditions we presented natural images and spatially localized sounds in (V)isual, Audio-visual (AV) and (A)uditory conditions and measured subjects' eye-movements. Our results demonstrate that eye-movements in AV conditions are spatially biased towards the part of the image corresponding to the sound source. Interestingly, this spatial bias is dependent on the probability of a given image region to be fixated (saliency) in the V condition. This indicates that fixation behaviour during the AV conditions is the result of an integration process. Regression analysis shows that this integration is best accounted for by a linear combination of unimodal saliencies.

Abstract: Recent studies have shown the importance of integrating multisensory information in the body representation for constituting self-consciousness. However, one idea that has received only scant attention is that our body representation is also constituted by knowledge of bodily visual characteristics (i.e. ‘what I look like’). Here in two experiments we used a full body crossmodal congruency task in which visual distractors were presented on a photograph of the participant, another person, who was either familiar or unfamiliar, or an object. Results revealed that during the ‘self-condition’ CCEs were enhanced compared to the ‘other condition’. The CCE was similar for unfamiliar and familiar others. CCEs for the object condition were significantly smaller. The results show that presentation of an irrelevant image of a body affects multimodal processing and that the effect is enhanced when that image is of the self. The results hold intriguing implications for body representation in social situations. [Copyright &y& Elsevier]

Background: Coloured-hearing (CH) synesthesia is a perceptual phenomenon in which an acoustic stimulus(the inducer) initiates a concurrent colour perception (the concurrent). Individuals with CH synesthesia "see" colours when hearing tones, words, or music; this specific phenomenon suggesting a close relationship between auditory and visual representations. To date, it is still unknown whether the perception of colours is associated with a modulation of brain functions in the inducing brain area, namely in the auditory-related cortex and associated brain areas. In addition, there is an on-going debate as to whether attention to the inducer is necessarily required for eliciting a visual concurrent, or whether the latter can emerge in a pre-attentive fashion. Results: By using the EEG technique in the context of a pre-attentive mismatch negativity (MMN) paradigm, we show that the binding of tones and colours in CH synesthetes is associated with increased MMN amplitudes in response to deviant tones supposed to induce novel concurrent colour perceptions. Most notably, the increased MMN amplitudes we revealed in the CH synesthetes were associated with stronger intracerebral current densities originating from the auditory cortex, parietal cortex, and ventral visual areas. Conclusions: The automatic binding of tones and colours in CH synesthetes is accompanied by an early pre-attentive process recruiting the auditory cortex, inferior and superior parietal lobules, as well as ventral occipital areas. [ABSTRACT FROM AUTHOR]

Abstract: Multisensory processing involving visual and auditory inputs is modulated by their relative temporal offsets. In order to assess whether multisensory integration alters the activation timing of primary visual and auditory cortices as a function of the temporal offsets between auditory and visual stimuli, a task was designed in which subjects had to judge the perceptual simultaneity of the onset of visual stimuli and brief acoustic tones. These were presented repeatedly with three different inter-stimulus intervals that were chosen to meet three perceptual conditions: (1) physical synchrony perceived as synchrony by subjects (SYNC); (2) physical asynchrony perceived as asynchrony (ASYNC); (3) physical asynchrony perceived ambiguously (AMB, i.e. 50% perceived as synchrony, 50% as asynchrony). Magnetoencephalographic activity was recorded during crossmodal sessions and unimodal control sessions. The activation of primary visual and auditory cortices peaked at a longer latency for the crossmodal conditions as compared to the unimodal conditions. Moreover, the latency in the auditory cortex was longer in the SYNC than in the ASYNC condition, whereas in the visual cortex the latency in the AMB condition was longer than in the ASYNC condition. These findings suggest that multisensory processing affects temporal dynamics already in primary cortices, that such activity can differ regionally and can be sensitive to the temporal offsets of multisensory inputs. In addition, in the AMB condition the conscious awareness of asynchrony might be associated to a later activation of the primary auditory cortex. [Copyright &y& Elsevier]

Here, we investigate how audiovisual context affects perceived event duration with experiments in which observers reported which of two stimuli they perceived as longer. Target events were visual and/or auditory and could be accompanied by nontargets in the other modality. Our results demonstrate that the temporal information conveyed by irrelevant sounds is automatically used when the brain estimates visual durations but that irrelevant visual information does not affect perceived auditory duration (Experiment 1). We further show that auditory influences on subjective visual durations occur only when the temporal characteristics of the stimuli promote perceptual grouping (Experiments 1 and 2). Placed in the context of scalar expectancy theory of time perception, our third and fourth experiments have the implication that audiovisual context can lead both to changes in the rate of an internal clock and to temporal ventriloquism-like effects on perceived on- and offsets. Finally, intramodal grouping of auditory stimuli diminished any crossmodal effects, suggesting a strong preference for intramodal over crossmodal perceptual grouping (Experiment 5). [ABSTRACT FROM AUTHOR]

There is not enough information about the interactions between visual and auditory stimuli in biological motion perception. Recent data suggests that there might be a brain area (STSp), which responds both to visual and auditory biological motion stimulation. It is also likely that the intermodal processes in biological motion should tend towards greater interaction and integration of stimuli. In this project, we intended to observe the nature of those interactions. To do so, we used strongly biased bistable visual stimuli. We then compared the visual, auditory and audiovisual conditions in two experiments. The results suggest interaction effects between the different stimuli. The audiovisual condition had the best results, both in the bias reduction and in the proportion of correct answers. [ABSTRACT FROM AUTHOR]

Abstract: By using meaningful stimuli, multisensory research has recently started to investigate the impact of stimulus content on crossmodal integration. Variations in this respect have often been termed as “semantic”. In this paper we will review work related to the question for which tasks the influence of semantic factors has been found and which cortical networks are most likely to mediate these effects. More specifically, the focus of this paper will be on processing of object stimuli presented in the auditory and visual sensory modalities. Furthermore, we will investigate which cortical regions are particularly responsive to experimental variations of content by comparing semantically matching (“congruent”) and mismatching (“incongruent”) experimental conditions. In this context, recent neuroimaging studies point toward a possible functional differentiation of temporal and frontal cortical regions, with the former being more responsive to semantically congruent and the latter to semantically incongruent audio-visual (AV) stimulation. To account for these differential effects, we will suggest in the final section of this paper a possible synthesis of these data on semantic modulation of AV integration with findings from neuroimaging studies and theoretical accounts of semantic memory. [Copyright &y& Elsevier]

The sense of body ownership represents a fundamental aspect of our self-awareness, but is disrupted in many neurological, psychiatric, and psychological conditions that are also characterized by disruption of skin temperature regulation, sometimes in a single limb. We hypothesized that skin temperature in a specific limb could be disrupted by psychologically disrupting the sense of ownership of that limb. In six separate experiments, and by using an established protocol to induce the rubber hand illusion, we demonstrate that skin temperature of the real hand decreases when we take ownership of an artificial counterpart. The decrease in skin temperature is limb-specific: it does not occur in the unstimulated hand, nor in the ipsilateral foot. The effect is not evoked by tactile or visual input per se, nor by simultaneous tactile and visual input per se, nor by a shift in attention toward the experimental side or limb. In fact, taking ownership of an artificial hand slows tactile processing of information from the real hand, which is also observed in patients who demonstrate body disownership after stroke. These findings of psychologically induced limb-specific disruption of temperature regulation provide the first evidence that: taking ownership of an artificial body part has consequences for the real body part; that the awareness of our physical self and the physiological regulation of self are closely linked in a top-down manner; and that cognitive processes that disrupt the sense of body ownership may in turn disrupt temperature regulation in numerous states characterized by both. [ABSTRACT FROM AUTHOR]

Abstract: In Alzheimer''s disease (AD), loss of connectivity in the patient''s brain has been evidenced by a range of electrophysiological and neuroimaging studies. However, few neuropsychological research projects have sought to interpret the cognitive modifications following the appearance of AD in terms of a disconnection syndrome. In this paper, we sought to investigate brain connectivity in AD via the study of a crossmodal effect. More precisely, we examined the integration of auditory and visual speech information (the McGurk effect) in AD patients and matched control subjects. Our results revealed impaired crossmodal integration during speech perception in AD, which was not associated with disturbances in the separate processing of auditory and visual speech stimuli. In conclusion, our data suggest the occurrence of a specific, audio-visual integration deficit in AD, which might be the consequence of a connectivity breakdown and corroborate the observation from other studies of crossmodal deficits between the auditory and visual modalities in this population. [Copyright &y& Elsevier]

Transient auditory stimuli have been shown to influence the perception of ambiguous 2D visual motion displays (the bouncing-disks effect; e.g. Sekuler et al. in Nature 385:308, ). The question addressed here was whether continuous moving auditory stimuli can also influence visual motion perception under the same experimental conditions. In Experiment 1, we used a modification of Sanabria et al.’s (Exp Brain Res 157:537–541, ) paradigm (involving an indirect behavioural measure of the bouncing-disks effect), in which the 2D visual display was presented together with either a brief tone, a continuous moving sound, or in the absence of any form of auditory stimulation. Crucially, the results showed that, together with the effect of the brief tone on bouncing trials, the presence of the continuous moving sound speeded-up participants’ responses on streaming trials as compared to the brief tone or no sound conditions. The results of a second experiment revealed that the effect of the continuous moving sound reported in Experiment 1 was not caused simply by the presence of continuous auditory stimulation per se. [ABSTRACT FROM AUTHOR]

Abstract: This review addresses a fundamental neuroscientific question in food perception: how multimodal features of food are integrated. Much research and conceptualization has emerged related to multisensory integration in vision, audition and somatosensation, while it remains poorly understood and researched within the chemical and mouth feel senses. This review aims to bridge this gap. We discuss the main concepts in the fields of auditory, visual and somatosensory multisensory integration and relate them to oral-sensory (gustatory and somatosensory) and olfactory (orolfactory) interactions. We systematically review the psychophysical literature pertaining to intra- and intermodal interactions related to food perception, while making explicit distinctions between peripheral and central interactions. As the neural bases of crossmodal orolfaction currently are poorly understood, we introduce several plausible neuroscientific models, which provide a framework for further neuroscientific exploration in this area. We are guided by a new meta-analysis of the odor–taste neuroimaging literature, as well as by single-unit, anatomical and psychophysical studies. Finally, we propose strong involvement of recurrent neural networks in multisensory integration and make suggestions for future research. [Copyright &y& Elsevier]

The aim of this study was to establish whether spatial attention triggered by bimodal exogenous cues acts differently as compared to unimodal and crossmodal exogenous cues due to crossmodal integration. In order to investigate this issue, we examined cuing effects in discrimination tasks and compared these effects in a condition wherein a visual target was preceded by both visual and auditory exogenous cues delivered simultaneously at the same side (bimodal cue), with conditions wherein the visual target was preceded by either a visual (unimodal cue) or an auditory cue (crossmodal cue). The results of two experiments revealed that cuing effects on RTs in these three conditions with an SOA of 200 ms had comparable magnitudes. Differences at a longer SOA of 600 ms (inhibition of return for bimodal cues, Experiment 1) disappeared when catch trials were included (in Experiment 2). The current data do not support an additional influence of crossmodal integration on exogenous orienting, but are well in agreement with the existence of a supramodal spatial attention module that allocates attentional resources towards stimulated locations for different sensory modalities. [ABSTRACT FROM AUTHOR]

Abstract: Evaluation of emotional scenes requires integration of information from different modality channels, most frequently from audition and vision. Neither the psychological nor neural basis of auditory–visual interactions during the processing of affect is well understood. In this study, possible interactions in affective processing were investigated via event-related potential (ERP) recordings during simultaneous presentation of affective pictures (from IAPS) and affectively sung notes that either matched or mismatched each other in valence. To examine the role of attention in multisensory affect-integration ERPs were recorded in two different rating tasks (voice affect rating, picture affect rating) as participants evaluated the affect communicated in one of the modalities, while that in the other modality was ignored. Both the behavioral and ERP data revealed some, although non-identical, patterns of cross-modal influences; modulation of the ERP-component P2 suggested a relatively early integration of affective information in the attended picture condition, though only for happy picture–voice pairs. In addition, congruent pairing of sad pictures and sad voice stimuli affected the late positive potential (LPP). Responses in the voice affect rating task were overall more likely to be modulated by the concomitant picture''s affective valence than vice versa. [Copyright &y& Elsevier]

Crossmodal spatial integration between auditory and visual stimuli is a common phenomenon in space perception. The principles underlying such integration have been outlined by neurophysiological and behavioral studies in animals; this study investigated whether the integrative effects observed in animals also apply to humans. In this experiment we systematically varied the spatial disparity (0°, 16°, and 32°) and the temporal interval (0, 100, 200, 300, 400, and 500 ms) between the visual and the auditory stimuli. Normal subjects were required to detect visual stimuli presented below threshold either in unimodal visual conditions or in crossmodal audiovisual conditions. Signal detection measures were used. An enhancement of the perceptual sensitivity (d') for luminance detection was found when the audiovisual stimuli followed a simple spatial and temporal rule, governing multisensory integration at the neuronal level. [ABSTRACT FROM AUTHOR]