Search

Sound localization and identification are challenging in acoustically rich environments. The relation between these two processes is still poorly understood. As natural sound-sources rarely occur exactly simultaneously, we wondered whether the auditory system could identify (“what”) and localize (“where”) two spatially separated sounds with synchronous onsets. While listeners typically report hearing a single source at an average location, one study found that both sounds may be accurately localized if listeners are explicitly being told two sources exist. We here tested whether simultaneous source identification (one vs. two) and localization is possible, by letting listeners choose to make either one or two head-orienting saccades to the perceived location(s). Results show that listeners could identify two sounds only when presented on different sides of the head, and that identification accuracy increased with their spatial separation. Notably, listeners were unable to accurately localize either sound, irrespective of whether one or two sounds were identified. Instead, the first (or only) response always landed near the average location, while second responses were unrelated to the targets. We conclude that localization of synchronous sounds in the absence of prior information is impossible. We discuss that the putative cortical ‘what’ pathway may not transmit relevant information to the ‘where’ pathway. We examine how a broadband interaural correlation cue could help to correctly identify the presence of two sounds without being able to localize them. We propose that the persistent averaging behavior reveals that the ‘where’ system intrinsically assumes that synchronous sounds originate from a single source.Significance StatementIt is poorly understood whether identification (‘what’) of sounds and their localization (‘where’) are inter-related, or independent neural processes. We measured sound-localization responses towards synchronous sounds to examine potential coupling of these processes. We varied the spatial configurations of two sounds and found that although identification improved considerably with larger spatial separation, their localization was unaffected: responses were always directed towards the average location. This shows absence of mutual coupling of information between the ‘what’ and ‘where’ streams in the auditory system. We also show how broadband interaural correlation could explain the improved identification results, without affecting localization performance, and explain how the persistent spatial averaging could be understood from strong internal priors regarding sound synchronicity.

The ventriloquism aftereffect (VAE) describes the persistent shift of perceived sound location after having been adapted to a ventriloquism condition, in which the sound was repeatedly paired with a displaced visual stimulus. In the latter case, participants consistently mislocalize the sound in the direction of the visual stimulus (ventriloquism effect, VE). Previous studies provide conflicting reports regarding the strength of the VAE, ranging from 0 to nearly 100%. Moreover, there is controversy about its generalization to different sounds than the one inducing the VE, ranging from no transfer at all, to full transfer across different sound spectra. Here, we imposed the VE for three different sounds: a low-frequency and a high-frequency narrow-band noise, and a broadband Gaussian white noise (GWN). In the adaptation phase, listeners generated fast goal-directed head movements to localize the sound, presented across a 70 deg range in the horizontal plane, while ignoring a visual distracter that was consistently displaced 10 deg to the right of the sound. In the post-adaptation phase, participants localized narrow-band sounds with center frequencies from 0.5 to 8 kHz, as well as GWN, without the visual distracter. Our results show that the VAE amounted to approximately 40% of the VE and generalized well across the entire frequency domain. We also found that the strength of the VAE correlated with the pre-adaptation sound-localization performance. We compare our results with previous reports and discuss different hypotheses regarding optimal audio-visual cue integration.Significance statementWhere we believe sounds to emanate from is heavily influenced by what we see. We experience this illusion - the ventriloquism effect - regularly. For example, when watching movies, we perceive the voices of actors to originate from the actors’ lips rather than from the speakers. Long exposure to a spatial mismatch between what we hear and what we see, leads to a recalibration of our localization of sounds - the ventriloquism aftereffect. Unclear was whether exposure to an audiovisual mismatch for one sound type would also lead to adaptive changes for other sounds. The present study shows that the ventriloquism aftereffect generalizes, implying that it does not act at an early auditory stage, but at a late multisensory stage.

Chronic single-sided deaf (CSSD) listeners lack the availability of binaural difference cues to localize sounds in the horizontal plane. Hence, for directional hearing they have to rely on different types of monaural cues: loudness perceived in their hearing ear, which is affected in a systematic way by the acoustic head shadow, on spectral cues provided by the low-pass filtering characteristic of the head, and on high-frequency spectral-shape cues from the pinna of their hearing ear. Presumably, these cues are differentially weighted against prior assumptions on the properties of sound sources in the environment. The rules guiding this weighting process are not well understood. In this preliminary study, we trained three CSSD listeners to localize a fixed intensity, high-pass filtered sound source at ten locations in the horizontal plane with visual feedback. After training, we compared their localization performance to sounds with different intensities, presented in the two-dimensional frontal hemifield to their pre-training results. We show that the training had rapidly readjusted the contributions of monaural cues and internal priors, which resulted to be imposed by the multisensory information provided during the training. We compare the results with the strategies found for the acute monaural hearing condition of normal-hearing listeners, described in an earlier study [1].

Contains fulltext : 215723.pdf (Publisher’s version ) (Open Access) We assessed how synchronous speech listening and lipreading affects speech recognition in acoustic noise. In simple audiovisual perceptual tasks, inverse effectiveness is often observed, which holds that the weaker the unimodal stimuli, or the poorer their signal-to-noise ratio, the stronger the audiovisual benefit. So far, however, inverse effectiveness has not been demonstrated for complex audiovisual speech stimuli. Here we assess whether this multisensory integration effect can also be observed for the recognizability of spoken words. To that end, we presented audiovisual sentences to 18 native-Dutch normal-hearing participants, who had to identify the spoken words from a finite list. Speech-recognition performance was determined for auditory-only, visual-only (lipreading), and auditory-visual conditions. To modulate acoustic task difficulty, we systematically varied the auditory signal-to-noise ratio. In line with a commonly observed multisensory enhancement on speech recognition, audiovisual words were more easily recognized than auditory-only words (recognition thresholds of -15 and -12 dB, respectively). We here show that the difficulty of recognizing a particular word, either acoustically or visually, determines the occurrence of inverse effectiveness in audiovisual word integration. Thus, words that are better heard or recognized through lipreading, benefit less from bimodal presentation. Audiovisual performance at the lowest acoustic signal-to-noise ratios (45%) fell below the visual recognition rates (60%), reflecting an actual deterioration of lipreading in the presence of excessive acoustic noise. This suggests that the brain may adopt a strategy in which attention has to be divided between listening and lipreading.