Search

The Acoustic Interactions for Robot Audition corpus is introduced for research on sound source localization and separation, and for multi-user speech recognition. Its aim is to evaluate and train Robot Audition techniques, as well as Auditory Scene Analysis in general. It was recorded in six real-life environments with different noise presence and reverberation time, using two array configurations: an equilateral triangle, and a three-dimensional 16-microphone array set over a hollow plastic body. It includes clean speech data for static sources and tracking information for mobile sources. It is freely available at https://aira.iimas.unam.mx/.

Auditory Scene Analysis (ASA) research has provided knowledge about the principles underlying auditory organization processes and has been successfully applied in computational ASA. Based on these findings, we applied these principles within a musical context. The aim is to understand and computationally model the perception of effects, such as blend and segregation, created by the combining and contrasting of properties of traditional Western instruments which result from three auditory processes: concurrent, sequential, and segmental grouping. The initial aim was to evaluate the extent to which the symbolic data provided in a musical score provide sufficient data to model the perception of these orchestral effects. Preliminary implementations have achieved an average accuracy score of 81%, suggesting that perceptual effects of orchestration can be partially retrieved by calculations based on ASA principles using symbolic data. However, many cases indicate that including properties of the acoustic signal would enhance the predictive power. This approach also provides us with the means to investigate the relative weights of the different principles involved in these grouping processes in order to understand their relative importance in musical contexts. These findings contribute to the creation of a framework for studying and understanding the perceptual characteristics of orchestration practice.

The ability to segregate simultaneous sound sources based on their spatial locations is an important aspect of auditory scene analysis. While the role of sound azimuth in segregation is well studied, the contribution of sound elevation remains unknown. Although previous studies in humans suggest that elevation cues alone are not sufficient to segregate simultaneous broadband sources, the current study demonstrates they can suffice. Listeners segregating a temporally modulated noise target from a simultaneous unmodulated noise distracter differing in elevation fall into two statistically distinct groups: one that identifies target direction accurately across a wide range of modulation frequencies (MF) and one that cannot identify target direction accurately and, on average, reports the opposite direction of the target for low MF. A non-spiking model of inferior colliculus neurons that process single-source elevation cues suggests that the performance of both listener groups at the population level can be accounted for by the balance of excitatory and inhibitory inputs in the model. These results establish the potential for broadband elevation cues to contribute to the computations underlying sound source segregation and suggest a potential mechanism underlying this contribution.

Periodic stimuli are common in natural environments and are ecologically relevant, for example, footsteps and vocalizations. This study reports a detectability enhancement for temporally cued, periodic sequences. Target noise bursts (embedded in background noise) arriving at the time points which followed on from an introductory, periodic "cue" sequence were more easily detected (by ∼1.5 dB SNR) than identical noise bursts which randomly deviated from the cued temporal pattern. Temporal predictability and corresponding neuronal "entrainment" have been widely theorized to underlie important processes in auditory scene analysis and to confer perceptual advantage. This is the first study in the auditory domain to clearly demonstrate a perceptual enhancement of temporally predictable, near-threshold stimuli.

Reverberation distorts the sounds produced in the world, but in doing so provides information about the environment. This distortion is characterized by the Impulse Response (IR) and depends upon the material and geometry composing the environment. For some tasks (voice recognition, source localization, acoustic tomography, etc.), reverberation must be discounted, but for others (room identification, distance estimation, etc.), it must be analyzed. Our recent work on the perception of reverberation has leveraged measurements of real-world IRs, which exhibit a number of regularities that the brain appears to have internalized for auditory scene analysis. Here, we present a library of these measurements and a toolbox for measuring and analyzing additional IRs. The library contains 271 IRs from spaces encountered by 7 volunteers over 2 weeks of daily life, and thus reflects the distribution of typical reverberation experienced by humans. The toolbox includes procedures for measuring IRs with low-cost, portab...

Sound localization is an important aspect of auditory scene analysis, allowing listeners to group acoustic components from the same location into a single stream (Bregman 1990). Binaural pitch fusion, the fusion of different frequency tones across ears, can be thought of as one type of auditory streaming. Little is known about how sound localization cues affect binaural fusion. The goal of this study was to investigate the effects of interaural level differences (ILDs), one cue for sound localization, on binaural fusion of dichotic tones. Binaural pitch fusion was measured in adult normal-hearing (NH) listeners, using five ILD conditions: ILD = 0, 1.25, 2.5, 5, and 10 dB. Fusion ranges were measured by simultaneous presentation of reference and comparison stimuli in opposite ears, and varying the comparison stimulus to find the frequency range that fused with the reference stimulus. Preliminary results (5 NH; data collection is ongoing) show that small ILDs increase fusion ranges, while larger ILDs decrease fusion ranges. These findings suggest that ILDs can affect fusion range in NH listeners, and imply that simulated sound source location may provide a top-down grouping cue for binaural fusion. [This research was funded by a NIH-NIDCD grant R01 DC013307.]Sound localization is an important aspect of auditory scene analysis, allowing listeners to group acoustic components from the same location into a single stream (Bregman 1990). Binaural pitch fusion, the fusion of different frequency tones across ears, can be thought of as one type of auditory streaming. Little is known about how sound localization cues affect binaural fusion. The goal of this study was to investigate the effects of interaural level differences (ILDs), one cue for sound localization, on binaural fusion of dichotic tones. Binaural pitch fusion was measured in adult normal-hearing (NH) listeners, using five ILD conditions: ILD = 0, 1.25, 2.5, 5, and 10 dB. Fusion ranges were measured by simultaneous presentation of reference and comparison stimuli in opposite ears, and varying the comparison stimulus to find the frequency range that fused with the reference stimulus. Preliminary results (5 NH; data collection is ongoing) show that small ILDs increase fusion ranges, while larger ILDs decrea...

Frequency-matched bilateral input is important for the optimum encoding of binaural cues to facilitate sound localization, auditory scene analysis, and speech understanding in noise. Current bilateral cochlear-implant (CI) programming procedures do not account for potential interaural place-of-stimulation mismatch. This study investigated whether a perceptual test of interaural-time-difference (ITD) sensitivity can effectively estimate relative interaural mismatch for bilateral CI listeners. Ten bilateral CI listeners were tested on a two-interval left-right ITD discrimination task. Loudness balanced 300-ms, 100 pulse-per-second constant-amplitude pulse trains were delivered to single-electrode pairs using direct stimulation. Measurements were made for five reference electrodes evenly distributed along the array in one ear, and for at least five closely spaced comparison electrodes in the other ear for each reference electrode. The pair with the smallest thresholds was assumed to be the most closely place...

Assessment of the neural correlates of auditory scene analysis, using an index of sound change detection that does not require the listener to attend to the sounds [a component of event-related brain potentials called the mismatch negativity (MMN)], has previously demonstrated that segregation processes can occur without attention focused on the sounds and that within-stream contextual factors influence how sound elements are integrated and represented in auditory memory. The current study investigated the relationship between the segregation and integration processes when they were called upon to function together. The pattern of MMN results showed that the integration of sound elements within a sound stream occurred after the segregation of sounds into independent streams and, further, that the individual streams were subject to contextual effects. These results are consistent with a view of auditory processing that suggests that the auditory scene is rapidly organized into distinct streams and the integration of sequential elements to perceptual units takes place on the already formed streams. This would allow for the flexibility required to identify changing within-stream sound patterns, needed to appreciate music or comprehend speech.

It is difficult to hear out individually the components of a "chord" of equal-amplitude pure tones with synchronous onsets and offsets. In the present study, this was confirmed using 300-ms random (inharmonic) chords with components at least 1/2 octave apart. Following each chord, after a variable silent delay, listeners were presented with a single pure tone which was either identical to one component of the chord or halfway in frequency between two components. These two types of sequence could not be reliably discriminated from each other. However, it was also found that if the single tone following the chord was instead slightly (e.g., 1/12 octave) lower or higher in frequency than one of its components, the same listeners were sensitive to this relation. They could perceive a pitch shift in the corresponding direction. Thus, it is possible to perceive a shift in a nonperceived frequency/pitch. This paradoxical phenomenon provides psychophysical evidence for the existence of automatic "frequency-shift detectors" in the human auditory system. The data reported here suggest that such detectors operate at an early stage of auditory scene analysis but can be activated by a pair of sounds separated by a few seconds.

Auditory scene analysis involves the simultaneous grouping and parsing of acoustic data into separate mental representations (i.e., objects). Over two experiments, we examined the sequence of neural processes underlying concurrent sound segregation by means of recording of human middle latency auditory evoked responses. Participants were presented with complex sounds comprising several harmonics, one of which could be mistuned such that it was not an integer multiple of the fundamental frequency. In both experiments, Na (approximately 22 ms) and Pa (approximately 32 ms) waves were reliably generated for all classes of stimuli. For stimuli with a fundamental frequency of 200 Hz, the mean Pa amplitude was significantly larger when the third harmonic was mistuned by 16% of its original value, relative to when it was tuned. The enhanced Pa amplitude was related to an increased likelihood in reporting the presence of concurrent auditory objects. Our results are consistent with a low-level stage of auditory scene analysis in which acoustic properties such as mistuning act as preattentive segregation cues that can subsequently lead to the perception of multiple auditory objects.

Assessing the impact of entertainment sounds in communities will need a multifaceted analysis. An evaluation process has to be considered that will cover the different facets with regard to people’s understanding about an acoustic environment where the impact of entertainment sound will occur. The “meeting of music and environment” discussed by R. M. Schafer opens the mind in many directions to learn about the different aspects and related impacts in such a soundscape that is relying on given activities in an area that will lead to a harmonized or an acoustically unbalanced situation. Understanding human auditory scene analysis and the important role of auditory attention forces to outline soundscape assessment methods and to come to enhanced methodologies for inventing action parameters with regard to a sounding community. This paper will introduce regulations to harmonize the different interests regarding interventions through entertainment sounds in communities but also introduce the soundscape approac...

Auditory scene analysis depends on knowledge of natural sound structure, but little is known about how source-specific structures might be learned and applied. We explored whether listeners internalize “schemas”—the abstract structure shared by different occurrences of the same type of sound source—during cocktail-party listening. We measured the ability to detect one of two concurrent “melodies” that did not differ in mean pitch (nor in timbre), ensuring that only the structure of these melodies over time could be used to distinguish them. Target melodies were cued by presenting them in isolation before each mixture, transposed to avoid exact repetition. The task was to determine if the cued melody was present in the subsequent mixture. Listeners performed above chance despite transposition between cue and target. Particular melodic schemas could recur across a subset of trials within a block, as well as across blocks separated by epochs in which the schema was absent. Recurrence across trials within a b...

The conceptualization and implementation of a psychoacoustic sound field synthesis system for music is presented. Critical bands, the precedence effect, and integration times of the auditory system as well as the radiation characteristics of musical instruments are implemented in the signal processing. Interaural coherence, masking and auditory scene analysis principles are considered as well. The sound field synthesis system creates a natural, spatial sound impression and precise source localization for listeners in extended listening area, even with a low number of loudspeakers. Simulations and a listening test provide a proof of concept. The method is particularly robust for signals with impulsive attacks and quasi-stationary phases as in the case of many instrumental sounds. It is compatible with many loudspeaker setups, such as 5.1, ambisonics systems and loudspeaker arrays for wave front synthesis. The psychoacoustic sound field synthesis approach is an alternative to physically centered wave field ...

Contemporary thinking and research on timbre and its use as a dynamic, structural component in music performance have been profoundly influenced by the insights and insounds of David Wessel. His intrepid and creative approach opened up vistas of timbre spaces navigable through multidimensional trajectories. Wessel’s experiments with timbre streaming [Computer Music J. 3, 4552, (1979)] inspired my own work on perceptual organization of complex-tone sequences [Singh, J. Acoust. Soc. Am. 82, 886-899 (1987)]. The finding of a timbre “interva” akin to a pitch interval as a threshold for streaming reinforced Wessel’s notion of timbral analogies [Ehresman and Wessel, IRCAM Rep 13/78, (1978)]. Later work on measuring timbre differences through FO thresholds for streaming (Singh and Bregman, J. Acoust. Soc. Am. 102(4), 1943-1952, (1997)) also lent support to the idea of intervallic relationship between timbres. More recently, my work relating Auditory Scene Analysis to Hindustani rhythms brought us together again,...

Despite its temporal precision, the auditory system does not encode fine detail of some classes of natural sounds. For example, sounds known as “auditory textures” seem to be encoded and retained with a lossy, compressed representation consisting of time-averaged statistics. One explanation is that the auditory system compresses stimuli that exceed its informational bandwidth. Decreased sensitivity to temporal detail of sound would reflect a limit of the auditory system to transmit sensory information above a certain rate. Here we instead propose a normative explanation. We assume that to minimize energy expenditure, the auditory system compresses stimuli that do not carry novel information about the environment. We developed practical measures of stimulus coding cost (the number of simulated auditory nerve spikes required to encode the sound) and stationarity (degree of change to the sound spectrum across successive time windows). We found that coding cost is not predictive of the ability to discriminate...

A front end for automatic speech recognizers is proposed and evaluated which is based on a quantitative model of the "effective" peripheral auditory processing. The model simulates both spectral and temporal properties of sound processing in the auditory system which were found in psychoacoustical and physiological experiments. The robustness of the auditory-based representation of speech was evaluated in speaker-independent, isolated word recognition experiments in different types of additive noise. The results show a higher robustness of the auditory front end in noise, compared to common mel-scale cepstral feature extraction. In a second set of experiments, different processing stages of the auditory front end were modified to study their contribution to robust speech signal representation in detail. The adaptive compression stage which enhances temporal changes of the input signal appeared to be the most important processing stage towards robust speech representation in noise. Low-pass filtering of the fast fluctuating envelope in each frequency band further reduces the influence of noise in the auditory-based representation of speech.

An important aspect of auditory scene analysis is the specialized neurocircuitry required for vocal production. in a dynamic acoustic environment. Although often taken for granted, enhanced audio-vocal feedback is relatively uncommon in animals and yet an important precursor to vocal learning. We argue that vocal complexity in bats is an exaptation of the highly specialized audio-vocal feedback system that has evolved for echolocation. First, we explore how audio-vocal feedback enhances echolocation. Second, we review how echolocation pulses serve social functions by providing information to receivers (like gender, identity, or food availability). Third, using our research on molossid bats (family Molossidae), we explore whether vocal plasticity in sonar has contributed to an expanded role for sonar pulses in social communication. In at least three molossids, roosting bats rapidly sing in response to echolocation pulses of flying conspecifics. However, more importantly, we show that in multiple species, echolocation is produced in purely social contexts. Roosting bats embed pulses and feeding buzzes into their courtship songs that are not acoustically distinct than when foraging. Finally, some molossids not only sing in roosts, but also in flight—that is, they echolocate and sing simultaneously. These findings indicate that echolocation plays even more of a role in social communication than commonly believed and that the production of echolocation and social communication is tightly coupled and coordinated at a high level.

cote interne IRCAM: McAdams97a; None / None; National audience; A study was conducted to determine whether newborn infants organize auditory streams in a manner similar to that of adults. A series of three experiments investigated the ability of three- to four-day-old infants to discriminate repeated rising and falling four-tone sequences in two configurations of source timbre and spatial position. It was hypothesized that if the sequences were organized into two auditory streams on the basis of timbre and spatial position, one of the configurations should be discriminable from its reversal, while the other should not. The sequences were tested with different pitch and temporal intervals separating the tones. Sequences were discriminated for the first configuration by adults at both fast tempo/small interval and slow tempo/large interval combinations, while only the latter was discriminated by newborns as measured with a non-nutritive, high-amplitude sucking paradigm. Neither adults nor infants could discriminate the sequence reversals for the second configuration. The results suggest that newborn infants organize auditory streams on the basis of source timbre and/or spatial position. They also suggest that newborns have limits in temporal and/or pitch resolution when discriminating tone sequences. [Article à Paraître]

An essential feature of intelligent sensory processing is the ability to focus on the part of the signal of interest against a background of distracting signals, and to be able to direct this focus at will. In this paper the problem of auditory streaming is considered and a model of the early stages of the process is proposed. The behavior of the model is shown to be in agreement with a number of well-known psychophysical results, including the relationship between presentation rate, frequency separation and streaming, the temporal development of streaming, and the effect of background organization on streaming. The principal contribution of this model is that it demonstrates how streaming might result from interactions between the tonotopic patterns of activity of incoming signals and traces of previous activity which feed back and influence the way in which subsequent signals are processed. The significance of these results for auditory scene analysis is considered and a framework for the integration of simultaneous and sequential grouping cues in the perception of auditory objects is proposed.

The perception of pitch forms the basis of musical melody and harmony. It is also among the most precise of all our human senses, and with imagination, this precision can be used experimentally to investigate the functioning of the auditory system. This tutorial presents auditory demonstrations from the zoo of pitch effects: pitch shifts, noise pitch, virtual pitch, dichotic pitch, and the pitches of things that are not there at all. It introduces models of auditory processing, derived from contemporary psychoacoustics and auditory physiology, and tests these models against the experimental effects. It concludes by describing the critical role played by pitch in the important human ability to disentangle overlapping sources of sound.

The perceptual organization of auditory scenes needs to parse the incoming flow of acoustic inputs into perceptual streams. It is likely that cues from several modalities are pooled for auditory scene analysis, including sensory-motor cues related to the active exploration of the scene. We show effects of source- and head-motion on auditory streaming. In a streaming paradigm, listeners hear a sequence of repeating tone triplets and indicate their perception of one or two subjective sources called streams. We used a robotic telepresence system, Telehead, to disentangle the effects of head motion: changes in acoustic cues at the ear, subjective location cues, and motor cues. We found that head motion induced perceptual reorganization even when the auditory scene had not changed. We further analyzed the data to probe the time course of sensory-motor integration. Motor cues impacted perceptual organization earlier and for a shorter time than acoustic or subjective location cues, with successive positive and n...

Speech perception performance is one of the most widely used measures in audiology and auditory research. Nevertheless, our understanding of how speech is processed by the auditory system is limited. This is not only due to the subject—the complexity of the speech signal itself and the intricacy of disentangling target speech from interfering sounds—but also to the way it is approached: different aspects of speech perception are studied by largely separate research communities. Also when only early processing of speech is considered, excluding psycholinguistic aspects and effects of memory and learning, at least three relevant research fields can be discriminated: audibility and masking, auditory scene analysis, and attention. Although there is increasing interest in the crosslinks between these fields, approaches are still insufficiently integrated and important questions remain unanswered. Furthermore, modeling efforts are lagging behind and are only partly driven by advances in neurophysiological resea...

Perception depends on discovering and forming representations of sensory patterns to make sense of the world. Patterns playing out both in time and feature space allow sensory systems to form expectations, and provide a basis for decomposing the world into discrete objects and for detecting meaningfully novel events. Without expectations, the system has no way of autonomously evaluating its understanding of the world. However, patterns come and go. So, the perceptual system needs some way of allowing its operational representations to form and dissolve, dominate and yield, in a way that facilitates veridical perception. In this talk, I will discuss some of the issues of pattern discovery, maintenance, and deletion at different timescales and levels of complexity using exemplar models of SSA and auditory streaming, and new perceptual data.

Two areas of interest within auditory scene analysis are the identification of cues which contribute to auditory grouping and/or segregation, and the methodology employed to measure such effects. The present study explored the roles of three purported grouping cues--correlated AM, correlated FM, and harmonicity--using a comodulation masking release interference paradigm. This method was reasoned to be free of alterations in the subject's mode of listening (analytic versus synthetic). The results suggested that correlated AM was a viable grouping cue. For some listeners, the presence of FM also appeared to provide a grouping cue, but, generally, no effect of coherence of FM was observed. No effect of harmonicity was observed in this paradigm.

Arnold Schering and others have partitioned music genres into traditions where the identification of individual instruments (Spaltklang) is the goal and others where instruments are blended to a homogeneous sound (Verschmelzung). Baroque music is often cited to emphasize the identifiability of individual instruments, whereas the fusion of orchestral instruments became one of the major goals in the romantic period. Historical vinyl record releases were used to investigate the extent to which the theory of identification and blending can be applied to 20th century popular music based on the fundamental principles of Auditory Scene Analysis. The identifiability of instruments was one of the major concerns for jazz combos, for example, in the Miles Davis sextet of the late fifties. Blending became more important in many types of rock music to create a huge sound, for example, in Phil Spector’s Wall of Sound of the 1960s. Artificial reverberation, dynamic range compression, and other effects were now used to c...

For well over 40 years, Dick Fay has been making broad, significant, and insightful contributions to our understanding of vertebrate, and particularly fish, hearing. His work has range from psychoacoustics to physiology to, most recently, behavioral responses. While Dick has worked on a number of different species, his primary research animal has been the goldfish (though he has also worked with sharks, cichlids, toadfish, and other species). Through ingenious studies that involved a variety of different psychophysical techniques, Dick not only asked question about what fish can hear, but also did pioneering work on things like masking, critical bands, and auditory scene analysis. Dick’s physiological studies have been of equal importance. These often involved the use of an ingenious “shaker table” that he developed which allow very precise control of the stimulus in three-dimensions. Using this approach, Dick has asked questions about function at many levels of the auditory system, providing important in...

Experiments on young adults studied the effects of suddenness of onset or offset on the discrimination of the order of pitches of individual tones in a 1-s, 4-tone cluster of overlapping pure tones. In experiment 1, the tones, all within a critical band, went on asynchronously. Each rose and decayed linearly in amplitude. Faster onsets, within the range 10 to 640 ms as measured on the first tone, increased the accuracy of the discrimination of the order of onsets, but 10-ms onsets were slightly worse than 40-ms onsets in early sessions. Experiment 2 found similar effects for the abruptness of offsets of tones in clusters whose components came on synchronously but went off asynchronously. Onset order was very much easier to detect than offset order. The auditory system may use neural onset and offset responses to reset itself and carry out new analyses at frequency-by-amplitude points of sudden amplitude change, thereby contributing to auditory scene analysis.

By integrating state-of-the-art auditory scene-analysis methods with artificial cognitive feedback, the current project challenges human performance in quality of experience tasks and auditory scene analysis. Human listeners are regarded as multi-modal agents that develop their concept of the world by exploratory interaction. The prominent goal of the project is to develop an intelligent, computational model of active auditory perception and experience in a multi-modal context. The resulting system framework will form a structural link from binaural perception to judgment and action, realized by interleaved signal-driven (bottom-up) and hypothesis-driven (top-down) feedback processing within an innovative expert-system architecture. A conceptual overview of the project framework is presented, and insight is given into the current state of research, focusing on CASA-related search-&-rescue (S&R) scenarios. In these scenarios, an autonomous robot is endowed with auditory/visual feature-analysis facilities that provide it with bottom-up information of its environment. Top-down evaluation of the extracted features then results in cognitive feedback loops that allow the machine to adapt to complex S&R scenarios and perform significantly better than would be possible by only employing feed-forward control mechanisms. [Work performed in the context of the EU project Two!Ears.]

Separating out and correctly grouping the sounds of a communicating animal from the natural acoustic environment poses significant challenges to models of auditory scene analysis, yet animals perform this task very effectively. To date, most models have focussed on simultaneous grouping cues and the segregation of discrete sound events, although some take the longer term context into account. Inspired by the important part that form plays in the segregation and recognition of visual objects, we consider the role of form in auditory scene analysis. By form in audition we mean the dynamic spectrotemporal patterns characterizing individual sound events as well as their timing with respect to each other. We present a model capable of segregating and recognizing natural communication calls within complex acoustic environments. Incoming sounds are processed using a model of the auditory periphery and fed into a recurrent neural network that rapidly tunes itself to respond preferentially to specific events. Representations of predictable patterns of events in the sequence are created on the fly and maintained on the basis of their predictive success and conflict with other representations. Activation levels of these representations are interpreted in terms of object recognition.

Communication between robots and humans is a challenge in complex built environments. In this project, we are exploring how previous achievements in this area apply to human/robot communication within immersive virtual displays. In our concrete example, we host human-scale robots that interact with humans in our Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVE-Lab). The CRAIVE-Lab provides a physical/digital environment for collaborative tasks using seamless video from multiple projectors and a 128-channel wave-field system. The system is designed to monitor the whole floor-space area (10 m × 12 m) with a camera network mounted to the ceiling grid and a microphone array combining shotgun, spherical and possibly people-worn microphones. Based on the sensor data, a visual analysis and an auditory scene analysis are performed. The latter includes sound localization, speech and musical-feature recognition. The robots, a Rethink Baxter robot mounted on an electric wheelchair and a Robokind Zeno, are used to perform assistive technology and social communication tasks. Both robots have direct access to these data using a digital feed over a wireless network to augment their own sensor systems of built-in cameras and microphones. [Work supported by NSF grant #1229391.]

An auditory scene is perceived in terms of its constituent auditory objects. Here, we investigate how auditory objects are individually represented in human auditory cortex, using magnetoencephalography (MEG) to record the neural responses of listeners. In a series of experiments, subjects selectively listen to one of two competing streams, in a variety of auditory scenes. In the acoustically richest example, subjects selectively listen to one of two competing speakers mixed into a single channel. Individual neural representations of the speech of each speaker are observed, with each being selectively phase locked to the rhythm of the corresponding speech stream, and from which can be exclusively reconstructed the temporal envelope of that speech stream. The neural representation of the attended speech, originating in posterior auditory cortex, dominates the responses. Critically, when the intensities of the attended and background speakers are separately varied over a wide intensity range, the neural representation of the attended speech adapts only to the intensity of that speaker, but not to the intensity of the background speaker. Overall, these results indicate that concurrent auditory objects, even if spectrally overlapping and not resolvable at the auditory periphery, are indeed neurally encoded individually as objects, in auditory cortex.

The notion that sensitivity to temporal regularity (TR) plays a pivotal role in auditory scene analysis (ASA) has recently garnered considerable attention. Nevertheless, evidence supporting a primary role for TR is based on experiments employing simple stimuli consisting of one, or two, concurrent sound sequences. Whether the role of TR in mediating ASA is robust to more complex listening environments is unknown. The present study investigates sensitivity to TR in the context of a change detection task, employing complex acoustic scenes comprised of up to 14 concurrent auditory objects. Sequences of sounds produced by each object were either temporally regular (REG) or irregular (RAND). Listeners had to detect occasional changes (appearances or disappearances of an object) within these “soundscapes.” Listeners' performance depended on the TR of both the changing object and the scene context (TR of other objects in the scene) such that RAND contexts were associated with slower response times and substantially reduced detection performance. Therefore, even in complex scenes, sensitivity to TR is critical to our ability to analyze and detect changes in a dynamic soundscape. Importantly, the data reveal that listeners are able to acquire the temporal patterning associated with at least 14 concurrently presented objects.

Aerial insectivorous bats face the challenge of efficient echo scene analysis for localizing obstacles and capturing prey in flight. Data collected with a telemetry microphone mounted on the foraging bat's head provide a valuable opportunity to reconstruct acoustic scenes comprised of echoes returning to the bat's ears. This study explores the information embedded in echoes from a tethered insect and background clutter recorded by the telemetry microphone in laboratory experiments. Using images from high-speed video cameras and recordings from a far-field microphone array, angular information about different objects in the echoes are restored by assimilating the reconstructed bat's position and echolocation beam aim with respect to the objects along its flight trajectory toward prey capture. This procedure is further augmented by theoretical simulations using acoustic scattering principles to circumvent the limitation imposed by the sensitivity and signal-to-noise ratio of the telemetry microphone. The reconstructed acoustic scenes offer an avenue for detailed analysis of important cues for figure-ground separation in a cluttered environment, and serve as a basis for subsequent neurocomputational modeling of auditory scene analysis performed by the bat's sonar receiver.

Change deafness is the auditory analog to change blindness. Both phenomena represent a tendency to miss large changes in the environment, suggesting that sensory experiences are not verbatim and may details crucial for detection and identification. Spatial separation facilitates change detection in vision, but its role in auditory change detection is unclear. In this study, we examined the impact of spatial separation on the detection of appearing and or disappearing sound sources in an auditory scene. Participants listened to a brief auditory scene (1000 ms) comprised of four sources followed by a scene where a sound source was added or subtracted from the scene or where no change occurred. There were two listening conditions, where the sound sources were each distributed across a loudspeaker array, or the sound sources were all played from a single loudspeaker. Results indicate that listeners were better able to detect appearing than disappearing sounds, and fewer errors were made when sound sources wer...

In order to make sense of natural auditory environments, the developing child must learn to ‘‘navigate’’ through complex auditory scenes in order to segment out relevant auditory information from irrelevant sounds. This study investigated some of the informational and attentional factors that constrain environmental sound detection in auditory scenes, and how these factors change over development. Thirty‐two children (aged 9–12 years) and 16 adults were asked to detect short target environmental sounds (e.g., a dog barking) within longer environmental background sounds (e.g., a barn) presented dichotically. The target environmental sounds were either congruent (i.e., normally associated with the background) or incongruent. Subjects heard either a single background presented to both the ipsilateral and contralateral ears or else different background sounds presented to the different ears. Results indicate that children’s, but not adults’ target detection is substantially less accurate when listening to two...

Human communication includes the capability of recognition. This is particularly true of auditory communication. Music information retrieval (MIR) turns out to be particularly challenging, since many problems remain still unsolved. Topics that should be included within the scope of MIR are automatic classification of musical instruments/phrases/styles, music representation and indexing, estimating musical similarity using both perceptual and musicological criteria, recognizing music using audio and/or semantic description, language modeling for music, auditory scene analysis, and others. Many features of music content description are based on perceptual phenomena and cognition. However, it can easily be observed that most of the low‐level descriptors used, for example, in musical instrument classification are more data‐ than human‐oriented. This is because the idea behind these features is to have data defined and linked in such a way as to be able to use it for more effective automatic discovery, integra...

Auditory scene analysis refers to mechanisms that parse a composite acoustic waveform into perceptually distinct auditory objects or streams that correspond to distinct sound sources in the environment. A familiar and well‐studied example of a difficult auditory scene analysis task is the cocktail party problem, which refers to the difficulty we have perceiving speech in noisy social environments comprised of multiple people talking simultaneously. Students of human hearing and computer engineering study auditory scene analysis, in part, because people with hearing impairments and computer algorithms for speech recognition do a poor job of solving the cocktail party problem compared to people with normal hearing. Of course, the cocktail party problem is not unique to humans. The animal behavior literature provides numerous examples of cocktail‐party‐like social environments for animal acoustic communication (e.g., choruses, roosts, colonies, creches). Compared to previous work on humans, however, students of animal bioacoustics have devoted little attention to how animals that signal in groups or networks solve their own cocktail party problems. This talk will briefly review some issues in auditory scene analysis relevant to the study of animal acoustic communication and present the case for why these issues deserve further study by students of bioacoustics.

Research on auditory scene analysis (ASA) began with some simple laboratory phenomena such as streaming and illusory continuity. Subsequently, research has gone in three directions, downwards toward underlying mechanisms (by neurophysiologists), upwards toward system organization (by computer scientists), and sideways toward other species (by neurobiologists). Each direction has its problems. The downward approach sometimes takes a phenomenon-oriented view of ASA, leading to simple explanations of a single ASA demonstration, such as streaming, with no obvious connection to any larger system. Research done by the upward approach usually takes the form of a computer program to achieve ASA in a working system, often ignoring known facts about human ASA, in favor of mathematically understood principles. The sideways approach often finds that non-human animals can respond to an important sound despite the presence of other interfering sounds. However, there is no reason to believe that a frog, a fish, and a human accomplish this by means of the same mechanisms. So finding out how some animal does this, while interesting in its own right, may shed little light on how humans do it. I will describe some properties of the human ASA system that should be borne in mind when manufacturing explanations.

Deciphering the auditory scene is a problem faced by humans and animals alike. However, when faced with overlapping sounds from multiple locations, listeners are still able to attribute the individual sound objects to their individual sound-producing sources. Here, we determined which characteristics of sounds are important for streaming versus segregating in birds. Budgerigars and zebra finches were trained using operant conditioning procedures on an identification task to peck one key when they heard a whole zebra finch song and to peck another when they heard a zebra finch song missing a middle syllable. Once the birds were trained to a criterion performance level on those endpoint stimuli, probe trials were introduced on a small proportion of all trials. The probe songs contained modifications of the incomplete training song's missing syllable. When the bird responded as if the probe was a whole song, it suggests they streamed together the altered syllable and the rest of the song. When the bird responded non-whole song, it suggests they segregated the altered probe from the rest of the song. Results show that some features, such as spectrotemporal similarity and location, are more important for streaming than other features, such as timing.

The three‐dimensional, time‐variant output of a computational model of the human auditory system was investigated with the aim of extracting perceptually patterns as relevant to spatial impression. To this end, the same stimuli with which the model was excited were also presented to human listeners which evaluated (i) the auditory source width (ASW), i.e., the horizontal extent of the auditory event, and (ii) the listener envelopment (LEV), i.e., the amount of envelopment by the auditory event. The listeners had to be trained beforehand as the discrimination of source‐ and room‐related parameters of auditory events is not a common task to them. To allow for natural localization and to enable interactive listening, the stimuli were presented through head‐tracked headphones. Trained listeners appeared to produce consistent results when depicting auditory objects on a compass‐rose‐shaped plane, where they had to mark the position and broadening of the auditory event(s) by the position(s) and length(s) of straight lines, and the envelopment by the position and size of an ellipse.

Auditory scene analysis has been suggested as a universal process that exists across all animals. Relative to humans, however, little work has been devoted to how animals isolate sound sources to create auditory objects. Frequency separation of sounds is arguably the most common parameter studied in auditory streaming, yet it is not the only factor. Elhilali et al. (2009) found that in humans, synchronous tones are heard as a single auditory stream, even at large frequency separations, compared to asynchronous tones with the same frequency separations, which are perceived as two sounds. These findings demonstrate how both timing and frequency separation of sounds are important for auditory scene analysis. It was unclear how animals, such as budgerigars (Melopsittacus undulatus), would perceive synchronous and asynchronous sounds. Budgerigars were tested on the perception of synchronous, asynchronous, and partially overlapping pure tones and budgerigar contact calls. Budgerigars segregate partially overlapping sounds in a manner predicted by computational models of streaming. However, overlapping budgerigar contact calls are more likely to be segregated than pure tone stimuli with the same temporal overlap. These results emphasize the necessity of using complex communication signals when examining complex sound perception processes such as auditory scene analysis.

Top-down schemas or prior probabilities associated with different auditory objects and their acoustic features are thought to improve auditory scene analysis and auditory object recognition processes. This study focused on whether listeners implicitly track the prior probabilities associated with different tone frequencies, and whether tracking those probabilities modulates attentional bandwidth to improve detection sensitivity. Using the target-probe paradigm, attentional bandwidth for varying levels of probe probability was estimated for 40 listeners. To estimate the attentional bandwidth, probes were presented in a continuous band-pass noise masker at distances of .75, 1.25, 1.75, and 3.75 Equivalent Rectangular Bandwidths from a target frequency of 250 Hz, at varying probabilities of occurrence. All tones were presented at approximately 90% detection when presented alone as measured during a separate thresholding procedure. Results indicate that prior probability plays a role in attentional bandwidth:...

Predicting the ability of listeners to discriminate between simultaneous auditory streams is a longstanding challenge in the design of auditory displays. The creation of an efficacious artificial auditory scene can require an immense amount of knowledge about how sounds are heard and interpreted; what is commonly called auditory scene analysis. Fortunately, musicians have been constructing novel auditory scenes with multiple simultaneous streams for many centuries, and the rules governing the composition of Western polyphonic music have even been explicitly codified in a range of techniques referred to as “voice-leading”. These relatively simple but effective rules have the potential to help guide designers of auditory displays by maximizing the distinctions between concurrent signals. An experiment was conducted to measure the discriminability of alarms designed with musical “voice-leading” features as compared with existing alarms from Learjet 31a and Learjet 35 aircraft. Signals designed with the auditory scene synthesis techniques embedded in musical “voice-leading” were found to significantly improve discriminability for up to five simultaneous alarms. By applying these principles to warning signals, this study has sought to implement a system for creating new auditory warnings that contain more efficient differentiating properties and furthermore conform to a more unified stylistic identity.

The present study aims to examine the effect of temporary visual deprivation on a process central to auditory segregation, namely harmonicity. A harmonicity discrimination task was administered twice, with an interval of 90 minutes, in two groups of individuals. One group was temporary deprived of visual information during that interval period. Control conditions revealed that auditory capabilities were similar across groups. However, results in the subsequent testing session revealed that temporarily deprived individuals showed a remarkable improvement in the auditory task in comparison with the non-visually-deprived group (i.e. control group). These results suggest that temporary blindness can improve auditory segregative processing.

The flexible nature of auditory cortex, the complexity of real-world sounds, and limitations of the methods for neural measurement in humans have made it difficult to investigate auditory feature information processing in the human brain. Which precise features are encoded in auditory cortex and the roles they play in different cognitive tasks remained unclear. New methods for functional magnetic resonance (fMRI) provide solutions to these limitations. New acquisition sequences make less noise, improving the suitability of fMRI for auditory research. Real-time adaptive fMRI and multivariate pattern analysis methods are robust to individual differences in anatomy and exploit information in distributed neural networks. They allow assessment of which acoustic and abstracted features of simple and natural sounds are represented in human auditory regions during perception, and cognitive tasks such as change detection and imagery. The results indicate that auditory cortex is recruited for processes beyond analy...

The object-based account of auditory scene analysis posits that attention operates on perceptual auditory objects. An important implication of such a theory is that perception of two simultaneous auditory objects may interfere with signal detection. In a series of experiments, we show that perception of concurrent sound objects, induced by varying frequency of one tonal component in an otherwise periodic sound complex, impaired gap detection. This effect was observed for a wide range of gap duration, and was greater when the mistuned harmonic was perceived as a separate object. These results suggest that one auditory object is processed at a time, which is consistent with the object-based theory. The impaired gap detection in the mistuned harmonic could be interpreted in terms of competition for attention between the gap and the mistuned harmonic: The perception of the mistuned harmonic as a separate object “wins” the competition for attentional resources.

Cocktail parties and other complex acoustic scenes present organisms with intricate sound mixtures and configurations. Perception in these complex settings relies on tracking regularities over time of sound patterns that arise from a statistical parsing of the scene as well as priors and expectations that bias how we organize the scene into its putative sound objects. Predictions arising from these expectations and sound regularities operate differently along different acoustic and cognitive domains. Here, we discuss the role of the interplay of expectations along these different domains in mediating the organization of complex acoustic scenes.

The functional models of auditory scene analysis (ASA) available in the literature have several limitations. First, they independently implement various principles and theories that are specific to the auditory modality. Second, they rarely account for top-down, high level, cognitive effects on ASA. The present paper aims to propose a new integrated model of ASA and reports results within a connectionist modeling framework to account for a wide range of effects on auditory scene analysis. The used connectionist framework is conformed to the known functional and anatomical constraints regarding the biological principles underlying auditory processing. This new neurocomputational model is specifically dedicated to account for top-down effects on ASA, such as attentional control, long-term memory knowledge effects and cross-modal interactions.

Many sound sources can only be recognised from the pattern of sounds they emit, and not from the individual sound events that make up their emission sequences. We propose a new model of auditory scene analysis, at the core of which is a process that seeks to discover predictable patterns in the ongoing sound sequence. Representations of predictable fragments are created on the fly, and are maintained, strengthened or weakened on the basis of their predictive success. Auditory perceptual organisation emerges from the competition between these representations (auditory proto-objects). Rather than positing a global interaction between all currently active proto-objects, competition is local and occurs between proto-objects that predict the same event at the same time. The model has been evaluated using the auditory streaming paradigm, and provides an intuitive account for many important phenomena including the emergence of, and switching between, alternative organisations, the influence of stimulus parameter...

Auditory scene analysis needs to parse the incoming flow of acoustic information into perceptual streams, such as distinct musical melodies or sentences from a single talker. Previous studies have demonstrated that the formation of auditory streams is not instantaneous: rather, streaming builds up over time and can be reset by sudden changes in the acoustics of the scene. The present study examined the effect of changes induced by voluntary head motion on streaming. A telepresence robot in a virtual-reality setup was used to disentangle all potential consequences of head motion: changes in acoustic cues at the ears, changes in apparent sound location, and changes in motor processes. The results showed that self-motion induced resetting of auditory streaming. An additive model analysis further revealed that resetting was largely influenced by acoustic cues and apparent sound location rather than by non-auditory factors related to head motion. Thus, low-level changes in sensory cues can affect perceptual or...