Search

How did Renaissance listeners experience the polyphonic mass ordinary cycle in the soundscape of the church? We hypothesize that the textural differences in complexity between mass movements allowed listeners to track the progress of the service, regardless of intelligibility of the text or sophisticated musical knowledge. Building on the principles of auditory scene analysis, this article introduces the Auditory Streaming Complexity Estimate, a measure to evaluate the blending or separation of each part in polyphony, resulting in a moment-by-moment tally of how many independent streams or sound objects might be heard. When applied to symbolic scores for a corpus of 216 polyphonic mass ordinary cycles composed between c. 1450 and 1600, we show that the Streaming Complexity Estimate captures information distinct from the number of parts in the score or the distribution of voices active through the piece. While composers did not all follow the same relative complexity strategy for mass ordinary movements, there is a robust hierarchy emergent from the corpus as a whole: a shallow V shape with the Credo as the least complex and the Agnus Dei as the most. The streaming complexity of masses also significantly increased over the years represented in this corpus.

The brain is highly sensitive to auditory regularities and exploits the predictable order of sounds in many situations, from parsing complex auditory scenes, to the acquisition of language. To understand the impact of stimulus predictability on perception, it is important to determine how the detection of predictable structure influences processing and attention. Here, we use pupillometry to gain insight into the effect of sensory regularity on arousal. Pupillometry is a commonly used measure of salience and processing effort, with more perceptually salient or perceptually demanding stimuli consistently associated with larger pupil diameters. In two experiments we tracked human listeners' pupil dynamics while they listened to sequences of 50-ms tone pips of different frequencies. The order of the tone pips was either random, contained deterministic (fully predictable) regularities (experiment 1, n = 18, 11 female) or had a probabilistic regularity structure (experiment 2, n = 20, 17 female). The sequences were rapid, preventing conscious tracking of sequence structure thus allowing us to focus on the automatic extraction of different types of regularities. We hypothesized that if regularity facilitates processing by reducing processing demands, a smaller pupil diameter would be seen in response to regular relative to random patterns. Conversely, if regularity is associated with heightened arousal and attention (i.e., engages processing resources) the opposite pattern would be expected. In both experiments we observed a smaller sustained (tonic) pupil diameter for regular compared with random sequences, consistent with the former hypothesis and confirming that predictability facilitates sequence processing. SIGNIFICANCE STATEMENT The brain is highly sensitive to auditory regularities. To appreciate the impact that the presence of predictability has on perception, we need to better understand how a predictable structure influences processing and attention. We recorded listeners' pupil responses to sequences of tones that followed either a predictable or unpredictable pattern, as the pupil can be used to implicitly tap into these different cognitive processes. We found that the pupil showed a smaller sustained diameter to predictable sequences, indicating that predictability eased processing rather than boosted attention. The findings suggest that the pupil response can be used to study the automatic extraction of regularities, and that the effects are most consistent with predictability helping the listener to efficiently process upcoming sounds.

This paper proposes an environmental sound segmentation method using Mask U-Net. In recent years, human–robot interactions, especially speech dialogue, have been assessed by auditory scene analysis...

Being able to pick out particular sounds, such as speech, against a background of other sounds represents one of the key tasks performed by the auditory system. Understanding how this happens is important because speech recognition in noise is particularly challenging for older listeners and for people with hearing impairments. Central to this ability is the capacity of neurons to adapt to the statistics of sounds reaching the ears, which helps to generate noise-tolerant representations of sounds in the brain. In more complex auditory scenes, such as a cocktail party — where the background noise comprises other voices, sound features associated with each source have to be grouped together and segregated from those belonging to other sources. This depends on precise temporal coding and modulation of cortical response properties when attending to a particular speaker in a multi-talker environment. Furthermore, the neural processing underlying auditory scene analysis is shaped by experience over multiple timescales.

Predictive coding is arguably the currently dominant theoretical framework for the study of perception. It has been employed to explain important auditory perceptual phenomena, and it has inspired theoretical, experimental and computational modelling efforts aimed at describing how the auditory system parses the complex sound input into meaningful units (auditory scene analysis). These efforts have uncovered some vital questions, addressing which could help to further specify predictive coding and clarify some of its basic assumptions. The goal of the current review is to motivate these questions and show how unresolved issues in explaining some auditory phenomena lead to general questions of the theoretical framework. We focus on experimental and computational modelling issues related to sequential grouping in auditory scene analysis (auditory pattern detection and bistable perception), as we believe that this is the research topic where predictive coding has the highest potential for advancing our understanding. In addition to specific questions, our analysis led us to identify three more general questions that require further clarification: (1) What exactly is meant by prediction in predictive coding? (2) What governs which generative models make the predictions? and (3) What (if it exists) is the correlate of perceptual experience within the predictive coding framework?

Auditory scene analysis is defined as a listener's ability to segregate a meaningful message from meaningless background noise in a listening environment. To gain better understanding of auditory perception in terms of message integration and segregation ability among concurrent signals, we aimed to systematically review the size of auditory scenes among individuals. A total of seven electronic databases were searched from 2000 to the present with related key terms. Using our inclusion criteria, 4,507 articles were classified according to four sequential steps-identification, screening, eligibility, included. Following study selection, the quality of four included articles was evaluated using the CAMARADES checklist. In general, studies concluded that the size of auditory scene increased as the number of sound sources increased; however, when the number of sources was five or higher, the listener's auditory scene analysis reached its maximum capability. Unfortunately, the score of study quality was not determined to be very high, and the number of articles used to calculate mean effect size and statistical significance was insufficient to draw significant conclusions. We suggest that study design and materials that consider realistic listening environments should be used in further studies to deep understand the nature of auditory scene analysis within various groups.

SummaryLocalizing and identifying sensory objects while navigating the environment are fundamental brain functions. However, how individual objects are neuronally represented during unrestricted self-motion is mostly unexplored. We trained gerbils on a behavioral foraging paradigm that required localization and identification of sound-sources during free navigation. Chronic multi-electrode recordings in primary auditory cortex during task performance revealed previously unreported sensory object representations. Strikingly, the egocentric angle preference of the majority of spatially sensitive neurons changed significantly depending on the task-specific identity (outcome association) of the sound-source. Spatial tuning also exhibited larger temporal complexity. Moreover, we encountered egocentrically untuned neurons whose response magnitude differed between source identities. Using a neural network decoder we show that together, these neuronal response ensembles provide spatio-temporally co-existent information about both the egocentric location and the identity of individual sensory objects during self-motion, revealing a novel cortical computation principle for naturalistic sensing.HighlightsLocalization task during free navigation prompts diverse spatial tuning in gerbil A1Spatial preference of individual neurons changes with sound-source identityEgo- and allocentric information are spatio-temporally coexistent in A1 ensemblesActive sensing reveals new cortical representations for sensory object identification

Temporal coherence of sound fluctuations across spectral channels is thought to aid auditory grouping and scene segregation. Although prior studies on the neural bases of temporal-coherence processing focused mostly on cortical contributions, neurophysiological evidence suggests that temporal-coherence-based scene analysis may start as early as the cochlear nucleus (i.e., the first auditory region supporting cross-channel processing over a wide frequency range). Accordingly, we hypothesized that aspects of temporal-coherence processing that could be realized in early auditory areas may shape speech understanding in noise. We then explored whether physiologically plausible computational models could account for results from a behavioral experiment that measured consonant categorization in different masking conditions. We tested whether within-channel masking of target-speech modulations predicted consonant confusions across the different conditions and whether predictions were improved by adding across-channel temporal-coherence processing mirroring the computations known to exist in the cochlear nucleus. Consonant confusions provide a rich characterization of error patterns in speech categorization, and are thus crucial for rigorously testing models of speech perception; however, to the best of our knowledge, they have not been used in prior studies of scene analysis. We find that within-channel modulation masking can reasonably account for category confusions, but that it fails when temporal fine structure cues are unavailable. However, the addition of across-channel temporal-coherence processing significantly improves confusion predictions across all tested conditions. Our results suggest that temporal-coherence processing strongly shapes speech understanding in noise and that physiological computations that exist early along the auditory pathway may contribute to this process.SIGNIFICANCE STATEMENTTemporal coherence of sound fluctuations across distinct frequency channels is thought to be important for auditory scene analysis. Prior studies on the neural bases of temporal-coherence processing focused mostly on cortical contributions, and it was unknown whether speech understanding in noise may be shaped by across-channel processing that exists in earlier auditory areas. Using physiologically plausible computational modeling to predict consonant confusions across different listening conditions, we find that across-channel temporal coherence contributes significantly to scene analysis and speech perception and that such processing may arise in the auditory pathway as early as the brainstem. By virtue of providing a richer characterization of error patterns not obtainable with just intelligibility scores, consonant confusions yield unique insight into scene analysis mechanisms.

Numerous neuroimaging studies demonstrated that the auditory cortex tracks ongoing speech and that, in multi-speaker environments, tracking of the attended speaker is enhanced compared to the other irrelevant speakers. In contrast to speech, multi-instrument music can be appreciated by attending not only on its individual entities (i.e., segregation) but also on multiple instruments simultaneously (i.e., integration). We investigated the neural correlates of these two modes of music listening using electroencephalography (EEG) and sound envelope tracking. To this end, we presented uniquely composed music pieces played by two instruments, a bassoon and a cello, in combination with a previously validated music auditory scene analysis behavioral paradigm (Disbergen et al., 2018). Similar to results obtained through selective listening tasks for speech, relevant instruments could be reconstructed better than irrelevant ones during the segregation task. A delay-specific analysis showed higher reconstruction for the relevant instrument during a middle-latency window for both the bassoon and cello and during a late window for the bassoon. During the integration task, we did not observe significant attentional modulation when reconstructing the overall music envelope. Subsequent analyses indicated that this null result might be due to the heterogeneous strategies listeners employ during the integration task. Overall, our results suggest that subsequent to a common processing stage, top-down modulations consistently enhance the relevant instrument’s representation during an instrument segregation task, whereas such an enhancement is not observed during an instrument integration task. These findings extend previous results from speech tracking to the tracking of multi-instrument music and, furthermore, inform current theories on polyphonic music perception.

Beamforming is a type of audio array processing techniques used for interference reduction, sound source localization, and as pre-processing stage for audio event classification and speaker identification. The auditory scene analysis community can benefit from a systemic evaluation and comparison between different beamforming techniques. In this paper, five popular beamforming techniques are evaluated in two different acoustic environments, while varying the number of microphones, the number of interferences, and the direction-of-arrival error, by using the Acoustic Interactions for Robot Audition (AIRA) corpus and a common software framework. Additionally, a highly efficient phase-based frequency masking beamformer is also evaluated, which is shown to outperform all five techniques. Both the evaluation corpus and the beamforming implementations are freely available and provided for experiment repeatability and transparency. Raw results are also provided as a complement to this work to the reader, to facilitate an informed decision of which technique to use. Finally, the insights and tendencies observed from the evaluation results are presented.

This paper considers the task of indoor subarea localization by acoustic sensing. In many practical scenarios, such as ordinary office rooms, subarea localization based on a single acoustic property without subsidiaries is always tricky due to similar space physical structures. To combat the indoor subarea localization problem without any dedicated devices, we propose a framework allowing the ordinary smartphone to quickly and easily determine its indoor subarea location. The proposed structure is to first construct the multidimensional fingerprint of each environmental background audio in a room using the auditory scene analysis method; then extract the room-level fingerprint by the acoustic background spectrum method, and cluster the subarea-level fingerprints using the Pearson correlation coefficient technology; subsequently, together with the offline stage long short-term memory recurrent neuron network and online stage least variance algorithm, the effective indoor subarea localization was achieved; finally, the localization results are presented not only in text form but also in a privacy-protected color sound pattern which is built by chroma DCT-reduced log pitch. Our experiments with real-world audios, which include 96-h environment background audios of 14 different rooms, presents that the room-level localization accuracy was higher than 90% and the subarea-level accuracy could be up to 97.64%. These results strongly proved that the proposed framework could perform well even in small complicated rooms.

This paper addresses the problem of 2D sound source localization using multiple microphone arrays in an outdoor environment. Two main issues exist in such localization. Since the localization perfo...

A listener who focusses on a sound source of interest must continuously integrate the sounds emitted by the attended source and ignore the sounds emitted by the remaining sources in the auditory scene. Little is known about how the ignored sound sources in the background are mentally represented after the source of interest has formed the perceptual foreground. This is due to a key methodological challenge: the background representation is by definition not overtly reportable. Here we developed a paradigm based on event-related brain potentials (ERPs) to assess the mental representation of background sounds. Participants listened to sequences of three repeatedly presented tones arranged in an ascending order (low, middle, high frequency). They were instructed to detect intensity deviants in one of the tones, creating the perceptual foreground. The remaining two background tones contained timing and location deviants. Those deviants were set up such that mismatch negativity (MMN) components would be elicited in distinct ways if the background was decomposed into two separate sound streams (background segregation) or if it was not further decomposed (background integration). Results provide MMN-based evidence for background segregation and integration in parallel. This suggests that mental representations of background integration and segregation can be concurrently available, and that collecting empirical evidence for only one of these background organization alternatives might lead to erroneous conclusions.

The Acoustic Interactions for Robot Audition corpus is introduced for research on sound source localization and separation, and for multi-user speech recognition. Its aim is to evaluate and train Robot Audition techniques, as well as Auditory Scene Analysis in general. It was recorded in six real-life environments with different noise presence and reverberation time, using two array configurations: an equilateral triangle, and a three-dimensional 16-microphone array set over a hollow plastic body. It includes clean speech data for static sources and tracking information for mobile sources. It is freely available at https://aira.iimas.unam.mx/.

Illusions give intriguing insights into perceptual and neural dynamics. In the auditory continuity illusion, two brief tones separated by a silent gap may be heard as one continuous tone if a noise burst with appropriate characteristics fills the gap. This illusion probes the conditions under which listeners link related sounds across time and maintain perceptual continuity in the face of sudden changes in sound mixtures. Conceptual explanations of this illusion have been proposed, but its neural basis is still being investigated. In this work we provide a dynamical systems framework, grounded in principles of neural dynamics, to explain the continuity illusion. We construct an idealized firing rate model of a neural population and analyze the conditions under which firing rate responses persist during the interruption between the two tones. First, we show that sustained inputs and hysteresis dynamics (a mismatch between tone levels needed to activate and inactivate the population) can produce continuous responses. Second, we show that transient inputs and bistable dynamics (coexistence of two stable firing rate levels) can also produce continuous responses. Finally, we combine these input types together to obtain neural dynamics consistent with two requirements for the continuity illusion as articulated in a well-known theory of auditory scene analysis: sustained responses occur if noise provides sufficient evidence that the tone continues and if there is no evidence of discontinuities between the tones and noise. By grounding these notions in a quantitative model that incorporates elements of neural circuits (recurrent excitation, and mutual inhibition, specifically), we identify plausible mechanisms for the continuity illusion. Our findings can help guide future studies of neural correlate of this illusion and inform development of more biophysically-based models of the auditory continuity illusion.

This paper addresses the low-dimensional embedding of sounds towards unsupervised auditory scene analysis. Summarizing long-time recordings by mapping them into lowdimensional space is an essential task in long-time environmental monitoring. In this paper, we propose a novel low-dimensional embedding system using an incremental embedding algorithm. To analyze long-time recordings, we design an incremental system consisting of recording, feature extraction, low-dimensional embedding, and visualization. Recently, many low-dimensional embedding methods for acoustic scenes have been studied; however, applicability of these methods to the long-time recording is not adequately evaluated. Thus, this paper describes the construction of the scene analysis system and evaluates the performance of this system. In this paper, we especially focus on two important viewpoints in long-time monitoring: incremental methods and effects of noisy data. To realize an incremental system, we use Self-Organizing Nebulous Growths (SONG), which can incrementally construct a low-dimensional embedding space. Also, in our experiments, we apply our system to bird song analysis under noise conditions. By the preliminary experiments using benchmark datasets, we discover noise sensitivity of our system and applicability to environmental monitoring.

In the speaker extraction problem, it is found that additional information from the target speaker contributes to the tracking and extraction of the target speaker, which includes voiceprint, lip movement, facial expression, and spatial information. However, no one cares for the cue of sound onset, which has been emphasized in the auditory scene analysis and psychology. Inspired by it, we explicitly modeled the onset cue and verified the effectiveness in the speaker extraction task. We further extended to the onset/offset cues and got performance improvement. From the perspective of tasks, our onset/offset-based model completes the composite task, a complementary combination of speaker extraction and speaker-dependent voice activity detection. We also combined voiceprint with onset/offset cues. Voiceprint models voice characteristics of the target while onset/offset models the start/end information of the speech. From the perspective of auditory scene analysis, the combination of two perception cues can promote the integrity of the auditory object. The experiment results are also close to state-of-the-art performance, using nearly half of the parameters. We hope that this work will inspire communities of speech processing and psychology, and contribute to communication between them. Our code will be available in https://github.com/aispeech-lab/wase/., Comment: Accepted by ICASSP 2021

Binaural cue processing requires central auditory function as damage to the auditory cortex and other cortical regions impairs sound localization. Sound localization cues are initially extracted by brainstem nuclei, but how the cerebral cortex supports spatial sound perception remains unclear. This chapter reviews the evidence that spatial encoding within and beyond the auditory cortex supports sound localization, including the integration of information across sound frequencies and localization cues. In particular, this chapter discusses the role of brain regions across the cerebral cortex that may be specialized for extracting and transforming the spatial aspects of sounds and extends from sensory to parietal and prefrontal cortices. The chapter considers how the encoding of spatial information changes with attention and how spatial processing fits within the broader context of auditory scene analysis by cortical networks. The importance of neural plasticity in binaural processing is outlined, including a discussion of how changes in the mapping of localization cues to spatial position allow listeners to adapt to changes in auditory input throughout life and after hearing loss. The chapter ends by summarizing some of the open questions about the central processing of binaural cues and how they may be answered.

Talker-independent monaural speaker separation aims to separate concurrent speakers from a single-microphone recording. Inspired by human auditory scene analysis (ASA) mechanisms, a two-stage deep CASA approach has been proposed recently to address this problem, which achieves state-of-the-art results in separating mixtures of two or three speakers. A main limitation of deep CASA is that it is a non-causal system, while many speech processing applications, e.g., telecommunication and hearing prosthesis, require causal processing. In this study, we propose a causal version of deep CASA to address this limitation. First, we modify temporal connections, normalization and clustering algorithms in deep CASA so that no future information is used throughout the deep network. We then train a $C$ -speaker ( $C \geq 2$ ) deep CASA system in a speaker-number-independent fashion, generalizable to speech mixtures with up to $C$ speakers without the prior knowledge about the speaker number. Experimental results show that causal deep CASA achieves excellent speaker separation performance with known or unknown speaker numbers.

International audience; The Internet of Audio Things (IoAuT) is an emerging research field positioned at the intersection of the Internet of Things, sound and music computing, artificial intelligence, and human-computer interaction. The IoAuT refers to the networks of computing devices embedded in physical objects (Audio Things) dedicated to the production, reception, analysis and understanding of audio in distributed environments. Audio Things, such as nodes of wireless acoustic sensor networks, are connected by an infrastructure that enables multidirectional communication, both locally and remotely. In this paper, we first review the state of the art of this field, then we present a vision for the IoAuT and its motivations. In the proposed vision, the IoAuT enables the connection of digital and physical domains by means of appropriate information and communication technologies, fostering novel applications and services based on auditory information. The ecosystems associated with the IoAuT include interoperable devices and services that connect humans and machines to support human-human and human-machines interactions. We discuss challenges and implications of this field, which lead to future research directions on the topics of privacy, security, design of Audio Things, and methods for the analysis and representation of audio-related information.

Sound is caused by physical events in the world. Do humans infer these causes when recognizing sound sources? We tested whether the recognition of common environmental sounds depends on the inference of a basic physical variable – the source intensity (i.e., the power that produces a sound). A source’s intensity can be inferred from the intensity it produces at the ear and its distance, which is normally conveyed by reverberation. Listeners could thus use intensity at the ear and reverberation to constrain recognition by inferring the underlying source intensity. Alternatively, listeners might separate these acoustic cues from their representation of a sound’s identity in the interest of invariant recognition. We compared these two hypotheses by measuring recognition accuracy for sounds with typically low or high source intensity (e.g., pepper grinders vs. trucks) that were presented across a range of intensities at the ear or with reverberation cues to distance. The recognition of low-intensity sources (e.g., pepper grinders) was impaired by high presentation intensities or reverberation that conveyed distance, either of which imply high source intensity. Neither effect occurred for high-intensity sources. The results suggest that listeners implicitly use the intensity at the ear along with distance cues to infer a source’s power and constrain its identity. The recognition of real-world sounds thus appears to depend upon the inference of their physical generative parameters, even generative parameters whose cues might otherwise be separated from the representation of a sound’s identity.

Artımlı öğrenme ile sahne analizi, farklı duyusal modaliteler kullanarak geçmiş deneyimlerden daha önce bilgi sahibi olunmayan olayları, eylemleri ve hatta gürültü modellerini aşamalı olarak öğrenmek için durmaksızın gerçekleşen bir süreçtir. Bu çalışmada, dinamik olarak değişen gerçek bir ev ortamında akustik olayları aşamalı olarak öğrenmek için artımlı bir öğrenme sistemine dayanan İşitsel Sahne Analizi (ASA) yaklaşımı sunulmuştur. Ortamdaki en baskın ses kaynakları olan olaylar, birden fazla kaynağın bulunduğu işitsel sahnede bu kaynaktan elde edilen sinyalleri verimli ve kesintisiz bir şekilde işlemek için bir Ses Kaynağı Yerelleştirme (SSL) yöntemi ile yer tespiti yapılmaktadır. Gerçek zamanlı sahne analizinde, ses örüntüleri, bu örüntülerden ses özniteliklerin çıkarılması ve öznitelik setinin oluşturulması için bu kaynağın akustik sinyal akışından segmente edilir. Artımlı öğrenme, kaynaklardan elde edilen akustik sinyallerden bu öznitelik kümelerinde zaman serisi algoritması tabanlı olan Gizli Markov Modeli (HMM) kullanılmıştır. Öğrenme süreci, Bilinmeyen Olay Algılama (UED), Akustik Olay Tanıma (AER) ve Hiyerarşik HMM yöntemi kullanarak sürekli öğrenmenin performansını değerlendirmek için çeşitli deneyler yapılarak geliştirilmiştir. Hiyerarşik HMM iki katmandan oluşur: 1) AER'nin her bir olay için HMM ve olay bazlı eşik değerleri kullanılarak gerçekleştirildiği bir alt katman; ve 2) bir ses önitelik seti için ilgili alt katman HMM’inden çıkartılan proto sembolleri ile ses özniteliklerinin birleştirilip bir HMM ile bir şüphe eşk değeri kullanılarak UED’nin gerçeklştirildiği bir üst katman. Artımlı öğrenme, AER ve UED’e sahip bu sistemin, Yanlış-Olumlu Oranlar, Doğru-Olumlu Oranlar, tanıma doğruluğu ve hesaplama süresi gözetilerek birden fazla olayın söz konusu olduğu gerçek zamanlı öğrenme için gereken gereksinimleri karşılayacak seviyede olduğunu doğruladık. AER sisteminin etkinliği, yüksek doğruluk ve dokuz farklı ses içeren gerçek zamanlı ASA'da kısa bir yeniden eğitim süresi ile doğrulanmıştır., Continual learning for scene analysis is a continuous process to incrementally learn distinct events, actions, and even noise models from past experiences using different sensory modalities. In this paper, an Auditory Scene Analysis (ASA) approach based on a continual learning system is developed to incrementally learn the acoustic events in a dynamically-changing domestic environment. The events being salient sound sources are localized by a Sound Source Localization (SSL) method to robustly process the signals of the localized sound source in the domestic scene where multiple sources can co-exist. For real-time ASA, audio patterns are segmented from the acoustic signal stream of the localized source for extraction of the audio features, and construction of a feature set for each pattern. The continual learning is employed via a time-series algorithm, Hidden Markov Model (HMM), on these feature sets from acoustic signals stemming from the sources. The learning process is investigated by conducting a variety of experiments to evaluate the performance of Unknown Event Detection (UED), Acoustic Event Recognition (AER), and continual learning using a Hierarchical HMM algorithm. The Hierarchical HMM consists of two layers: 1) a lower layer in which AER is performed using an HMM for each event and the event-wise likelihood thresholds; and 2) an upper layer in which UED is achieved by one HMM with a suspicion threshold through the audio features with their proto symbols stemming from the lower layer HMMs. We verified the effectiveness of the proposed system capable of continual learning, AER and UED in terms of False-Positive Rates, True-Positive Rates, recognition accuracy and computational time to meet the demands in a learning task of multiple events in real-time. The effectiveness of the AER system has been verified with high accuracy, and a short retraining time in real-time ASA having nine different sounds.

Auditory cortex (AC) is necessary for the detection of brief gaps in ongoing sounds, but not for the detection of longer gaps or other stimuli such as tones or noise. It remains unclear why this is so, and what is special about brief gaps in particular. Here, we used both optogenetic suppression and conventional lesions to show that the cortical dependence of brief gap detection hinges specifically on gap termination. We then identified a cortico-collicular gap detection circuit that amplifies cortical gap termination responses before projecting to inferior colliculus (IC) to impact behavior. We found that gaps evoked off-responses and on-responses in cortical neurons, which temporally overlapped for brief gaps, but not long gaps. This overlap specifically enhanced cortical responses to brief gaps, whereas IC neurons preferred longer gaps. Optogenetic suppression of AC reduced collicular responses specifically to brief gaps, indicating that under normal conditions, the enhanced cortical representation of brief gaps amplifies collicular gap responses. Together these mechanisms explain how and why AC contributes to the behavioral detection of brief gaps, which are critical cues for speech perception, perceptual grouping, and auditory scene analysis.

This article unifies neural modeling results that illustrate several basic design principles and mechanisms that are used by advanced brains to develop cortical maps with multiple psychological functions. One principle concerns how brains use a strip map that simultaneously enables one feature to be represented throughout its extent, as well as an ordered array of another feature at different positions of the strip. Strip maps include circuits to represent ocular dominance and orientation columns, place-value numbers, auditory streams, speaker-normalized speech, and cognitive working memories that can code repeated items. A second principle concerns how feature detectors for multiple functions develop in topographic maps, including maps for optic flow navigation, reinforcement learning, motion perception, and category learning at multiple organizational levels. A third principle concerns how brains exploit a spatial gradient of cells that respond at an ordered sequence of different rates. Such a rate gradient is found along the dorsoventral axis of the entorhinal cortex, whose lateral branch controls the development of time cells, and whose medial branch controls the development of grid cells. Populations of time cells can be used to learn how to adaptively time behaviors for which a time interval of hundreds of milliseconds, or several seconds, must be bridged, as occurs during trace conditioning. Populations of grid cells can be used to learn hippocampal place cells that represent the large spaces in which animals navigate. A fourth principle concerns how and why all neocortical circuits are organized into layers, and how functionally distinct columns develop in these circuits to enable map development. A final principle concerns the role of Adaptive Resonance Theory top-down matching and attentional circuits in the dynamic stabilization of early development and adult learning. Cortical maps are modeled in visual, auditory, temporal, parietal, prefrontal, entorhinal, and hippocampal cortices.

Online audio source separation has been an important part of auditory scene analysis and robot audition. The main type of technique to carry this out, because of its online capabilities, has been spatial filtering (or beamforming), where it is assumed that the location (mainly, the direction of arrival; DOA) of the source of interest (SOI) is known. However, these techniques suffer from considerable interference leakage in the final result. In this paper, we propose a two step technique: 1) a phase-based beamformer that provides, in addition to the estimation of the SOI, an estimation of the cumulative environmental interference; and 2) a BLSTM-based TF binary masking stage that calculates a binary mask that aims to separate the SOI from the cumulative environmental interference. In our tests, this technique provides a signal-to-interference ratio (SIR) above 20 dB with simulated data. Because of the nature of the beamformer outputs, the label permutation problem is handled from the beginning. This makes the proposed solution a lightweight alternative that requires considerably less computational resources (almost an order of magnitude) compared to current deep-learning based techniques, while providing a comparable SIR performance., Comment: Accepted for publication in Computaci\'on y Sistemas

Multisensory responses have been observed throughout the central auditory pathway, yet the origin, function and perceptual consequences of cross-modal integration remain unresolved. Recent studies have applied modern neuroanatomical and functional perturbation techniques to dissect the circuits that might enable multisensory information to sculpt the processing of sound. These highlight in particular the role that subcortical pathways might play in relaying multisensory information to, and between, sensory cortical fields. We also examine the consequences of integrating non-auditory information into auditory processing, identifying key areas where this may be critical for successful listening and potential roles for visual information in augmenting auditory scene analysis, and for non-auditory information in facilitating coordinate frame transformations.

Current developments in modeling the auditory system lead to increasing inclusion of cognitive functions, such as dynamic auditory scene analysis. This qualifies these systems as auditory front-ends for autonomous agents. Such agents can, for example, be mobile robotic systems, that is, they can move around in their environments, explore them, and develop internal models of them. Thereby, they can monitor their environments and become active in cases where potentially hazardous things happen. For example, in a Search-&-Rescue scenario (SAR), the agents could identify and save persons in dangerous situations. In this chapter, a virtual testbed for such systems is described that was developed in the EU project Two!Ears (www.twoears.eu) There, in simulated scenarios, the agents have to localize and identify potential victims and, consequently, rescue them according to dynamic SAR plans. The actions are predominantly based on binaural cues, derived from the two ear signals of head-and-torso simulators (dummy heads) on carriages that can actively move about in the scenes to be explored. Such a simulation system can provide a tool to monitor and evaluate the cognitive processes of autonomous systems while these are dynamically executing assigned tasks.

Acoustic communication in the natural world requires both emitter and receiver to adapt to the loss of information due to the transmission of sound in the environment. At the emitter’s end, encoding information into propagation-resistant features may ensure its transmission on ecologically relevant distances. At the receiver’s end, making sense of the degraded signal merged with added noise enables animals to produce behaviorally relevant responses. In this chapter, I use a songbird model to investigate both sides of the transmission chain, examining how individual vocal signatures are encoded, degraded and finally discriminated, with a particular emphasis on the receiver’s neural encoding of degraded information. The cornerstone of this focal study is the use of naturally degraded vocalizations, combining various aspects of the challenges faced by animals performing auditory scene analysis, in order to address the complexity of real-life ecological constraints. While the individual signature of our songbird model, the zebra finch, is very resistant to propagation-induced degradation, single neurons in the avian auditory cortex have the ability to discriminate highly degraded individual vocal signatures, without prior familiarization or training. In the light of past and recent findings in birds and mammals, I delve further into significant insights uncovered by the research on neural processing and ensemble coding, and discuss the roles of perceptual plasticity and learning in the neural interface between brain and behavior.

This study examines sound perception and tests the functionality of harmonicity as a metric of auditory preference and regional brain activity. Most salient sounds contain harmonic overtones, a rel...

Speech separation based on auditory scene analysis (ASA) has been widely studied. A computational ASA (CASA) model, in which a mixed signal is sequentially decomposed into frequency signals, has been also proposed. Four feature types of ASA are extracted from the decomposed frequency signals, and the decomposed frequency signals are regrouped by examining the characteristics of the extracted features. Finally separated speeches are obtained. In this study, the CASA model is improved and pieced out, and the separation performance is examined via a computer simulation.

Polyphonic music listening well exemplifies processes typically involved in daily auditory scene analysis situations, relying on an interactive interplay between bottom-up and top-down processes. Most studies investigating scene analysis have used elementary auditory scenes, however real-world scene analysis is far more complex. In particular, music, contrary to most other natural auditory scenes, can be perceived by either integrating or, under attentive control, segregating sound streams, often carried by different instruments. One of the prominent bottom-up cues contributing to multi-instrument music perception is their timbre difference. In this work, we introduce and validate a novel paradigm designed to investigate, within naturalistic musical auditory scenes, attentive modulation as well as its interaction with bottom-up processes. Two psychophysical experiments are described, employing custom-composed two-voice polyphonic music pieces within a framework implementing a behavioral performance metric to validate listener instructions requiring either integration or segregation of scene elements. In Experiment 1, the listeners' locus of attention was switched between individual instruments or the aggregate (i.e., both instruments together), via a task requiring the detection of temporal modulations (i.e., triplets) incorporated within or across instruments. Subjects responded post-stimulus whether triplets were present in the to-be-attended instrument(s). Experiment 2 introduced the bottom-up manipulation by adding a three-level morphing of instrument timbre distance to the attentional framework. The task was designed to be used within neuroimaging paradigms; Experiment 2 was additionally validated behaviorally in the functional Magnetic Resonance Imaging (fMRI) environment. Experiment 1 subjects (N = 29, non-musicians) completed the task at high levels of accuracy, showing no group differences between any experimental conditions. Nineteen listeners also participated in Experiment 2, showing a main effect of instrument timbre distance, even though within attention-condition timbre-distance contrasts did not demonstrate any timbre effect. Correlation of overall scores with morph-distance effects, computed by subtracting the largest from the smallest timbre distance scores, showed an influence of general task difficulty on the timbre distance effect. Comparison of laboratory and fMRI data showed scanner noise had no adverse effect on task performance. These Experimental paradigms enable to study both bottom-up and top-down contributions to auditory stream segregation and integration within psychophysical and neuroimaging experiments.

In order to solve the problem of instability of the traditional speech separation algorithm, a kind of reverberation speech separation model based on deep learning is proposed. The problem of speech separation in reverberation environment has been studied. The auditory scene analysis is used to simulate the human auditory perception ability. According to the ideal two value mode principle, the target speech signal can be extracted. Moreover, the deep neural network (DNN) shows great learning ability in speech recognition and artificial intelligence. In this paper, a DNN model is proposed to learn the inverse reverberation and denoising by learning the spectrum mapping between “contaminated” speech and pure speech. By extracting a series of spectrum features, the time dynamic information of adjacent frames is fused. The DNN is used to transform the coded spectrum, and restore the pure voice frequency spectrum. Finally, the time domain signal is reconstructed. In addition, the feature classification ability of DNN is also proposed to complete the separation of double sound reverberation speech. The binaural features ITD and ILD and the mono features GFCC are fused to form a long eigenvector. The DNN is pre-trained by RBM to complete the classification task. The results show that the proposed model improves the quality and intelligibility of the speech separation, and enhances the stability of the system significantly.

Echolocating bats use echoes of their sonar emissions to determine the position and distance of objects or prey. Target distance is represented as a map of echo delay in the auditory cortex (AC) of bats. During a bat’s flight through a natural complex environment, echo streams are reflected from multiple objects along its flight path. Separating such complex streams of echoes or other sounds is a challenge for the auditory system of bats as well as other animals. We investigated the representation of multiple echo streams in the AC of anesthetized bats ( Phyllostomus discolor) and tested the hypothesis that neurons can lock on echoes from specific objects in a complex echo-acoustic pattern while the representation of surrounding objects is suppressed. We combined naturalistic pulse/echo sequences simulating a bat’s flight through a virtual acoustic space with extracellular recordings. Neurons could selectively lock on echoes from one object in complex echo streams originating from two different objects along a virtual flight path. The objects were processed sequentially in the order in which they were approached. Object selection depended on sequential changes of echo delay and amplitude, but not on absolute values. Furthermore, the detailed representation of the object echo delays in the cortical target range map was not fixed but could be dynamically adapted depending on the temporal pattern of sonar emission during target approach within a simulated flight sequence. NEW & NOTEWORTHY Complex signal analysis is a challenging task in sensory processing for all animals, particularly for bats because they use echolocation for navigation in darkness. Recent studies proposed that the bat’s perceptional system might organize complex echo-acoustic information into auditory streams, allowing it to track specific auditory objects during flight. We show that in the auditory cortex of bats, neurons can selectively respond to echo streams from specific objects.

Perceptually, grouping sounds based on their sources is critical for communication. This is especially true in túngara frog breeding aggregations, where multiple males produce overlapping calls that consist of an FM 'whine' followed by harmonic bursts called 'chucks'. Phonotactic females use at least two cues to group whines and chucks: whine-chuck spatial separation and sequence. Spatial separation is a primitive cue, whereas sequence is schema-based, as chuck production is morphologically constrained to follow whines, meaning that males cannot produce the components simultaneously. When one cue is available, females perceptually group whines and chucks using relative comparisons: components with the smallest spatial separation or those closest to the natural sequence are more likely grouped. By simultaneously varying the temporal sequence and spatial separation of a single whine and two chucks, this study measured between-cue perceptual weighting during a specific grouping task. Results show that whine-chuck spatial separation is a stronger grouping cue than temporal sequence, as grouping is more likely for stimuli with smaller spatial separation and non-natural sequence than those with larger spatial separation and natural sequence. Compared to the schema-based whine-chuck sequence, we propose that spatial cues have less variance, potentially explaining their preferred use when grouping during directional behavioral responses.

Compared to sighted listeners, blind listeners often display enhanced auditory spatial abilities such as localization in azimuth. However, less is known about whether blind humans can accurately judge distance in extrapersonal space using auditory cues alone. Using virtualization techniques, we show that auditory spatial representations of the world beyond the peripersonal space of blind listeners are compressed compared to those for normally sighted controls. Blind participants overestimated the distance to nearby sources and underestimated the distance to remote sound sources, in both reverberant and anechoic environments, and for speech, music, and noise signals. Functions relating judged and actual virtual distance were well fitted by compressive power functions, indicating that the absence of visual information regarding the distance of sound sources may prevent accurate calibration of the distance information provided by auditory signals.

Sound sources in the world are experienced as stable even when intermittently obscured, implying perceptual completion mechanisms that “fill in” missing sensory information. We demonstrate a filling-in phenomenon in which the brain extrapolates the statistics of background sounds (textures) over periods of several seconds when they are interrupted by another sound, producing vivid percepts of illusory texture. The effect differs from previously described completion effects in that 1) the extrapolated sound must be defined statistically given the stochastic nature of texture, and 2) the effect lasts much longer, enabling introspection and facilitating assessment of the underlying representation. Illusory texture biases subsequent texture statistic estimates indistinguishably from actual texture, suggesting that it is represented similarly to actual texture. The illusion appears to represent an inference about whether the background is likely to continue during concurrent sounds, providing a stable statistical representation of the ongoing environment despite unstable sensory evidence., Auditory textures are sounds defined by a particular statistical distribution, e.g. as is produced by rain, or a swarm of insects. Here, the authors describe a striking perceptual illusion in which sound textures are heard to continue, even though they have in fact been replaced by white noise.

Auditory stream segregation is a perceptual process by which the human auditory system groups sounds from different sources into perceptually meaningful elements (e.g., a voice or a melody). The perceptual segregation of sounds is important, for example, for the understanding of speech in noisy scenarios, a particularly challenging task for listeners with a cochlear implant (CI). It has been suggested that some aspects of stream segregation may be explained by relatively basic neural mechanisms at a cortical level. During the past decades, a variety of models have been proposed to account for the data from stream segregation experiments in normal-hearing (NH) listeners. However, little attention has been given to corresponding findings in CI listeners. The present study investigated whether a neural model of sequential stream segregation, proposed to describe the behavioral effects observed in NH listeners, can account for behavioral data from CI listeners. The model operates on the stimulus features at the cortical level and includes a competition stage between the neuronal units encoding the different percepts. The competition arises from a combination of mutual inhibition, adaptation, and additive noise. The model was found to capture the main trends in the behavioral data from CI listeners, such as the larger probability of a segregated percept with increasing the feature difference between the sounds as well as the build-up effect. Importantly, this was achieved without any modification to the model's competition stage, suggesting that stream segregation could be mediated by a similar mechanism in both groups of listeners.

In a complex auditory scene, speech comprehension involves several stages: for example segregating the target from the background, recognizing syllables and integrating syllables into linguistic units (e.g., words). Although speech segregation is robust as shown by invariant neural tracking to target speech envelope, whether neural tracking to linguistic units is also robust and how this robustness is achieved remain unknown. To investigate these questions, we concurrently recorded neural responses tracking a rhythmic speech stream at its syllabic and word rates, using electroencephalography. Human participants listened to that target speech under a speech or noise distractor at varying signal-to-noise ratios. Neural tracking at the word rate was not as robust as neural tracking at the syllabic rate. Robust neural tracking to target's words was only observed under the speech distractor but not under the noise distractor. Moreover, this robust word tracking correlated with a successful suppression of distractor tracking. Critically, both word tracking and distractor suppression correlated with behavioural comprehension accuracy. In sum, our results suggest that a robust neural tracking of higher-level linguistic units relates to not only the target tracking, but also the distractor suppression.

Voice leading is a common task in Western music composition whose conventions are consistent with fundamental principles of auditory perception. Here we introduce a computational cognitive model of voice leading, intended both for analyzing voice-leading practices within encoded musical corpora and for generating new voice leadings for unseen chord sequences. This model is feature-based, quantifying the desirability of a given voice leading on the basis of different features derived from Huron’s (2001) perceptual account of voice leading. We use the model to analyze a corpus of 370 chorale harmonizations by J. S. Bach, and demonstrate the model’s application to the voicing of harmonic progressions in different musical genres. The model is implemented in a new R package, “voicer”, which we release alongside this paper.

Despite many behavioral and neuroimaging investigations, it remains unclear how the human cortex represents spectrotemporal sound features during auditory imagery, and how this representation compares to auditory perception. To assess this, we recorded electrocorticographic signals from an epileptic patient with proficient music ability in 2 conditions. First, the participant played 2 piano pieces on an electronic piano with the sound volume of the digital keyboard on. Second, the participant replayed the same piano pieces, but without auditory feedback, and the participant was asked to imagine hearing the music in his mind. In both conditions, the sound output of the keyboard was recorded, thus allowing precise time-locking between the neural activity and the spectrotemporal content of the music imagery. This novel task design provided a unique opportunity to apply receptive field modeling techniques to quantitatively study neural encoding during auditory mental imagery. In both conditions, we built encoding models to predict high gamma neural activity (70-150 Hz) from the spectrogram representation of the recorded sound. We found robust spectrotemporal receptive fields during auditory imagery with substantial, but not complete overlap in frequency tuning and cortical location compared to receptive fields measured during auditory perception.

Highlights • Perception relies on sensitivity to predictable structure in the environment. • We used artificial acoustic scenes to investigate this in the auditory modality. • Listeners track the temporal structure of multiple concurrent acoustic streams. • Sensitivity to predictable structure supports auditory scene analysis, even when scenes are complex. • Benefit of regularity observed even when listeners are unaware of the predictable structure., The notion that sensitivity to the statistical structure of the environment is pivotal to perception has recently garnered considerable attention. Here we investigated this issue in the context of hearing. Building on previous work (Sohoglu and Chait, 2016a; elife), stimuli were artificial ‘soundscapes’ populated by multiple (up to 14) simultaneous streams (‘auditory objects’) comprised of tone-pip sequences, each with a distinct frequency and pattern of amplitude modulation. Sequences were either temporally regular or random. We show that listeners’ ability to detect abrupt appearance or disappearance of a stream is facilitated when scene streams were characterized by a temporally regular fluctuation pattern. The regularity of the changing stream as well as that of the background (non-changing) streams contribute independently to this effect. Remarkably, listeners benefit from regularity even when they are not consciously aware of it. These findings establish that perception of complex acoustic scenes relies on the availability of detailed representations of the regularities automatically extracted from multiple concurrent streams.