Search

Research into multi-modal perception, human cognition, behavior, and attention can benefit from high-fidelity content that may recreate real-life-like scenes when rendered on head-mounted displays. Moreover, aspects of audiovisual perception, cognitive processes, and behavior may complement questionnaire-based Quality of Experience (QoE) evaluation of interactive virtual environments. Currently, there is a lack of high-quality open-source audiovisual databases that can be used to evaluate such aspects or systems capable of reproducing high-quality content. With this paper, we provide a publicly available audiovisual database consisting of twelve scenes capturing real-life nature and urban environments with a video resolution of 7680x3840 at 60 frames-per-second and with 4th-order Ambisonics audio. These 360 video sequences, with an average duration of 60 seconds, represent real-life settings for systematically evaluating various dimensions of uni-/multi-modal perception, cognition, behavior, and QoE. The paper provides details of the scene requirements, recording approach, and scene descriptions. The database provides high-quality reference material with a balanced focus on auditory and visual sensory information. The database will be continuously updated with additional scenes and further metadata such as human ratings and saliency information., 6 pages, 2 figures, accepted and presented at the 2022 14th International Conference on Quality of Multimedia Experience (QoMEX). Database is publicly accessible at https://qoevave.github.io/database/

The accuracy of computational models for acoustics is often limited by a lack of reliable information concerning the frequency-dependent impedance of surface materials. This lack of information stems from the unavailability of reliable measurement methods for low frequencies. In this work, an approach is proposed, using eigenvalue analysis, for estimating the locally reacting, frequency-dependent impedance of a sound-absorbing sample. In particular, an eigenvalue approximation method is proposed and used in tandem with an optimization routine to obtain surface impedance estimates of an installed sample at modal frequencies. It is shown, using finite element simulations of an impedance tube and a small reverberation room, that the proposed method can provide reasonable estimates of the surface impedance of a sample placed on a boundary surface.

Auditory localization is affected by visual cues. The study at hand focuses on a scenario where dynamic sound localization cues are induced by lateral listener self-translation in relation to a stationary sound source with matching or mismatching dynamic visual cues. The audio-only self-translation minimum audible angle (ST-MAA) is previously shown to be 3.3° in the horizontal plane in front of the listener. The present study found that the addition of visual cues has no significant effect on the ST-MAA.

The estimation of reverberation time from real-world signals plays a central role in a wide range of applications. In many scenarios, acoustic conditions change over time which in turn requires the estimate to be updated continuously. Previously proposed methods involving deep neural networks were mostly designed and tested under the assumption of static acoustic conditions. In this work, we show that these approaches can perform poorly in dynamically evolving acoustic environments. Motivated by a recent trend towards data-centric approaches in machine learning, we propose a novel way of generating training data and demonstrate, using an existing deep neural network architecture, the considerable improvement in the ability to follow temporal changes in reverberation time., accepted for publication in ICASSP 2022

Many quality evaluation methods are used to assess uni-modal audio or video content without considering perceptual, cognitive, and interactive aspects present in virtual reality (VR) settings. Consequently, little is known regarding the repercussions of the employed evaluation method, content, and subject behavior on the quality ratings in VR. This mixed between- and within-subjects study uses four subjective audio quality evaluation methods (viz. multiple-stimulus with and without reference for direct scaling, and rank-order elimination and pairwise comparison for indirect scaling) to investigate the contributing factors present in multi-modal 6-DoF VR on quality ratings of real-time audio rendering. For each between-subjects employed method, two sets of conditions in five VR scenes were evaluated within-subjects. The conditions targeted relevant attributes for binaural audio reproduction using scenes with various amounts of user interactivity. Our results show all referenceless methods produce similar results using both condition sets. However, rank-order elimination proved to be the fastest method, required the least amount of repetitive motion, and yielded the highest discrimination between spatial conditions. Scene complexity was found to be a main effect within results, with behavioral and task load index results implying more complex scenes and interactive aspects of 6-DoF VR can impede quality judgments.

Signal extraction from a single-channel mixture with additional undesired signals is most commonly performed using time-frequency (TF) masks. Typically, the mask is estimated with a deep neural network (DNN), and element-wise applied to the complex mixture short-time Fourier transform (STFT) representation to perform the extraction. Ideal mask magnitudes are zero for solely undesired signals in a TF bin and undefined for total destructive interference. Usually, masks have an upper bound to provide well-defined DNN outputs at the cost of limited extraction capabilities. We propose to estimate with a DNN a complex TF filter for each mixture TF bin which maps an STFT area in the respective mixture to the desired TF bin to address destructive interference in mixture TF bins. The DNN is optimized by minimizing the error between the extracted and the ground-truth desired signal allowing to learn the TF filters without having to specify ground-truth TF filters. We compare our approach with complex and real-valued TF masks by separating speech from a variety of different sound and noise classes from the Google AudioSet corpus. We also process the mixture STFT with notch-filters and zero whole time-frames, to simulate packet-loss during transmission, to demonstrate the reconstruction capabilities of our approach. The proposed method outperformed the baselines, especially when notch-filters and time-frame zeroing were applied.

Outdoor recording is particularly challenging in the presence of wind, which induces highly non-stationary noise in the microphone signals. To enhance a desired signal, e.g., speech, a dedicated noise reduction processing is required. The reduction is usually performed by estimating an unknown set of parameters, e.g., the noise and the speech power spectral densities (PSDs). In contrast to the commonly used assumption of uncorrelated wind noise in multi-channel recordings, we assume the spatial correlation of wind noise contributions to be non-zero at low frequencies when closely-spaced microphones are employed. In our earlier work, we assumed that the spatial coherence was known, i.e., given by a model which depends on the free-field speed and direction of the air stream. As these quantities are unknown in practice, we propose in this work a method to recursively estimate the spatial coherence matrix based on the microphone observations. In addition, we prove the equivalence of two recently developed noise PSD estimation methods when uncorrelated wind noise is assumed, and we propose an approximation of both estimators which is independent of the propagation vector of the speech source at sufficiently low frequency and for a small inter-microphone distance. An evaluation in terms of improvements in speech quality, signal-to-noise ratio and intelligibility is carried out using both simulated and measured wind noise samples.

Sound reproduction systems may highly benefit from detailed knowledge of the acoustic space to enhance the spatial sound experience. This article presents a room geometry inference method based on identification of reflective boundaries using a high-resolution direction-of-arrival map produced via room impulse responses (RIRs) measured with a linear loudspeaker array and a single microphone. Exploiting the sparse nature of the early part of the RIRs, Elastic Net regularization is applied to obtain a 2D polar-coordinate map, on which the direct path and early reflections appear as distinct peaks, described by their propagation distance and direction of arrival. Assuming a separable room geometry with four side-walls perpendicular to the floor and ceiling, and imposing pre-defined geometrical constraints on the walls, the 2D-map is segmented into six regions, each corresponding to a particular wall. The salient peaks within each region are selected as candidates for the first-order wall reflections, and a set of potential room geometries is formed by considering all possible combinations of the associated peaks. The room geometry is then inferred using a cost function evaluated on the higher-order reflections computed via beam tracing. The proposed method is tested with both simulated and measured data.

Ambisonics encodes the directional information of a sound field w.r.t. a reference position in an efficient and scalable way. However, the sound field might contain undesired noise. Reducing the noise while preserving the directional distribution of all sound field components is a challenging task. Recently, the present authors proposed a direction-preserving noise reduction method for higher-order Ambisonics (HOA) signals, which, in contrast to e.g. binaural beamforming methods, yields a HOA signal at the output. In this work, we investigate the direction-preserving noise reduction method further and compare it against a beamforming-based method and the matrix multi-channel Wiener filter. Different methods to estimate the power spectral densities which are needed for the noise reduction methods are discussed. Moreover, a method to preserve the reverberation of the desired signal is proposed. In the evaluation, the discussed methods are compared for different speech sources in anechoic and reverberant conditions and different noise types.

For many inverse filtering problems, finite impulse response filters are designed according to least-squares criteria, where time-domain and frequency-domain weights are often applied to achieve optimal results for the considered application. While least-squares-optimal filter coefficients are given by an explicit formula, the computation cost to compute its solution is proportional up to the third power of the number of jointly optimized filter coefficients. A joint optimization of all filter coefficients is necessary whenever a time-domain or a frequency-domain weight is introduced. This imposes limits for filter lengths and numbers of channels in many real-world scenarios. In this contribution, an algorithm is presented that yields time-domain filter coefficients optimized to meet such a weighted least-squares criterion, while performing the most expensive computation steps efficiently in the discrete Fourier transform domain. As a consequence, the demands on computational power and memory are kept on a moderate level, even for large-scale problems. A rigorous mathematical derivation is provided that identifies all approximations used in the algorithm. Additionally, an effective regularization method is proposed that does not depend much on the regularization parameters. Furthermore, the proposed approach is experimentally evaluated considering a sound-zones scenario, which is one of many possible application areas. In that way, the applicability of the proposed approach is verified.

This paper presents a time–frequency masking based online multi-channel speech enhancement approach that uses a convolutional recurrent neural network to estimate the mask. The magnitude and phase components of the short-time Fourier transform coefficients for multiple time frames are provided as an input such that the network is able to discriminate between the directional speech and the noise components based on the spatial characteristics of the individual signals as well as their spectro-temporal structure. The estimation of two different masks, namely, ideal ratio mask (IRM) and ideal binary mask (IBM), along with two different approaches for incorporating the mask to obtain the desired signal are discussed. In the first approach, the mask is directly applied as a real valued gain to a reference microphone signal, whereas in the second approach, the masks are used as an activity indicator for the recursive update of power spectral density (PSD) matrices to be used within a beamformer. The performance of the proposed system with the two different estimated masks utilized within the two different enhancement approaches is evaluated with both simulated as well as measured room impulse responses, where it is shown that the IBM is better suited as an indicator for the PSD updates while direct application of IRM as a real valued gain leads to a better improvement in terms of short term objective intelligibility. Analysis of the performance of the proposed system also demonstrates the robustness of the system to different angular positions of the speech source.

Several techniques exist to estimate the directions of arrival (DOAs) of sound sources captured with a spherical microphone array. The eigenbeam rotational invariance technique (EB-ESPRIT) uses recurrence relations of spherical harmonics to estimate the DOAs. In this letter, we propose a new EB-ESPRIT method which uses three recurrence relations and a joint-diagonalization procedure to estimate the unit-vectors pointing to the source DOAs. We evaluate the angular estimation errors of the proposed method under noisy and reverberant conditions and compare it to existing EB-ESPRIT methods. We find that our proposed method can estimate the source DOAs with higher accuracy compared to the discussed existing EB-ESPRIT methods, and the estimation accuracy does not depend on the source DOAs.

A recursive maximum-likelihood algorithm (RML) is proposed that can be used when both the observations and the hidden data have continuous values and are statistically dependent between different time samples. The algorithm recursively approximates the probability density functions of the observed and hidden data by analytically computing the integrals with respect to the state variables, where the parameters are updated using gradient steps. A full convergence proof is given, based on the ordinary differential equation approach, which shows that the algorithm converges to a local minimum of the Kullback-Leibler divergence between the true and the estimated parametric probability density functions-a result that is useful even for a miss-specified parametric model. Compared to other RML algorithms proposed in the literature, this contribution extends the state-space model and provides a theoretical analysis in a nontrivial statistical model that was not analyzed so far. We further extend the RML analysis to constrained parameter estimation problems. Two examples, including nonlinear state-space models, are given to highlight this contribution.

Many modern communication and smart devices are equipped with several microphones, in addition to one or more loudspeakers. Each microphone not only acquires sounds produced in the near-end room, i.e., desired near-end speech, background noise, and other interferences, but also a far-end signal that is reproduced by the loudspeaker(s). This particular type of acoustic coupling, commonly denoted as acoustic echo, can be reduced in a distortionless manner by employing multi-microphone acoustic echo cancellation (MM-AEC) techniques. However, under noisy conditions, the performance of AEC is limited by the echo-to-noise ratio, and additional echo reduction is needed. Further, to ensure high-quality end-to-end communication in noisy environments, background noise has to be reduced as well. To achieve the latter, multi-microphone speech enhancement techniques, such as beamforming (BF), are often used as they are capable of reducing undesired signal components while causing little distortion to the desired near-end speech. In spite of its high computational cost, the most effective solution to reduce acoustic echoes and background noise is to cascade MM-AEC and BF. In this work, a low-complexity multi-microphone echo controller is introduced, which not only combines low-complexity MM-AEC with BF, but also integrates residual echo reduction into the beamformer design.

In dynamic virtual reality, visual cues and motor actions aid auditory perception. With multimodal integration and auditory adaptation effects, generic head-related transfer functions (HRTFs) may yield no significant disadvantage to individual HRTFs regarding accurate auditory perception. This study compares two individual HRTF sets against a generic HRTF set by way of objective analysis and two subjective experiments. First, auditory-model-based predictions examine the objective deviations in localization cues between the sets. Next, the HRTFs are compared in a static subjective (N=8) localization experiment. Finally, the localization accuracy, timbre, and overall quality of the HRTF sets are evaluated subjectively (N=12) in a six-degrees-of-freedom audio-visual virtual environment. The results show statistically significant objective deviations between the sets, but no perceived localization or overall quality differences in the dynamic virtual reality.

Virtual reality (VR) technology enables and requires new ways of user experience testing in immersive environments. For various aspects of user experience, objective assessment of the cognitive load can be a useful parameter. With eye-tracking becoming a more widespread feature of current VR headsets, pupillometry is an appealing option to unobtrusively measure cognitive load during a VR experience. This paper shows that pupil size measured by an off-the-shelf VR headset with an integrated eye tracker positively correlates with the self-reported cognitive load during a standard n-back task adapted to a VR environment. To overcome the need for steady scene-lighting conditions, we present a method to correct for the light-induced pupil size changes, otherwise masking the cognitive load effects. Our results show that a commercially available VR headset with eye tracking can be used to measure the cognitive load in unpredictable lighting conditions without additional hardware.

Noise reduction in B-format recordings is particularly challenging as it concurrently requires to suppress undesired signals and preserve the spatial properties of the acoustic environment. In particular, wind noise poses an undesirable acoustic condition outdoors. In this work, methods to reduce wind noise while limiting the spatial distortions of the original signal are proposed based on recent works of the present authors. The main contributions are the derivation of the mentioned methods for B-format signals and the usage of the dipole-to-omnidirectional power ratio to control the trade-off between the desired-signal distortion and the noise reduction. The proposed methods are evaluated using B-format wind-noise recordings.

Deep learning (DL) based direction of arrival (DOA) estimation is an active research topic and currently represents the state-of-the-art. Usually, DL-based DOA estimators are trained with recorded data or computationally expensive generated data. Both data types require significant storage and excessive time to, respectively, record or generate. We propose a low complexity online data generation method to train DL models with a phase-based feature input. The data generation method models the phases of the microphone signals in the frequency domain by employing a deterministic model for the direct path and a statistical model for the late reverberation of the room transfer function. By an evaluation using data from measured room impulse responses, we demonstrate that a model trained with the proposed training data generation method performs comparably to models trained with data generated based on the source-image method., Submitted to ICASSP 2021

Publisher Copyright: © 2021 Acoustical Society of America. Copyright: Copyright 2021 Elsevier B.V., All rights reserved. The spatial properties of a noise field can be described by a spatial coherence function. Synthetic multichannel noise signals exhibiting a specific spatial coherence can be generated by properly mixing a set of uncorrelated, possibly non-stationary, signals. The mixing matrix can be obtained by decomposing the spatial coherence matrix. As proposed in a widely used method, the factorization can be performed by means of a Choleski or eigenvalue decomposition. In this work, the limitations of these two methods are discussed and addressed. In particular, specific properties of the mixing matrix are analyzed, namely, the spectral smoothness and the mix balance. The first quantifies the mixing matrix-filters variation across frequency and the second quantifies the number of input signals that contribute to each output signal. Three methods based on the unitary Procrustes solution are proposed to enhance the spectral smoothness, the mix balance, and both properties jointly. A performance evaluation confirms the improvements of the mixing matrix in terms of objective measures. Furthermore, the evaluation results show that the error between the target and the generated coherence is lowered by increasing the spectral smoothness of the mixing matrix.

Time difference of arrival (TDOA) based indoor ultrasound localization systems are prone to multiple disruptions and demand reliable, and resilient position accuracy during operation. In this challenging context, a missing link to evaluate the performance of such systems is a simulation approach to test their robustness in the presence of disruptions. This approach cannot only replace experiments in early phases of development but could also be used to study susceptibility, robustness, response, and recovery in case of disruptions. The paper presents a simulation framework for a TDOA-based indoor ultrasound localization system and ways to introduce different types of disruptions. This framework can be used to test the performance of TDOA-based localization algorithms in the presence of disruptions. Resilience quantification results are presented for representative disruptions. Based on these quantities, it is found that localization with arc-tangent cost function is approximately 30% more resilient than the linear cost function. The simulation approach is shown to apply to resilience engineering and can be used to increase the efficiency and quality of indoor localization methods.

Audio-visual speech enhancement (AVSE) methods use both audio and visual features for the task of speech enhancement and the use of visual features has been shown to be particularly effective in multi-speaker scenarios. In the majority of deep neural network (DNN) based AVSE methods, the audio and visual data are first processed separately using different sub-networks, and then the learned features are fused to uti-lize the information from both modalities. There have been various studies on suitable audio input features and network architectures, however, to the best of our knowledge, there is no published study that has investigated which visual features are best suited for this specific task. In this work, we perform an empirical study of the most commonly used visual features for DNN based AVSE, the pre-processing requirements for each of these features, and investigate their influence on the performance. Our study shows that despite the overall better performance of embedding-based features, their computationally intensive pre-processing makes their use difficult in low resource systems. For such systems, optical flow or raw pixels-based features might be better suited.

The mismatch between the data distributions of training and test data acquired under different recording conditions and using different devices is known to severely impair the performance of acoustic scene classification (ASC) systems. To address this issue, we propose an unsupervised domain adaptation method for ASC based on the projection of spectro-temporal features extracted from both the source and target domain onto the principal subspace spanned by the eigenvectors of the sample covariance matrix of source-domain training data. Using the TUT Urban Acoustic Scenes 2018 Mobile Development dataset we show that the proposed method outperforms state-of-the-art unsupervised domain adaptation techniques when applied jointly with a convolutional ASC model and can also be practically employed as a feature extraction procedure for shallower artificial neural networks.

Feedback delay networks (FDNs) are recursive filters, which are widely used for artificial reverberation and decorrelation. One central challenge in the design of FDNs is the generation of sufficient echo density in the impulse response without compromising the computational efficiency. In a previous contribution, we have demonstrated that the echo density of an FDN can be increased by introducing so-called delay feedback matrices where each matrix entry is a scalar gain and a delay. In this contribution, we generalize the feedback matrix to arbitrary lossless filter feedback matrices (FFMs). As a special case, we propose the velvet feedback matrix, which can create dense impulse responses at a minimal computational cost. Further, FFMs can be used to emulate the scattering effects of non-specular reflections. We demonstrate the effectiveness of FFMs in terms of echo density and modal distribution.

A deep neural network (DNN) based approach for estimating the speed of airflows using closely-spaced microphones is proposed. The spatial characteristics of wind noise measured with a smallaperture array are exploited, i.e., the low-frequency spatial coherence of wind noise signals is used as an input feature. The output is an estimate of the wind speed averaged over a specific time interval. The DNN is trained using synthetic wind noise, which overcomes the time-consuming data collection and allows to isolate wind noise from different acoustic sources. The dataset used for testing comprises wind noise measured outdoors with a circular linear array and a ground truth obtained using an ultrasonic anemometer. The obtained model is applied to generated and measured wind noise. The performance of the proposed method is assessed across a wide range of wind speeds and directions, using different time resolutions.

Speech signals captured by a microphone mounted to a smart soundbar or speaker are inherently contaminated by echos. Modern smart devices are usually characterized by low computational capabilities and low memory resources; in these cases, a low-complexity acoustic echo canceller (AEC) may be preferred even though a tolerable degradation in the cancellation occurs. In principle, devices with multiple loudspeakers need an individual AEC for each loudspeaker because the transfer function (TF) from each loudspeaker to the microphone must be estimated. In this paper, we present an normalized least mean square (NLMS) algorithm for a multi-loudspeaker case using relative loudspeaker transfer functions (RLTFs). In each iteration, the RLTFs between each loudspeaker and the reference loudspeaker are estimated first, and then the primary TF between the reference loudspeaker and the microphone. Assuming loudspeakers that are close to each other, the RLTFs can be estimated using fewer coefficients w.r.t. the primary TF, yielding a reduction of 3:4 in computational complexity and 1:2 in memory usage. The algorithm is evaluated using both simulated and real room impulse responses (RIRs) of two loudspeakers with a reverberation time set to 0.3 s and several distances between the loudspeakers.

Multichannel linear prediction-based dereverberation in the short-time Fourier transform (STFT) domain has been shown to be highly effective. Using this framework, the desired dereverberated multichannel signal is obtained by filtering the noise-free reverberant signals using the estimated multichannel autoregressive (MAR) coefficients. To use such methods in the presence of noise, especially in the case of online processing, remains a challenging problem. Existing sequential enhancement structures, which first remove the noise and then estimate the MAR coefficients, suffer from a causality problem as both the optimal noise reduction and dereverberation stages depend on the current output of each other. To address this problem, an algorithm that consists of two alternating Kalman filters to estimate the noise-free reverberant signals and the (MAR) coefficients is proposed. The causality of the estimation procedure is important when dealing with time-variant acoustic scenarios, where the MAR coefficients are time-varying. The proposed method is evaluated using simulated and measured acoustic impulse responses and is compared to a method based on the same signal model. In addition, a method to control the reverberation reduction and noise reduction independently is derived.

Room geometry inference is concerned with the localization of reflective boundaries in an enclosed space. This paper outlines a method for inferring room geometry based on the positions of loudspeakers and real or image microphones, which are computed using sets of times of arrival (TOAs) obtained from room impulse responses (RIRs). These RIRs describe the acoustic propagation between the loudspeakers in an array and a single microphone. First, peaks corresponding to TOAs in these RIRs are detected and labeled using an automated method. Second, the labeled TOA sets are used to estimate the real and image microphone positions, with knowledge of the loudspeaker array geometry. Third, using all these positions, the positions of reflection points on the available reflectors in the room are determined. The reflection points determine the reflectors' locations and orientations. This approach is largely automated and usable in real-world scenarios.

Sparsity-based blind source separation (BSS) algorithms in the short time–frequency (TF) domain have received a lot of attention due to their versatility and noise reduction capabilities. In most of these algorithms, the estimation of the BSS filters relies on the accurate association of each time–frequency bin to the dominant source at that bin. The TF bin associations are then used to estimate the statistics of the source signals, and BSS is achieved by optimal spatial filters computed using the estimated statistics. The main objective of this paper is to apply such a framework to scenarios with an unknown number of moving sources. While state-of-the-art approaches employ online clustering algorithms to solve the problem for moving sources, we propose an approximate Bayesian tracker and perform the association of each TF bin to the dominant source using the tracker's measurement-to-source association probabilities. Therefore, the choice of the underlying narrowband models and measurements for the tracker as well as the resulting tracking algorithm constitute the main contributions of this paper. The TF bin associations obtained from the tracker are then used to estimate the statistics of the source signals. The performance of the resulting BSS filters is compared to the performance of state-of-the-art sparsity-based and independent vector analysis-based BSS algorithms. Our proposed approach targets scenarios with at least two spatially separated microphone arrays, with known microphone positions and relative orientations. The framework also allows for efficient management of a time-varying number of sources.

A Bayesian approach to spatial filtering is presented, which is robust to uncertain or erroneous direction-of-arrival (DOA) information. The proposed framework aims to capture multiple sound sources at each time-frequency instant with an arbitrary direction-dependent gain, while attenuating diffuse sound and noise. For robustness, the DOA corresponding to each sound source is assumed to be a discrete random variable with a prior defined on a discrete set of candidate DOAs over the whole DOA space. With this assumption, the desired spatial filter is given as a weighted sum of spatial filters corresponding to a specific combination of probable DOA values, where the weights are given by the joint posterior probabilities of the combination of DOA values. Assuming the whole DOA space as the support for each random variable results in redundant computations and contributes to a high computational cost. To alleviate this problem, a narrowband DOA estimate-based posterior probability approximation method is proposed, which isolates regions in the DOA space with high probability of containing the actual source DOAs to compute time-adaptive supports for each random variable. Through experimental analysis, we demonstrate the robustness of the proposed framework against DOA estimation errors. Experimental evaluation with simulated and measured room impulse responses, in terms of objective performance measures, demonstrates the effectiveness of the framework to perform spatial filtering in noisy and reverberant acoustic environments.

The reverberation power spectral density (PSD) is often required for dereverberation and noise reduction algorithms. In this work, we compare two maximum likelihood (ML) estimators of the reverberation PSD in a noisy environment. In the first estimator, the direct path is first blocked. Then, the ML criterion for estimating the reverberation PSD is stated according to the probability density function of the blocking matrix (BM) outputs. In the second estimator, the speech component is not blocked. Instead, the ML criterion for estimating the speech and reverberation PSD is stated according to the probability density function of the microphone signals. To compare the expected mean square error (MSE) between the two ML estimators of the reverberation PSD, the Cramer–Rao Bounds (CRBs) for the two ML estimators are derived. We show that the CRB for the joint reverberation and speech PSD estimator is lower than the CRB for estimating the reverberation PSD from the BM outputs. Experimental results show that the MSE of the two estimators indeed obeys the CRB curves. Experimental results of multimicrophone dereverberation and noise reduction algorithm show the benefits of using the ML estimators in comparison with another baseline estimators.

The performance of machine learning algorithms is known to be negatively affected by possible mismatches between training (source) and test (target) data distributions. In fact, this problem emerges whenever an acoustic scene classification system which has been trained on data recorded by a given device is applied to samples acquired under different acoustic conditions or captured by mismatched recording devices. To address this issue, we propose an unsupervised domain adaptation method that consists of aligning the first- and second-order sample statistics of each frequency band of target-domain acoustic scenes to the ones of the source-domain training dataset. This model-agnostic approach is devised to adapt audio samples from unseen devices before they are fed to a pre-trained classifier, thus avoiding any further learning phase. Using the DCASE 2018 Task 1-B development dataset, we show that the proposed method outperforms the state-of-the-art unsupervised methods found in the literature in terms of both source- and target-domain classification accuracy., Comment: 5 pages, 1 figure, 3 tables, submitted to EUSIPCO 2020

Recorded signals can be clipped in case the sound pressure or analog signal amplification is too large. Clipping is a non-linear distortion, which limits the maximal magnitude modulation of the signal and changes the energy distribution in frequency domain and hence degrades the quality of the recording. Consequently, for declipping, some frequencies have to be amplified, and others attenuated. We propose a declipping method by using the recently proposed deep filtering technique which is capable of extracting and reconstructing a desired signal from a degraded input. Deep filtering operates in the short-time Fourier transform (STFT) domain estimating a complex multidimensional filter for each desired STFT bin. The filters are applied to defined areas of the clipped STFT to obtain for each filter a single complex STFT bin estimation of the declipped STFT. The filter estimation, thereby, is performed via a deep neural network trained with simulated data using soft- or hard-clipping. The loss function minimizes the reconstruction mean-squared error between the non-clipped and the declipped STFTs. We evaluate our approach using simulated data degraded by hard- and soft-clipping and conducted a pairwise comparison listening test with measured signals comparing our approach to one commercial and one open-source declipping method. Our approach outperformed the baselines for declipping speech signals for measured data for strong and medium clipping.

Feedback delay networks (FDNs) belong to a general class of recursive filters which are widely used in artificial reverberation and decorrelation applications. One central challenge in the design of FDNs is the generation of sufficient echo density in the impulse response without compromising the computational efficiency. In a previous contribution, we have demonstrated that the echo density of an FDN grows polynomially over time, and that the growth depends on the number and lengths of the delays. In this work, we introduce so-called delay feedback matrices (DFMs) where each matrix entry is a scalar gain and a delay. While the computational complexity of DFMs is similar to a scalar-only feedback matrix, we show that the echo density grows significantly faster over time, however, at the cost of non-uniform modal decays.

Eigenbeam ESPRIT (EB-ESPRIT) is a subspace based method to estimate directions-of-arrival (DOAs) of sound sources from a spherical microphone array recording in the spherical harmonics domain (SHD). In recent works, nonsingular EB-ESPRIT methods have been proposed which can estimate the source DOA-vectors without ambiguities. In another recent publication, a subspace based pseudointensity-vector method (SS-PIV) has been proposed for DOA estimation. In this work, we derive the mathematical relation between the DOA-vector EB-ESPRIT and the SS-PIV method. We show that the SS-PIV can be seen as a special case of the DOA-vector EB-ESPRIT. Using this relation, we propose a novel DOA-estimator denoted as extended pseudointensity-vector (PIV). In the evaluation, we compare the DOA-vector EB-ESPRIT and the extended PIV with the SS-PIV and PIV under noisy and reverberant conditions.

Narrowband direction-of-arrival (DOA) estimates are commonly used for source localization, parametric spatial audio coding, and directional filtering. As previously shown, a linear least squares direction estimate can be obtained by minimizing the difference of expected and observed inter-microphone phase differences. In this work, it is shown that phase wrapping induces severe estimation errors especially at frequencies just below spatial aliasing frequencies and in low signal-to-noise ratios. A cost function to mitigate the influence of phase wrapping errors on the DOA estimation is proposed. Even though the proposed cost function is nonlinear, it is shown that one iteration of a gradient descent method with proper initialization provides a large improvement when compared to the linear least squares solution.

We present a spatial matrix filtering framework for noise reduction in the spherical harmonics (ambisonics) domain (SHD), which outputs an SHD signal vector rather than one signal as commonly provided by beamforming approaches. We discuss two spatial matrix filtering methods: A multi-beamformer method using known propagation vectors of the desired signal components and a method preserving the directional information in an optimal way. Parametric multi-channel Wiener filter solutions for both methods are discussed and a performance evaluation is conducted. It is shown that the direction preserving method preserves the spatial distribution of the desired sounds and residual noise at the cost of less noise reduction and higher signal distortion when compared to the multi-beamformer approach. Moreover, no spatial parameters have to be estimated.

Outdoor recordings of speech are often corrupted by wind noise, which is difficult to reduce due to its high non-stationarity. In this work, a multi-channel wind noise reduction method is presented, based on a joint estimation of the speech and wind noise power spectral densities. In contrast to existing methods that assume uncorrelated wind noise, the estimation phase is performed exploiting the spatial characteristics of wind noise measured by closely-spaced microphones. Here the characteristics are approximated by a fluid-dynamics model, termed the Corcos model. An additional contribution is the employment of a frequency dependent trade-off parameter in the reduction phase, which depends on the ratio of the difference-signal power to the sum-signal power of a sub-set of two microphones. In particular, the trade-off parameter of the parametric multi-channel Wiener filter is used to control the trade-off between noise reduction and speech distortion. The proposed method can be used also to reduce spatially uncorrelated wind noise. An evaluation in terms of speech quality and signal-to-noise ratio improvements is carried out to compare the proposed method to an existing multichannel wind noise reduction method, using recorded and simulated wind noise samples.

Flexible spatial sound capturing and reproduction can be achieved with multiple microphones by using linear spatial filtering or non-linear parametric processing. The non-linear approaches usually provide a superior spatial resolution compared to the linear approaches but can result in artifacts due to violations of the sound field model. In this paper, we combine both approaches to achieve a high robustness against model violations and a high spatial resolution. We assume linear spatial filters that approximate the spatial responses of the desired output format and compensate remaining deviations with an optimal post filter. The post filter is computed such that the proposed approach behaves like a linear system when the spatial filters achieve the desired spatial response, and scales towards a non-linear system otherwise. Experimental results show that the proposed approach can significantly reduce distortions of existing parametric processing schemes especially when a sufficiently high number of microphones is available.

International audience; Estimating the maximum number of concurrent speakers from single-channel mixtures is a challenging problem and an essential first step to address various audio-based tasks such as blind source separation, speaker diarization, and audio surveillance. We propose a unifying probabilistic paradigm, where deep neural network architectures are used to infer output posterior distributions. These probabilities are in turn processed to yield discrete point estimates. Designing such architectures often involves two important and complementary aspects that we investigate and discuss. First, we study how recent advances in deep architectures may be exploited for the task of speaker count estimation. In particular, we show that convolutional recurrent neural networks outperform recurrent networks used in a previous study when adequate input features are used. Even for short segments of speech mixtures, we can estimate up to five speakers, with a significantly lower error than other methods. Second, through comprehensive evaluation, we compare the best-performing method to several baselines, as well as the influence of gain variations, different data sets, and reverberation. The output of our proposed method is compared to human performance. Finally, we give insights into the strategy used by our proposed method.

In speech communication systems, the microphone signals are degraded by reverberation and ambient noise. The reverberant speech can be separated into two components, namely, an early speech component that consists of the direct path and some early reflections and a late reverberant component that consists of all late reflections. In this paper, a novel algorithm to simultaneously suppress early reflections, late reverberation, and ambient noise is presented. The expectation-maximization (EM) algorithm is used to estimate the signals and spatial parameters of the early speech component and the late reverberation components. As a result, a spatially filtered version of the early speech component is estimated in the E-step. The power spectral density (PSD) of the anechoic speech, the relative early transfer functions, and the PSD matrix of the late reverberation are estimated in the M-step of the EM algorithm. The algorithm is evaluated using real room impulse response recorded in our acoustic lab with a reverberation time set to 0.36 s and 0.61 s and several signal-to-noise ratio levels. It is shown that significant improvement is obtained and that the proposed algorithm outperforms baseline single-channel and multichannel dereverberation algorithms, as well as a state-of-the-art multichannel dereverberation algorithm.