Search

We study the problem of learning robust acoustic models in adverse environments, characterized by a significant mismatch between training and test conditions. This problem is of paramount importance for the deployment of speech recognition systems that need to perform well in unseen environments. First, we characterize data augmentation theoretically as an instance of vicinal risk minimization, which aims at improving risk estimates during training by replacing the delta functions that define the empirical density over the input space with an approximation of the marginal population density in the vicinity of the training samples. More specifically, we assume that local neighborhoods centered at training samples can be approximated using a mixture of Gaussians, and demonstrate theoretically that this can incorporate robust inductive bias into the learning process. We then specify the individual mixture components implicitly via data augmentation schemes, designed to address common sources of spurious correlations in acoustic models. To avoid potential confounding effects on robustness due to information loss, which has been associated with standard feature extraction techniques (e.g., FBANK and MFCC features), we focus on the waveform-based setting. Our empirical results show that the approach can generalize to unseen noise conditions, with 150% relative improvement in out-of-distribution generalization compared to training using the standard risk minimization principle. Moreover, the results demonstrate competitive performance relative to models learned using a training sample designed to match the acoustic conditions characteristic of test utterances.

In this paper, we investigate multi-stream acoustic modelling using the raw real and imaginary parts of the Fourier transform of speech signals. Using the raw magnitude spectrum, or features derived from it, as a proxy for the real and imaginary parts leads to irreversible information loss and suboptimal information fusion. We discuss and quantify the importance of such information in terms of speech quality and intelligibility. In the proposed framework, the real and imaginary parts are treated as two streams of information, pre-processed via separate convolutional networks, and then combined at an optimal level of abstraction, followed by further post-processing via recurrent and fully-connected layers. The optimal level of information fusion in various architectures, training dynamics in terms of cross-entropy loss, frame classification accuracy and WER as well as the shape and properties of the filters learned in the first convolutional layer of single- and multi-stream models are analysed. We investigated the effectiveness of the proposed systems in various tasks: TIMIT/NTIMIT (phone recognition), Aurora-4 (noise robustness), WSJ (read speech), AMI (meeting) and TORGO (dysarthric speech). Across all tasks we achieved competitive performance: in Aurora-4, down to 4.6% WER on average, in WSJ down to 4.6% and 6.2% WERs for Eval-92 and Eval-93, for Dev/Eval sets of the AMI-IHM down to 23.3%/23.8% WERs and in the AMI-SDM down to 43.7%/47.6% WERs have been achieved. In TORGO, for dysarthric and typical speech we achieved down to 31.7% and 10.2% WERs, respectively.

We investigate the potential of stochastic neural networks for learning effective waveform-based acoustic models. The waveform-based setting, inherent to fully end-to-end speech recognition systems, is motivated by several comparative studies of automatic and human speech recognition that associate standard non-adaptive feature extraction techniques with information loss which can adversely affect robustness. Stochastic neural networks, on the other hand, are a class of models capable of incorporating rich regularization mechanisms into the learning process. We consider a deep convolutional neural network that first decomposes speech into frequency sub-bands via an adaptive parametric convolutional block where filters are specified by cosine modulations of compactly supported windows. The network then employs standard non-parametric 1D convolutions to extract relevant spectro-temporal patterns while gradually compressing the structured high dimensional representation generated by the parametric block. We rely on a probabilistic parametrization of the proposed neural architecture and learn the model using stochastic variational inference. This requires evaluation of an analytically intractable integral defining the Kullback-Leibler divergence term responsible for regularization, for which we propose an effective approximation based on the Gauss-Hermite quadrature. Our empirical results demonstrate a superior performance of the proposed approach over comparable waveform-based baselines and indicate that it could lead to robustness. Moreover, the approach outperforms a recently proposed deep convolutional neural network for learning of robust acoustic models with standard FBANK features.

This article studies the effects of inter-channel time and level differences in stereophonic reproduction on perceived localization uncertainty, which is defined as how difficult it is for a listener to tell where a sound source is located. Towards this end, a computational model of localization uncertainty is proposed first. The model calculates inter-aural time and level difference cues, and compares them to those associated to free-field point-like sources. The comparison is carried out using a particular distance functional that replicates the increased uncertainty observed experimentally with inconsistent inter-aural time and level difference cues. The model is validated by formal listening tests, achieving a Pearson correlation of 0.99. The model is then used to predict localization uncertainty for stereophonic setups and a listener in central and off-central positions. Results show that amplitude methods achieve a slightly lower localization uncertainty for a listener positioned exactly in the center of the sweet spot. As soon as the listener moves away from that position, the situation reverses, with time-amplitude methods achieving a lower localization uncertainty.

Algorithms for learning a dictionary to sparsely represent a given dataset typically alternate between sparse coding and dictionary update stages. Methods for dictionary update aim to minimise expansion error by updating dictionary vectors and expansion coefficients given patterns of non-zero coefficients obtained in the sparse coding stage. We propose a block total least squares (BLOTLESS) algorithm for dictionary update. BLOTLESS updates a block of dictionary elements and the corresponding sparse coefficients simultaneously. In the error free case, three necessary conditions for exact recovery are identified. Lower bounds on the number of training data are established so that the necessary conditions hold with high probability. Numerical simulations show that the bounds approximate well the number of training data needed for exact dictionary recovery. Numerical experiments further demonstrate several benefits of dictionary learning with BLOTLESS update compared with state-of-the-art algorithms especially when the amount of training data is small.

Developments in immersive audio technologies have been evolving in two directions: physically motivated systems and perceptually motivated systems. Physically motivated techniques aim to reproduce a physically accurate approximation of desired sound fields by employing a very high equipment load and sophisticated, computationally intensive algorithms. Perceptually motivated techniques, however, aim to render only the perceptually relevant aspects of the sound scene by means of modest computational and equipment load. This article presents an overview of perceptually motivated techniques, with a focus on multichannel audio recording and reproduction, audio source and reflection culling, and artificial reverberators.

It is well known that samplers are linear time varying systems, so in general, the commutativity of samplers does not hold. There are some existing results on the commutativity of conventional decimators and expanders, block samplers with the same integer block lengths but different integer sampling ratios, and block samplers with different integer block lengths and integer sampling ratios. This paper extends the existing results to a necessary and sufficient condition for the commutativity of block decimators and expanders with arbitrary rational sampling ratios and block lengths.

Multichannel dereverberation amounts to the inversion of a multiple-input/multiple-output linear time-invariant system. In this paper, necessary and sufficient conditions for perfect dereverberation using stable and finite impulse response (FIR) filters are established. It is then shown that the inverse system given by the pseudoinverse of the original transfer function matrix exhibits a noise reduction property. A necessary and sufficient condition under which this pseudoinverse system is FIR is also given. Further, an FIR approximation to the pseudoinverse system is considered and the effects of the length of this approximation on the dereverberation accuracy are investigated. Finally, an analytical and numerical assessment of the dependence of the dereverberation accuracy on the accuracy of the acquisition of room impulse responses is provided.

A novel systematic approach to the design of directivity patterns of higher order differential microphones is proposed. The directivity patterns are obtained by optimizing a cost function which is a convex combination of a front-back energy ratio and uniformity within a frontal sector of interest. Most of the standard directivity patterns - omnidirectional, cardioid, subcardioid, hypercardioid, supercardioid - are particular solutions of this optimization problem with specific values of two free parameters: the angular width of the frontal sector and the convex combination factor. More general solutions of practical use are obtained by varying these two parameters. Many of these optimal directivity patterns are trigonometric polynomials with complex roots. A new differential array structure that enables the implementation of general higher order directivity patterns, with complex or real roots, is then proposed. The effectiveness of the proposed design framework and the implementation structure are illustrated by design examples, simulations, and measurements.

This paper proposes methods for combining cepstral and acoustic waveform representations for a front-end of support vector machine (SVM)-based speech recognition systems that are robust to additive noise. The key issue of kernel design and noise adaptation for the acoustic waveform representation is addressed first. Cepstral and acoustic waveform representations are then compared on a phoneme classification task. Experiments show that the cepstral features achieve very good performance in low noise conditions, but suffer severe performance degradation already at moderate noise levels. Classification in the acoustic waveform domain, on the other hand, is less accurate in low noise but exhibits a more robust behavior in high noise conditions. A combination of the cepstral and acoustic waveform representations achieves better classification performance than either of the individual representations over the entire range of noise levels tested, down to - 18-dB SNR.

Digital waveguide mesh (DWM) models are time-domain numerical methods providing computationally simple solutions for wave propagation problems. They have been used in various acoustical modeling and audio synthesis applications including synthesis of musical instrument sounds and speech, and modeling of room acoustics. A successful model of room acoustics should be able to account for source and receiver directivity. Methods for the simulation of directional sources in DWM models were previously proposed. This paper presents a method for the simulation of directional microphones in DWM-based models of room acoustics. The method is based on the directional weighting of the microphone response according to the instantaneous direction of incidence at a given point. The direction of incidence is obtained from instantaneous intensity that is calculated from local pressure values in the DWM model. The calculation of instantaneous intensity in DWM meshes and the directional accuracies of different mesh topologies are discussed. An intensity-based formulation for the response of a directional microphone is given. Simulation results for an actual microphone with frequency-dependent, non-ideal directivity function are presented.

In emerging audio technology applications, there is a need for decompositions of audio signals into oversampled subband components with time-frequency resolution which mimics that of the cochlear filter bank and with high aliasing attenuation in each of the subbands independently, rather than aliasing cancellation properties. We present a design of nearly perfect reconstruction nonuniform oversampled filter banks which implement signal decompositions of this kind.

An acoustic reverberator consisting of a network of delay lines connected via scattering junctions is proposed. All parameters of the reverberator are derived from physical properties of the enclosure it simulates. It allows for simulation of unequal and frequency-dependent wall absorption, as well as directional sources and microphones. The reverberator renders the first-order reflections exactly, while making progressively coarser approximations of higher-order reflections. The rate of energy decay is close to that obtained with the image method (IM) and consistent with the predictions of Sabine and Eyring equations. The time evolution of the normalized echo density, which was previously shown to be correlated with the perceived texture of reverberation, is also close to that of the IM. However, its computational complexity is one to two orders of magnitude lower, comparable to the computational complexity of a feedback delay network and its memory requirements are negligible.

Weyl-Heisenberg frames are a principal tool of short-time Fourier analysis. We present a comprehensive study of Weyl-Heisenberg frames in l/sup 2/(Z), with a focus on frames that are tight. A number of properties of these frames are derived. A complete parameterization of finite-length windows for tight Weyl-Heisenberg frames in l/sup 2/(Z) is described. Design of windows for tight Weyl-Heisenberg frames requires optimization of their frequency characteristics under nonlinear constraints. We propose an efficient design method based on expansions with respect to prolate spheroidal sequences. The advantages of the proposed method over standard optimization procedures include a reduction in computational complexity and the ability to provide long windows that can be specified concisely using only a few parameters; these advantages become increasingly pronounced as the frame redundancy increases. The resilience of overcomplete Weyl-Heisenberg expansions to additive noise and quantization is also studied. We show that manifestations of degradation due to uncorrelated zero-mean additive noise are inversely proportional to the expansion redundancy, whereas the quantization error is for a given quantization step inversely proportional to the square of the expansion redundancy.

Tight Weyl-Heisenberg frames in l/sup 2/(Z) are the tool for short-time Fourier analysis in discrete time. They are closely related to paraunitary modulated filter banks and are studied here using techniques of the filter bank theory. Good resolution of short-time Fourier analysis in the joint time-frequency plane is not attainable unless some redundancy is introduced. That is the reason for considering overcomplete Weyl-Heisenberg expansions. The main result of this correspondence is a complete parameterization of finite length tight Weyl-Heisenberg frames in l/sup 2/(Z) with arbitrary rational oversampling ratios. This parameterization follows from a factorization of polyphase matrices of paraunitary modulated filter banks, which is introduced first.

Fast recursive algorithms for the computation of the discrete cosine and sine transforms are developed. An N-point discrete cosine transform (DCT) or discrete sine transform (DST) can be computed from two N/2-point DCTs or DSTs. Compared to the existing algorithms the algorithms have less multiplications by two, and add operations are better positioned, giving rise to faster computation and easier VLSI implementation. >

The paper studies wavelet transform extrema and zero-crossings representations within the framework of convex representations in /spl Lscr/(Z). Wavelet zero-crossings representation of two-dimensional signals is introduced as a convex multiscale edge representation as well. One appealing property of convex representations is that the reconstruction problem can be solved, at least theoretically, using the method of alternating projections onto convex sets. It turns out that in the case of the wavelet extrema and wavelet zero-crossings representations this method yields simple and practical reconstruction algorithms. Nonsubsampled filter banks that implement the wavelet transform for the two representations are also studied in the paper. Relevant classes of nonsubsampled perfect reconstruction FIR filter banks are characterized. This characterization gives a broad class of wavelets for the representations which are derived from those of the filter banks which satisfy a regularity condition. >

Perfect reconstruction oversampled filter banks are equivalent to a particular class of frames in l/sup 2/(Z). These frames are the subject of this paper. First, the necessary and sufficient conditions of a filter bank for implementing a frame or a tight frame expansion are established, as well as a necessary and sufficient condition for perfect reconstruction using FIR filters after an FIR analysis. Complete parameterizations of oversampled filter banks satisfying these conditions are given. Further, we study the condition under which the frame dual to the frame associated with an FIR filter bank is also FIR and give a parameterization of a class of filter banks satisfying this property. Then, we focus on non-subsampled filter banks. Non-subsampled filter banks implement transforms similar to continuous-time transforms and allow for very flexible design. We investigate the relations of these filter banks to continuous-time filtering and illustrate the design flexibility by giving a procedure for designing maximally flat two-channel filter banks that yield highly regular wavelets with a given number of vanishing moments.