Search

This paper presents a novel multi-channel speech enhancement approach, FoVNet, that enables highly efficient speech enhancement within a configurable field of view (FoV) of a smart-glasses user without needing specific target-talker(s) directions. It advances over prior works by enhancing all speakers within any given FoV, with a hybrid signal processing and deep learning approach designed with high computational efficiency. The neural network component is designed with ultra-low computation (about 50 MMACS). A multi-channel Wiener filter and a post-processing module are further used to improve perceptual quality. We evaluate our algorithm with a microphone array on smart glasses, providing a configurable, efficient solution for augmented hearing on energy-constrained devices. FoVNet excels in both computational efficiency and speech quality across multiple scenarios, making it a promising solution for smart glasses applications., Comment: Accepted by INTERSPEECH2024

Adding visual cues to audio-based speech separation can improve separation performance. This paper introduces AV-CrossNet, an audiovisual (AV) system for speech enhancement, target speaker extraction, and multi-talker speaker separation. AV-CrossNet is extended from the CrossNet architecture, which is a recently proposed network that performs complex spectral mapping for speech separation by leveraging global attention and positional encoding. To effectively utilize visual cues, the proposed system incorporates pre-extracted visual embeddings and employs a visual encoder comprising temporal convolutional layers. Audio and visual features are fused in an early fusion layer before feeding to AV-CrossNet blocks. We evaluate AV-CrossNet on multiple datasets, including LRS, VoxCeleb, and COG-MHEAR challenge. Evaluation results demonstrate that AV-CrossNet advances the state-of-the-art performance in all audiovisual tasks, even on untrained and mismatched datasets., Comment: 10 pages, 4 Figures, and 4 Tables

Self-supervised learned models have been found to be very effective for certain speech tasks such as automatic speech recognition, speaker identification, keyword spotting and others. While the features are undeniably useful in speech recognition and associated tasks, their utility in speech enhancement systems is yet to be firmly established, and perhaps not properly understood. In this paper, we investigate the uses of SSL representations for single-channel speech enhancement in challenging conditions and find that they add very little value for the enhancement task. Our constraints are designed around on-device real-time speech enhancement -- model is causal, the compute footprint is small. Additionally, we focus on low SNR conditions where such models struggle to provide good enhancement. In order to systematically examine how SSL representations impact performance of such enhancement models, we propose a variety of techniques to utilize these embeddings which include different forms of knowledge-distillation and pre-training., Comment: 8 pages; Shorter form accepted in ICASSP 2024

We present precise photometric estimates of stellar parameters, including effective temperature, metallicity, luminosity classification, distance, and stellar age, for nearly 26 million stars using the methodology developed in the first paper of this series, based on the stellar colors from the Stellar Abundances and Galactic Evolution Survey (SAGES) DR1 and Gaia EDR3. The optimal design of stellar-parameter sensitive $uv$ filters by SAGES has enabled us to determine photometric-metallicity estimates down to $-3.5$, similar to our previous results with the SkyMapper Southern Survey (SMSS), yielding a large sample of over five million metal-poor (MP; [Fe/H]$\le -1.0$) stars and nearly one million very metal-poor (VMP; [Fe/H]$\le -2.0$) stars. The typical precision is around $0.1$ dex for both dwarf and giant stars with [Fe/H]$>-1.0$, and 0.15-0.25/0.3-0.4 dex for dwarf/giant stars with [Fe/H]$<-1.0$. Using the precise parallax measurements and stellar colors from Gaia, effective temperature, luminosity classification, distance and stellar age are further derived for our sample stars. This huge data set in the Northern sky from SAGES, together with similar data in the Southern sky from SMSS, will greatly advance our understanding of the Milky Way, in particular its formation and evolution., Comment: 14 pages, 14 figures, 3 tables, accepted by ApJ. arXiv admin note: text overlap with arXiv:2104.14154

Measuring quality and intelligibility of a speech signal is usually a critical step in development of speech processing systems. To enable this, a variety of metrics to measure quality and intelligibility under different assumptions have been developed. Through this paper, we introduce tools and a set of models to estimate such known metrics using deep neural networks. These models are made available in the well-established TorchAudio library, the core audio and speech processing library within the PyTorch deep learning framework. We refer to it as TorchAudio-Squim, TorchAudio-Speech QUality and Intelligibility Measures. More specifically, in the current version of TorchAudio-squim, we establish and release models for estimating PESQ, STOI and SI-SDR among objective metrics and MOS among subjective metrics. We develop a novel approach for objective metric estimation and use a recently developed approach for subjective metric estimation. These models operate in a ``reference-less" manner, that is they do not require the corresponding clean speech as reference for speech assessment. Given the unavailability of clean speech and the effortful process of subjective evaluation in real-world situations, such easy-to-use tools would greatly benefit speech processing research and development., Comment: ICASSP 2023

We report the discovery of a new stream (dubbed as Yangtze) detected in $Gaia$ Data Release 3. The stream is at a heliocentric distance of $\sim$ 9.12 kpc and spans nearly 27$\deg$ by 1.9$\deg$ on sky. The colour-magnitude diagram of Yangtze indicates a stellar population of Age $\sim$ 11 Gyr and [M/H] $\sim$ -0.7 dex. It has a number density of about 5.5 stars degree$^{-2}$ along with a surface brightness of $\Sigma_G \simeq$ 34.9 mag arcsec$^{-2}$. The dynamics and metallicity estimate suggest that Yangtze may be closely related to Palomar 1 and the Anticenter stream., Comment: 8 pages, 4 figures, accepted by ApJL. arXiv admin note: substantial text overlap with arXiv:2208.05176

Despite multiple efforts made towards adopting complex-valued deep neural networks (DNNs), it remains an open question whether complex-valued DNNs are generally more effective than real-valued DNNs for monaural speech enhancement. This work is devoted to presenting a critical assessment by systematically examining complex-valued DNNs against their real-valued counterparts. Specifically, we investigate complex-valued DNN atomic units, including linear layers, convolutional layers, long short-term memory (LSTM), and gated linear units. By comparing complex- and real-valued versions of fundamental building blocks in the recently developed gated convolutional recurrent network (GCRN), we show how different mechanisms for basic blocks affect the performance. We also find that the use of complex-valued operations hinders the model capacity when the model size is small. In addition, we examine two recent complex-valued DNNs, i.e. deep complex convolutional recurrent network (DCCRN) and deep complex U-Net (DCUNET). Evaluation results show that both DNNs produce identical performance to their real-valued counterparts while requiring much more computation. Based on these comprehensive comparisons, we conclude that complex-valued DNNs do not provide a performance gain over their real-valued counterparts for monaural speech enhancement, and thus are less desirable due to their higher computational costs.

Most speech enhancement (SE) models learn a point estimate and do not make use of uncertainty estimation in the learning process. In this paper, we show that modeling heteroscedastic uncertainty by minimizing a multivariate Gaussian negative log-likelihood (NLL) improves SE performance at no extra cost. During training, our approach augments a model learning complex spectral mapping with a temporary submodel to predict the covariance of the enhancement error at each time-frequency bin. Due to unrestricted heteroscedastic uncertainty, the covariance introduces an undersampling effect, detrimental to SE performance. To mitigate undersampling, our approach inflates the uncertainty lower bound and weights each loss component with their uncertainty, effectively compensating severely undersampled components with more penalties. Our multivariate setting reveals common covariance assumptions such as scalar and diagonal matrices. By weakening these assumptions, we show that the NLL achieves superior performance compared to popular loss functions including the mean squared error (MSE), mean absolute error (MAE), and scale-invariant signal-to-distortion ratio (SI-SDR)., Comment: 5 pages. Accepted at ICASSP 2023

Permutation-invariant training (PIT) is a dominant approach for addressing the permutation ambiguity problem in talker-independent speaker separation. Leveraging spatial information afforded by microphone arrays, we propose a new training approach to resolving permutation ambiguities for multi-channel speaker separation. The proposed approach, named location-based training (LBT), assigns speakers on the basis of their spatial locations. This training strategy is easy to apply, and organizes speakers according to their positions in physical space. Specifically, this study investigates azimuth angles and source distances for location-based training. Evaluation results on separating two- and three-speaker mixtures show that azimuth-based training consistently outperforms PIT, and distance-based training further improves the separation performance when speaker azimuths are close. Furthermore, we dynamically select azimuth-based or distance-based training by estimating the azimuths of separated speakers, which further improves separation performance. LBT has a linear training complexity with respect to the number of speakers, as opposed to the factorial complexity of PIT. We further demonstrate the effectiveness of LBT for the separation of four and five concurrent speakers., Comment: submitted to ICASSP 22

Most existing deep learning based binaural speaker separation systems focus on producing a monaural estimate for each of the target speakers, and thus do not preserve the interaural cues, which are crucial for human listeners to perform sound localization and lateralization. In this study, we address talker-independent binaural speaker separation with interaural cues preserved in the estimated binaural signals. Specifically, we extend a newly-developed gated recurrent neural network for monaural separation by additionally incorporating self-attention mechanisms and dense connectivity. We develop an end-to-end multiple-input multiple-output system, which directly maps from the binaural waveform of the mixture to those of the speech signals. The experimental results show that our proposed approach achieves significantly better separation performance than a recent binaural separation approach. In addition, our approach effectively preserves the interaural cues, which improves the accuracy of sound localization., Comment: 5 pages, accepted by IEEE Signal Processing Letters

Background noise, interfering speech and room reverberation frequently distort target speech in real listening environments. In this study, we address joint speech separation and dereverberation, which aims to separate target speech from background noise, interfering speech and room reverberation. In order to tackle this fundamentally difficult problem, we propose a novel multimodal network that exploits both audio and visual signals. The proposed network architecture adopts a two-stage strategy, where a separation module is employed to attenuate background noise and interfering speech in the first stage and a dereverberation module to suppress room reverberation in the second stage. The two modules are first trained separately, and then integrated for joint training, which is based on a new multi-objective loss function. Our experimental results show that the proposed multimodal network yields consistently better objective intelligibility and perceptual quality than several one-stage and two-stage baselines. We find that our network achieves a 21.10% improvement in ESTOI and a 0.79 improvement in PESQ over the unprocessed mixtures. Moreover, our network architecture does not require the knowledge of the number of speakers., Comment: 13 pages, accepted by IEEE JSTSP Special Issue on Deep Learning for Multi-modal Intelligence across Speech, Language, Vision, and Heterogeneous Signals

Monaural speech enhancement has made dramatic advances since the introduction of deep learning a few years ago. Although enhanced speech has been demonstrated to have better intelligibility and quality for human listeners, feeding it directly to automatic speech recognition (ASR) systems trained with noisy speech has not produced expected improvements in ASR performance. The lack of an enhancement benefit on recognition, or the gap between monaural speech enhancement and recognition, is often attributed to speech distortions introduced in the enhancement process. In this study, we analyze the distortion problem, compare different acoustic models, and investigate a distortion-independent training scheme for monaural speech recognition. Experimental results suggest that distortion-independent acoustic modeling is able to overcome the distortion problem. Such an acoustic model can also work with speech enhancement models different from the one used during training. Moreover, the models investigated in this paper outperform the previous best system on the CHiME-2 corpus.

This study investigates phase reconstruction for deep learning based monaural talker-independent speaker separation in the short-time Fourier transform (STFT) domain. The key observation is that, for a mixture of two sources, with their magnitudes accurately estimated and under a geometric constraint, the absolute phase difference between each source and the mixture can be uniquely determined; in addition, the source phases at each time-frequency (T-F) unit can be narrowed down to only two candidates. To pick the right candidate, we propose three algorithms based on iterative phase reconstruction, group delay estimation, and phase-difference sign prediction. State-of-the-art results are obtained on the publicly available wsj0-2mix and 3mix corpus., Comment: 5 pages, in submission to ICASSP-2019

To investigate in more details of Stellar Abundance and Galactic Evolution (SAGE) and in a huge sample, we are performing a northern sky photometric survey named SAGES with the SAGE photometric system, which consists of 8 filters: Str\"omgren-$u$, SAGE-$v$, SDSS $g$, $r$, $i$, DDO-$51$, $H\alpha_{wide}$, and $H\alpha _{narrow}$, including three Sloan broadband filters, three intermediate-band filters and two narrow-band filters, and one newly-designed narrow-band filter. SAGES covers $\sim$12,000 square degrees of the northern sky with $\delta > -5 ^{\circ}$, excluding the Galactic disk ($|b|<10^{\circ}$) and the sky area of 12 hr $<$ R.A. $<$ 18\,hr. The photometric detection limit depth at signal-to-noise ratio $5\sigma$ can be as deep as $V\sim$20\,mag. The SAGES will produce a depth-uniformed photometric catalogue for $\sim$500 million stars with atmospheric parameters including effective temperature $T_{\rm eff}$, surface gravity log\,g, and metallicity [Fe/H], as well as interstellar extinction to each individual target. In this work, we will briefly introduce the SAGE photometric system, the SAGE survey, and a preliminary test field of the open cluster NGC\,6791 and around.

Sky survey is one of the most important motivations to improve the astrophysics development, especially when using new photometric bands. We are performing the SAGE (Stellar Abundance and Galactic Evolution) survey with a self-designed SAGE photometric system, which is composed of eight photometric bands. The project mainly aims to study the stellar atmospheric parameters of $\sim$0.5 billion stars in the $\sim12,000$ deg$^2$ of the northern sky, which mainly focuses on the Galactic sciences, as well as some extragalactic sciences. This work introduces the detailed data reduction process of the testing field NGC\,6791, including the data reduction of single-exposure image and stacking multi-exposure images, and properties of the final catalogue.