Search

The rapid advancement of large language models (LLMs) has significantly propelled the development of text-based chatbots, demonstrating their capability to engage in coherent and contextually relevant dialogues. However, extending these advancements to enable end-to-end speech-to-speech conversation bots remains a formidable challenge, primarily due to the extensive dataset and computational resources required. The conventional approach of cascading automatic speech recognition (ASR), LLM, and text-to-speech (TTS) models in a pipeline, while effective, suffers from unnatural prosody because it lacks direct interactions between the input audio and its transcribed text and the output audio. These systems are also limited by their inherent latency from the ASR process for real-time applications. This paper introduces Style-Talker, an innovative framework that fine-tunes an audio LLM alongside a style-based TTS model for fast spoken dialog generation. Style-Talker takes user input audio and uses transcribed chat history and speech styles to generate both the speaking style and text for the response. Subsequently, the TTS model synthesizes the speech, which is then played back to the user. While the response speech is being played, the input speech undergoes ASR processing to extract the transcription and speaking style, serving as the context for the ensuing dialogue turn. This novel pipeline accelerates the traditional cascade ASR-LLM-TTS systems while integrating rich paralinguistic information from input speech. Our experimental results show that Style-Talker significantly outperforms the conventional cascade and speech-to-speech baselines in terms of both dialogue naturalness and coherence while being more than 50% faster., Comment: CoLM 2024

Diffusion-based audio and music generation models commonly perform generation by constructing an image representation of audio (e.g., a mel-spectrogram) and then convert it to audio using a phase reconstruction model or vocoder. Typical vocoders, however, produce monophonic audio at lower resolutions (e.g., 16-24 kHz), which limits their usefulness. We propose MusicHiFi -- an efficient high-fidelity stereophonic vocoder. Our method employs a cascade of three generative adversarial networks (GANs) that convert low-resolution mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth extension, and upmixes to stereophonic audio. Compared to past work, we propose 1) a unified GAN-based generator and discriminator architecture and training procedure for each stage of our cascade, 2) a new fast, near downsampling-compatible bandwidth extension module, and 3) a new fast downmix-compatible mono-to-stereo upmixer that ensures the preservation of monophonic content in the output. We evaluate our approach using objective and subjective listening tests and find our approach yields comparable or better audio quality, better spatialization control, and significantly faster inference speed compared to past work. Sound examples are at \url{https://MusicHiFi.github.io/web/}., Comment: Accepted to IEEE Signal Processing Letters

Despite recent advancements in audio-text modeling, audio-text contrastive models still lag behind their image-text counterparts in scale and performance. We propose a method to improve both the scale and the training of audio-text contrastive models. Specifically, we craft a large-scale audio-text dataset containing 13,000 hours of text-labeled audio, using pretrained language models to process noisy text descriptions and automatic captioning to obtain text descriptions for unlabeled audio samples. We first train on audio-only data with a masked autoencoder (MAE) objective, which allows us to benefit from the scalability of unlabeled audio datasets. We then, initializing our audio encoder from the MAE model, train a contrastive model with an auxiliary captioning objective. Our final model, which we name Cacophony, achieves state-of-the-art performance on audio-text retrieval tasks, and exhibits competitive results on the HEAR benchmark and other downstream tasks such as zero-shot classification., Comment: Work in Progress

Audio diffusion models can synthesize a wide variety of sounds. Existing models often operate on the latent domain with cascaded phase recovery modules to reconstruct waveform. This poses challenges when generating high-fidelity audio. In this paper, we propose EDMSound, a diffusion-based generative model in spectrogram domain under the framework of elucidated diffusion models (EDM). Combining with efficient deterministic sampler, we achieved similar Fr\'echet audio distance (FAD) score as top-ranked baseline with only 10 steps and reached state-of-the-art performance with 50 steps on the DCASE2023 foley sound generation benchmark. We also revealed a potential concern regarding diffusion based audio generation models that they tend to generate samples with high perceptual similarity to the data from training data. Project page: https://agentcooper2002.github.io/EDMSound/, Comment: Accepted at NeurIPS Workshop: Machine Learning for Audio (Camera Ready)

A new nonlinear magnetothermal effect, namely unidirectional Seebeck effect(USE), has been recently reported in magnetic/nonmagnetic topological insulator (TI) heterostructure and is ascribed to the asymmetry magnon scattering. Here, we show a new mechanism to generate USE in the Rashba two dimensional electron gas (2DEG), in which the magnetism or magnetic order is completely absent. It's found that the USE has a quantum origin from the spin-momentum locking generated from Rashba spin-orbit coupling. We find that the USE exhibits a sine dependence on the orientation of the magnetic field with respect to the direction of the temperature gradient and is dominant from the states above the Lifshitz point. The USE in the semiconductor Ge(111) subsurface states has been theoretically and systematically investigated., Comment: 9 pages, 4 figures

Non-linguistic filler words, such as "uh" or "um", are prevalent in spontaneous speech and serve as indicators for expressing hesitation or uncertainty. Previous works for detecting certain non-linguistic filler words are highly dependent on transcriptions from a well-established commercial automatic speech recognition (ASR) system. However, certain ASR systems are not universally accessible from many aspects, e.g., budget, target languages, and computational power. In this work, we investigate filler word detection system that does not depend on ASR systems. We show that, by using the structured state space sequence model (S4) and neural semi-Markov conditional random fields (semi-CRFs), we achieve an absolute F1 improvement of 6.4% (segment level) and 3.1% (event level) on the PodcastFillers dataset. We also conduct a qualitative analysis on the detected results to analyze the limitations of our proposed system., Comment: Accepted by ICASSP 2023

Current methods aggregate multi-level features or introduce edge and skeleton to get more refined saliency maps. However, little attention is paid to how to obtain the complete salient object in cluttered background, where the targets are usually similar in color and texture to the background. To handle this complex scene, we propose a sharp eyes network (SENet) that first seperates the object from scene, and then finely segments it, which is in line with human visual characteristics, i.e., to look first and then focus. Different from previous methods which directly integrate edge or skeleton to supplement the defects of objects, the proposed method aims to utilize the expanded objects to guide the network obtain complete prediction. Specifically, SENet mainly consists of target separation (TS) brach and object segmentation (OS) branch trained by minimizing a new hierarchical difference aware (HDA) loss. In the TS branch, we construct a fractal structure to produce saliency features with expanded boundary via the supervision of expanded ground truth, which can enlarge the detail difference between foreground and background. In the OS branch, we first aggregate multi-level features to adaptively select complementary components, and then feed the saliency features with expanded boundary into aggregated features to guide the network obtain complete prediction. Moreover, we propose the HDA loss to further improve the structural integrity and local details of the salient objects, which assigns weight to each pixel according to its distance from the boundary hierarchically. Hard pixels with similar appearance in border region will be given more attention hierarchically to emphasize their importance in completeness prediction. Comprehensive experimental results on five datasets demonstrate that the proposed approach outperforms the state-of-the-art methods both quantitatively and qualitatively.

The empirical classification of gamma-ray bursts (GRBs) into long and short GRBs based on their durations is already firmly established. This empirical classification is generally linked to the physical classification of GRBs originating from compact binary mergers and GRBs originating from massive star collapses, or Type I and II GRBs, with the majority of short GRBs belonging to Type I and the majority of long GRBs belonging to Type II. However, there is a significant overlap in the duration distributions of long and short GRBs. Furthermore, some intermingled GRBs, i.e., short-duration Type II and long-duration Type I GRBs, have been reported. A multi-parameter classification scheme of GRBs is evidently needed. In this paper, we seek to build such a classification scheme with supervised machine learning methods, chiefly XGBoost. We utilize the GRB Big Table and Greiner's GRB catalog and divide the input features into three subgroups: prompt emission, afterglow, and host galaxy. We find that the prompt emission subgroup performs the best in distinguishing between Type I and II GRBs. We also find the most important distinguishing feature in prompt emission to be T_{90}, hardness ratio, and fluence. After building the machine learning model, we apply it to the currently unclassified GRBs to predict their probabilities of being either GRB class, and we assign the most probable class of each GRB to be its possible physical class., Comment: 23 pages, 8 tables, 11 figures, accepted for publication by ApJ. Full version of Table 5 is available as ancillary material

Fast radio bursts (FRBs) are one of the most mysterious astronomical transients. Observationally, they can be classified into repeaters and apparently non-repeaters. However, due to the lack of continuous observations, some apparently repeaters may have been incorrectly recognized as non-repeaters. In a series of two papers, we intend to solve such problem with machine learning. In this second paper of the series, we focus on an array of unsupervised machine learning methods. We apply multiple unsupervised machine learning algorithms to the first CHIME/FRB catalog to learn their features and classify FRBs into different clusters without any premise about the FRBs being repeaters or non-repeaters. These clusters reveal the differences between repeaters and non-repeaters. Then, by comparing with the identities of the FRBs in the observed classes, we evaluate the performance of various algorithms and analyze the physical meaning behind the results. Finally, we recommend a list of most credible repeater candidates as targets for future observing campaigns to search for repeated bursts in combination of the results presented in Paper I using supervised machine learning methods., Comment: 15 pages, 15 figures, 2 tables, published on MNRAS. Updated to match the published version

Observationally, the mysterious fast radio bursts (FRBs) are classified as repeating ones and apparently non-repeating ones. While repeating FRBs cannot be classified into the non-repeating group, it is unknown whether the apparently non-repeating FRBs are actually repeating FRBs whose repetitions are yet to be discovered, or whether they belong to another physically distinct type from the repeating ones. In a series of two papers, we attempt to disentangle this mystery with machine learning methods. In this first paper, we focus on an array of supervised machine learning methods. We train the machine learning algorithms with a fraction of the observed FRBs in the first CHIME/FRB catalog, telling them which ones are apparently non-repeating and which ones are repeating. We then let the trained models predict the repetitiveness of the rest of the FRB data with the observed parameters, and we compare the predictions with the observed repetitiveness. We find that the models can predict most FRBs correctly, hinting towards distinct mechanisms behind repeating and non-repeating FRBs. We also find that the two most important distinguishing factors between non-repeating and repeating FRBs are brightness temperature and rest-frame frequency bandwidth. By applying the trained models back to the entire first CHIME catalog, we further identify some potentially repeating FRBs currently reported as non-repeating. We recommend a list of these bursts as targets for future observing campaigns to search for repeated bursts in a combination with the results presented in Paper II using unsupervised machine learning methods., Comment: 14 pages, 5 figures, 4 tables, published on MNRAS. Updated to match the published version

Fully-supervised models for source separation are trained on parallel mixture-source data and are currently state-of-the-art. However, such parallel data is often difficult to obtain, and it is cumbersome to adapt trained models to mixtures with new sources. Source-only supervised models, in contrast, only require individual source data for training. In this paper, we first leverage flow-based generators to train individual music source priors and then use these models, along with likelihood-based objectives, to separate music mixtures. We show that in singing voice separation and music separation tasks, our proposed method is competitive with a fully-supervised approach. We also demonstrate that we can flexibly add new types of sources, whereas fully-supervised approaches would require retraining of the entire model., Comment: Accepted by Signal Processing Letters

Filler words such as `uh' or `um' are sounds or words people use to signal they are pausing to think. Finding and removing filler words from recordings is a common and tedious task in media editing. Automatically detecting and classifying filler words could greatly aid in this task, but few studies have been published on this problem to date. A key reason is the absence of a dataset with annotated filler words for model training and evaluation. In this work, we present a novel speech dataset, PodcastFillers, with 35K annotated filler words and 50K annotations of other sounds that commonly occur in podcasts such as breaths, laughter, and word repetitions. We propose a pipeline that leverages VAD and ASR to detect filler candidates and a classifier to distinguish between filler word types. We evaluate our proposed pipeline on PodcastFillers, compare to several baselines, and present a detailed ablation study. In particular, we evaluate the importance of using ASR and how it compares to a transcription-free approach resembling keyword spotting. We show that our pipeline obtains state-of-the-art results, and that leveraging ASR strongly outperforms a keyword spotting approach. We make PodcastFillers publicly available, in the hope that our work serves as a benchmark for future research., Comment: To appear at Insterspeech 2022

The performance of automatic speaker verification (ASV) systems could be degraded by voice spoofing attacks. Most existing works aimed to develop standalone spoofing countermeasure (CM) systems. Relatively little work targeted at developing an integrated spoofing aware speaker verification (SASV) system. In the recent SASV challenge, the organizers encourage the development of such integration by releasing official protocols and baselines. In this paper, we build a probabilistic framework for fusing the ASV and CM subsystem scores. We further propose fusion strategies for direct inference and fine-tuning to predict the SASV score based on the framework. Surprisingly, these strategies significantly improve the SASV equal error rate (EER) from 19.31% of the baseline to 1.53% on the official evaluation trials of the SASV challenge. We verify the effectiveness of our proposed components through ablation studies and provide insights with score distribution analysis., Comment: 8 pages, 5 figures, to be appear in Odyssey 2022

In this paper, we conduct a cross-dataset study on parametric and non-parametric raw-waveform based speaker embeddings through speaker verification experiments. In general, we observe a more significant performance degradation of these raw-waveform systems compared to spectral based systems. We then propose two strategies to improve the performance of raw-waveform based systems on cross-dataset tests. The first strategy is to change the real-valued filters into analytic filters to ensure shift-invariance. The second strategy is to apply variational dropout to non-parametric filters to prevent them from overfitting irrelevant nuance features.

In this paper, we present UR-AIR system submission to the logical access (LA) and the speech deepfake (DF) tracks of the ASVspoof 2021 Challenge. The LA and DF tasks focus on synthetic speech detection (SSD), i.e. detecting text-to-speech and voice conversion as spoofing attacks. Different from previous ASVspoof challenges, the LA task this year presents codec and transmission channel variability, while the new task DF presents general audio compression. Built upon our previous research work on improving the robustness of the SSD systems to channel effects, we propose a channel-robust synthetic speech detection system for the challenge. To mitigate the channel variability issue, we use an acoustic simulator to apply transmission codec, compression codec, and convolutional impulse responses to augmenting the original datasets. For the neural network backbone, we propose to use Emphasized Channel Attention, Propagation and Aggregation Time Delay Neural Networks (ECAPA-TDNN) as our primary model. We also incorporate one-class learning with channel-robust training strategies to further learn a channel-invariant speech representation. Our submission achieved EER 20.33% in the DF task; EER 5.46% and min-tDCF 0.3094 in the LA task., Comment: To appear in Proc. ASVspoof 2021 Workshop

Spoofing countermeasure (CM) systems are critical in speaker verification; they aim to discern spoofing attacks from bona fide speech trials. In practice, however, acoustic condition variability in speech utterances may significantly degrade the performance of CM systems. In this paper, we conduct a cross-dataset study on several state-of-the-art CM systems and observe significant performance degradation compared with their single-dataset performance. Observing differences of average magnitude spectra of bona fide utterances across the datasets, we hypothesize that channel mismatch among these datasets is one important reason. We then verify it by demonstrating a similar degradation of CM systems trained on original but evaluated on channel-shifted data. Finally, we propose several channel robust strategies (data augmentation, multi-task learning, adversarial learning) for CM systems, and observe a significant performance improvement on cross-dataset experiments., Comment: 5 pages, 6 figures, in Proc. INTERSPEECH 2021