Search

This technical report presents our initial attempt to build a spoken large language model (LLM) for Taiwanese Mandarin, specifically tailored to enable real-time, speech-to-speech interaction in multi-turn conversations. Our end-to-end model incorporates a decoder-only transformer architecture and aims to achieve seamless interaction while preserving the conversational flow, including full-duplex capabilities allowing simultaneous speaking and listening. The paper also details the training process, including data preparation with synthesized dialogues and adjustments for real-time interaction. We also developed a platform to evaluate conversational fluency and response coherence in multi-turn dialogues. We hope the release of the report can contribute to the future development of spoken LLMs in Taiwanese Mandarin., Comment: Work in progress

In this work, we introduce Speech-Copilot, a modular framework for instruction-oriented speech-processing tasks that minimizes human effort in toolset construction. Unlike end-to-end methods using large audio-language models, Speech-Copilot builds speech processing-specific toolsets by analyzing pre-collected task instructions and breaking tasks into manageable sub-tasks. It features a flexible agent based on large language models that performs tasks through program generation. Our approach achieves state-of-the-art performance on the Dynamic-SUPERB benchmark, demonstrating its effectiveness across diverse speech-processing tasks. Key contributions include: 1) developing an innovative framework for speech processing-specific toolset construction, 2) establishing a high-performing agent based on large language models, and 3) offering a new perspective on addressing challenging instruction-oriented speech-processing tasks. Without additional training processes required by end-to-end approaches, our method provides a flexible and extendable solution for a wide range of speech-processing applications., Comment: Accepted to SLT 2024

Deep Learning-based end-to-end Automatic Speech Recognition (ASR) has made significant strides but still struggles with performance on out-of-domain samples due to domain shifts in real-world scenarios. Test-Time Adaptation (TTA) methods address this issue by adapting models using test samples at inference time. However, current ASR TTA methods have largely focused on non-continual TTA, which limits cross-sample knowledge learning compared to continual TTA. In this work, we first propose a Fast-slow TTA framework for ASR that leverages the advantage of continual and non-continual TTA. Following this framework, we introduce Dynamic SUTA (DSUTA), an entropy-minimization-based continual TTA method for ASR. To enhance DSUTA robustness for time-varying data, we design a dynamic reset strategy to automatically detect domain shifts and reset the model, making it more effective at handling multi-domain data. Our method demonstrates superior performance on various noisy ASR datasets, outperforming both non-continual and continual TTA baselines while maintaining robustness to domain changes without requiring domain boundary information., Comment: Accepted by EMNLP 2024

Large audio-language models (LALMs) enhance traditional large language models by integrating audio perception capabilities, allowing them to tackle audio-related tasks. Previous research has primarily focused on assessing the performance of LALMs across various tasks, yet overlooking their reliability, particularly concerning issues like object hallucination. In our study, we introduce methods to assess the extent of object hallucination of publicly available LALMs. Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content, but struggle to answer discriminative questions, specifically those requiring the identification of the presence of particular object sounds within an audio clip. This limitation highlights a critical weakness in current LALMs: their inadequate understanding of discriminative queries. Moreover, we explore the potential of prompt engineering to enhance LALMs' performance on discriminative questions., Comment: Accepted to Interspeech 2024

This paper presents an effective transfer learning framework for language adaptation in text-to-speech systems, with a focus on achieving language adaptation using minimal labeled and unlabeled data. While many works focus on reducing the usage of labeled data, very few consider minimizing the usage of unlabeled data. By utilizing self-supervised features in the pretraining stage, replacing the noisy portion of pseudo labels with these features during fine-tuning, and incorporating an embedding initialization trick, our method leverages more information from unlabeled data compared to conventional approaches. Experimental results show that our framework is able to synthesize intelligible speech in unseen languages with only 4 utterances of labeled data and 15 minutes of unlabeled data. Our methodology continues to surpass conventional techniques, even when a greater volume of data is accessible. These findings highlight the potential of our data-efficient language adaptation framework., Comment: Accepted by ASRU 2023

We reveal that the criticality of the chiral phase transition in QCD at the macroscale arises from the microscopic energy levels of its fundamental constituents, the quarks. We establish a novel relation between cumulants of the chiral order parameter (i.e., chiral condensate) and correlations among the energy levels of quarks (i.e., eigenspectra of the massless Dirac operator), which naturally leads to a generalization of the Banks-Casher relation. Based on this novel relation and through (2+1)-flavor lattice QCD calculations using the HISQ action with varying light quark masses in the vicinity of the chiral phase transition, we demonstrate that the correlations among the infrared part of the Dirac eigenspectra exhibit same universal scaling behaviors as expected of the cumulants of the chiral condensate. We find that these universal scaling behaviors extend up to the physical values of the up and down quark masses., Comment: 4 pages, 2 figures, talk presented at the 30th International Conference on Ultra-relativistic Nucleus-Nucleus Collisions (Quark Matter 2023), September 3-9, 2023, Houston, Texas, USA. arXiv admin note: text overlap with arXiv:2401.10263

We reveal that the universal scaling properties of the chiral phase transition in quantum chromodynamics (QCD) at the macroscale are, in fact, encoded within the microscopic energy levels of its fundamental constituents, the quarks. We introduce a novel relation between the cumulants of the chiral order parameter, i.e., the chiral condensate, and the correlations among the energy levels of quarks, i.e., the eigenspectra of the massless QCD Dirac operator. This relation elucidates how the fluctuations of the chiral condensate arise from the correlations within the infrared part of the energy spectra of quarks, and naturally leads to a generalization of the Banks-Casher relation for the cumulants of the chiral condensate. Then, through (2+1)-flavor lattice QCD calculations using HISQ action with varying light quark masses around the chiral phase transition temperature, we demonstrate that the correlations among the infrared part of the Dirac eigenvalue spectra exhibit same universal scaling behaviors as expected of the cumulants of the chiral condensate. We find that these universal scaling behaviors extend up to the physical values of the up and down quark masses., Comment: 9 pages, 3 figures, talk presented at the 40th International Symposium on Lattice Field Theory (Lattice2023), July 31st - August 4th, 2023, Fermi National Accelerator Laboratory

The 2023 Multilingual Speech Universal Performance Benchmark (ML-SUPERB) Challenge expands upon the acclaimed SUPERB framework, emphasizing self-supervised models in multilingual speech recognition and language identification. The challenge comprises a research track focused on applying ML-SUPERB to specific multilingual subjects, a Challenge Track for model submissions, and a New Language Track where language resource researchers can contribute and evaluate their low-resource language data in the context of the latest progress in multilingual speech recognition. The challenge garnered 12 model submissions and 54 language corpora, resulting in a comprehensive benchmark encompassing 154 languages. The findings indicate that merely scaling models is not the definitive solution for multilingual speech tasks, and a variety of speech/voice types present significant challenges in multilingual speech processing., Comment: Accepted by ASRU

This paper emphasizes the importance of reporting experiment details in subjective evaluations and demonstrates how such details can significantly impact evaluation results in the field of speech synthesis. Through an analysis of 80 papers presented at INTERSPEECH 2022, we find a lack of thorough reporting on critical details such as evaluator recruitment and filtering, instructions and payments, and the geographic and linguistic backgrounds of evaluators. To illustrate the effect of these details on evaluation outcomes, we conducted mean opinion score (MOS) tests on three well-known TTS systems under different evaluation settings and we obtain at least three distinct rankings of TTS models. We urge the community to report experiment details in subjective evaluations to improve the reliability and interpretability of experimental results., Comment: Interspeech 2023 camera-ready version

Macroscopic properties of the strong interaction near its chiral phase transition exhibit scaling behaviors, which are the same as those observed close to the magnetic transition in a 3-dimensional classical spin system with $O(4)$ symmetry. We show that the universal scaling properties of the chiral phase transition in Quantum Chromodynamics (QCD) at the macroscale are, in fact, encoded within the microscopic energy levels of its fundamental constituents, the quarks. We establish a connection between the cumulants of the chiral order parameter, i.e., the chiral condensate, and the correlations among the energy levels of quarks, i.e., the eigenspectra of the massless QCD Dirac operator. This relation elucidates how the fluctuations of the chiral condensate arise from the correlations within the infrared part of the energy spectra of quarks, and naturally leads to a generalization of the Banks-Casher relation for the cumulants of the chiral condensate. Then, through (2+1)-flavor lattice QCD calculations with varying light quark masses near the QCD chiral transition, we demonstrate that the correlations among the infrared part of the Dirac eigenvalue spectra exhibit same universal scaling behaviors as expected of the cumulants of the chiral condensate. We find that these universal scaling behaviors extend up to the physical values of the up and down quark masses. Our study reveals how the hidden scaling features at the microscale give rise to the macroscopic universal properties of QCD., Comment: 5 pages main text + 3 pages supplemental material, version to appear in PRL

Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks. However, SUPERB largely considers English speech in its evaluation. This paper presents multilingual SUPERB (ML-SUPERB), covering 143 languages (ranging from high-resource to endangered), and considering both automatic speech recognition and language identification. Following the concept of SUPERB, ML-SUPERB utilizes frozen SSL features and employs a simple framework for multilingual tasks by learning a shallow downstream model. Similar to the SUPERB benchmark, we find speech SSL models can significantly improve performance compared to FBANK features. Furthermore, we find that multilingual models do not always perform better than their monolingual counterparts. We will release ML-SUPERB as a challenge with organized datasets and reproducible training scripts for future multilingual representation research., Comment: Accepted by Interspeech

Self-Supervised Learning (SSL) from speech data has produced models that have achieved remarkable performance in many tasks, and that are known to implicitly represent many aspects of information latently present in speech signals. However, relatively little is known about the suitability of such models for prosody-related tasks or the extent to which they encode prosodic information. We present a new evaluation framework, SUPERB-prosody, consisting of three prosody-related downstream tasks and two pseudo tasks. We find that 13 of the 15 SSL models outperformed the baseline on all the prosody-related tasks. We also show good performance on two pseudo tasks: prosody reconstruction and future prosody prediction. We further analyze the layerwise contributions of the SSL models. Overall we conclude that SSL speech models are highly effective for prosody-related tasks., Comment: Accepted to IEEE SLT 2022

This paper studies a transferable phoneme embedding framework that aims to deal with the cross-lingual text-to-speech (TTS) problem under the few-shot setting. Transfer learning is a common approach when it comes to few-shot learning since training from scratch on few-shot training data is bound to overfit. Still, we find that the naive transfer learning approach fails to adapt to unseen languages under extremely few-shot settings, where less than 8 minutes of data is provided. We deal with the problem by proposing a framework that consists of a phoneme-based TTS model and a codebook module to project phonemes from different languages into a learned latent space. Furthermore, by utilizing phoneme-level averaged self-supervised learned features, we effectively improve the quality of synthesized speeches. Experiments show that using 4 utterances, which is about 30 seconds of data, is enough to synthesize intelligible speech when adapting to an unseen language using our framework., Comment: Submitted to Interspeech 2022

We investigate the criticality of chiral phase transition manifested in the first and second order derivatives of Dirac eigenvalue spectrum with respect to light quark mass in (2+1)-flavor lattice QCD. Simulations are performed at temperatures from about 137 MeV to 176 MeV on $N_{\tau}=8$ lattices using the highly improved staggered quarks and the tree-level improved Symanzik gauge action. The strange quark mass is fixed to its physical value $m_s^{\text{phy}}$ and the light quark mass is set to $m_s^{\text{phy}}/40$ which corresponds to a Goldstone pion mass $m_{\pi}=110$ MeV. We find that in contrast to the case at $T\simeq 205$ MeV $m_l^{-1} \partial \rho(\lambda, m_l)/\partial m_l$ is no longer equal to $\partial ^2\rho(\lambda, m_l)/\partial m_l^2$ and $\partial ^2\rho(\lambda, m_l)/\partial m_l^2$ even becomes negative at certain low temperatures. This means that as temperature getting closer to $T_c$ $\rho(\lambda, m_l)$ is no longer proportional to $m_l^2$ and thus dilute instanton gas approximation is not valid for these temperatures. We demonstrate the temperature dependence can be factored out in $\partial \rho(\lambda, m_l)/ \partial m_l$ and $\partial^2 \rho(\lambda, m_l)/ \partial m_l^2$ at $T \in [137, 153]$ MeV, and then we propose a feasible method to estimate the power $c$ given $\rho \propto m_l^{c}$., Comment: Talk presented at the 38th International Symposium on Lattice Field Theory - LATTICE2021, 26th-30th, July, 2021, Zoom/Gather@Massachusetts Institute of Technology