Search

Recent advances in large language models (LLMs) have gained interest in speech-text multimodal foundation models, achieving strong performance on instruction-based speech translation (ST). However, expanding language pairs from an existing instruction-tuned ST system is costly due to the necessity of re-training on a combination of new and previous datasets. We propose to expand new language pairs by merging the model trained on new language pairs and the existing model, using task arithmetic. We find that the direct application of task arithmetic for ST causes the merged model to fail to follow instructions; thus, generating translation in incorrect languages. To eliminate language confusion, we propose an augmented task arithmetic method that merges an additional language control model. It is trained to generate the correct target language token following the instructions. Our experiments demonstrate that our proposed language control model can achieve language expansion by eliminating language confusion. In our MuST-C and CoVoST-2 experiments, it shows up to 4.66 and 4.92 BLEU scores improvement, respectively. In addition, we demonstrate the use of our task arithmetic framework can expand to a language pair where neither paired ST training data nor a pre-trained ST model is available. We first synthesize the ST system from machine translation (MT) systems via task analogy, then merge the synthesized ST system to the existing ST model.

This study investigates the efficacy of data augmentation techniques for low-resource automatic speech recognition (ASR), focusing on two endangered Austronesian languages, Amis and Seediq. Recognizing the potential of self-supervised learning (SSL) in low-resource settings, we explore the impact of data volume on the continued pre-training of SSL models. We propose a novel data-selection scheme leveraging a multilingual corpus to augment the limited target language data. This scheme utilizes a language classifier to extract utterance embeddings and employs one-class classifiers to identify utterances phonetically and phonologically proximate to the target languages. Utterances are ranked and selected based on their decision scores, ensuring the inclusion of highly relevant data in the SSL-ASR pipeline. Our experimental results demonstrate the effectiveness of this approach, yielding substantial improvements in ASR performance for both Amis and Seediq. These findings underscore the feasibility and promise of data augmentation through cross-lingual transfer learning for low-resource language ASR.

We report a lattice QCD calculation of the parton distribution function (PDF) of a deuteron-like dibaryon system using large-momentum effective theory. The calculation is done on three Wilson Clover ensembles with a fixed lattice spacing a=0.105 fm and two pion masses. The lattice matrix elements are computed at proton momenta up to 2.46 GeV with the signal of high momentum modes being improved by applying the momentum smearing technique. The state-of-the-art renormalization, matching and extrapolation are then applied to obtain the final result of the light-cone PDF. A comparison between the result of the dibaryon system and the sum of the proton and neutron PDFs is also given.

We propose two approaches for extracting the total gluon helicity contribution to proton spin from lattice QCD, one from local operator matrix elements in a fixed gauge accessible on lattice with feasible renormalization, and the other from gauge-invariant nonlocal gluon correlators. Neither of these approaches requires a matching procedure when converted to the MS scheme. Our proposal resolves a long-standing inconsistency in the literature regarding lattice calculations of the total gluon helicity, and has the potential to greatly facilitate these calculations., Comment: 14 pages

Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features such as spectrograms are often used as the input for the subsequent model. However, they can still be redundant. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations, which significantly compresses the size of speech data. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length. Hence, training time is significantly reduced while retaining notable performance. In this study, we undertake a comprehensive and systematic exploration into the application of discrete units within end-to-end speech processing models. Experiments on 12 automatic speech recognition, 3 speech translation, and 1 spoken language understanding corpora demonstrate that discrete units achieve reasonably good results in almost all the settings. We intend to release our configurations and trained models to foster future research efforts., Comment: Submitted to IEEE ICASSP 2024

We present a unified framework for the perturbative factorization connecting Euclidean correlations to light-cone correlations. Starting from nonlocal quark and gluon bilinear correlators, we derive the relevant hard-matching kernel up to the next-to-leading-order, both for the flavor singlet and non-singlet combinations, in non-forward and forward kinematics, and in coordinate and momentum space. The results for the generalized distribution functions (GPDs), parton distribution functions (PDFs), and distribution amplitudes (DAs) are obtained by choosing appropriate kinematics. The renormalization and matching are done in a state-of-the-art scheme. We also clarify some issues raised on the perturbative matching of GPDs in the literature. Our results provide a complete manual for extracting all leading-twist GPDs, PDFs as well as DAs from lattice simulations of Euclidean correlations in a state-of-the-art strategy, either in coordinate or in momentum space factorization approach., Comment: 31 pages, 4 figures, typos corrected, additional appendix added

The lack of clean speech is a practical challenge to the development of speech enhancement systems, which means that there is an inevitable mismatch between their training criterion and evaluation metric. In response to this unfavorable situation, we propose a training and inference strategy that additionally uses enhanced speech as a target by improving the previously proposed noisy-target training (NyTT). Because homogeneity between in-domain noise and extraneous noise is the key to the effectiveness of NyTT, we train various student models by remixing 1) the teacher model's estimated speech and noise for enhanced-target training or 2) raw noisy speech and the teacher model's estimated noise for noisy-target training. Experimental results show that our proposed method outperforms several baselines, especially with the teacher/student inference, where predicted clean speech is derived successively through the teacher and final student models., Comment: Accepted by Interspeech 2023

Recording channel mismatch between training and testing conditions has been shown to be a serious problem for speech separation. This situation greatly reduces the separation performance, and cannot meet the requirement of daily use. In this study, inheriting the use of our previously constructed TAT-2mix corpus, we address the channel mismatch problem by proposing a channel-aware audio separation network (CasNet), a deep learning framework for end-to-end time-domain speech separation. CasNet is implemented on top of TasNet. Channel embedding (characterizing channel information in a mixture of multiple utterances) generated by Channel Encoder is introduced into the separation module by the FiLM technique. Through two training strategies, we explore two roles that channel embedding may play: 1) a real-life noise disturbance, making the model more robust, or 2) a guide, instructing the separation model to retain the desired channel information. Experimental results on TAT-2mix show that CasNet trained with both training strategies outperforms the TasNet baseline, which does not use channel embeddings., Comment: Submitted to ICASSP 2023

In the large-momentum expansion for parton distribution functions (PDFs), the natural physics scale is the longitudinal momentum ($p_z$) of the quarks (or gluons) in a large-momentum hadron. We show how to expose this scale dependence through resumming logarithms of the type $\ln^n p_z/\mu$ in the matching coefficient, where $\mu$ is a fixed renormalization scale. The result enhances the accuracy of the expansion at moderate $p_z>1$ GeV, and at the same time, clearly shows that the partons cannot be approximated from quarks with $p_z\sim \Lambda_{\rm QCD}$ which are not predominantly collinear with the parent hadron momentum, consistent with power counting of the large-momentum effective theory. The same physics mechanism constrains the coordinate space expansion at large distances $z$, the conjugate of $p_z$, as illustrated in the example of fitting the moments of the PDFs., Comment: 17 pages, 9 figures

We report a state-of-the-art lattice QCD calculation of the isovector quark transversity distribution of the proton in the continuum and physical mass limit using large-momentum effective theory. The calculation is done at four lattice spacings $a=\{0.098,0.085,0.064,0.049\}$~fm and various pion masses ranging between $220$ and $350$ MeV, with proton momenta up to $2.8$ GeV. The result is non-perturbatively renormalized in the hybrid scheme with self renormalization which treats the infrared physics at large correlation distance properly, and extrapolated to the continuum, physical mass and infinite momentum limit. We also compare with recent global analyses for the nucleon isovector quark transversity distribution., Comment: 16 pages, 18 figures, 2 tables

To calculate the transverse-momentum-dependent parton distribution functions (TMDPDFs) from lattice QCD, an important goal yet to be realized, it is crucial to establish a viable non-perturbative renormalization approach for linear divergences in the corresponding Euclidean quasi-TMDPDF correlators in large-momentum effective theory. We perform a first systematic study of the renormalization property of the quasi-TMDPDFs by calculating the relevant matrix elements in a pion state at 5 lattice spacings ranging from 0.03 fm to 0.12 fm. We demonstrate that the square root of the Wilson loop combined with the short distance hadron matrix element provides a successful method to remove all ultraviolet divergences of the quasi-TMD operator, and thus provide the necessary justification to perform a continuum limit calculation of TMDPDFs. In contrast, the popular RI/MOM renormalization scheme fails to eliminate all linear divergences., Comment: 17 pages, 17 figures. accepted version with the TMD wave function case in the supplemental materials

Compensation for channel mismatch and noise interference is essential for robust automatic speech recognition. Enhanced speech has been introduced into the multi-condition training of acoustic models to improve their generalization ability. In this paper, a noise-aware training framework based on two cascaded neural structures is proposed to jointly optimize speech enhancement and speech recognition. The feature enhancement module is composed of a multi-task autoencoder, where noisy speech is decomposed into clean speech and noise. By concatenating its enhanced, noise-aware, and noisy features for each frame, the acoustic-modeling module maps each feature-augmented frame into a triphone state by optimizing the lattice-free maximum mutual information and cross entropy between the predicted and actual state sequences. On top of the factorized time delay neural network (TDNN-F) and its convolutional variant (CNN-TDNNF), both with SpecAug, the two proposed systems achieve word error rate (WER) of 3.90% and 3.55%, respectively, on the Aurora-4 task. Compared with the best existing systems that use bigram and trigram language models for decoding, the proposed CNN-TDNNF-based system achieves a relative WER reduction of 15.20% and 33.53%, respectively. In addition, the proposed CNN-TDNNF-based system also outperforms the baseline CNN-TDNNF system on the AMI task., Comment: Published in ISCSLP 2022

In our previous work, we proposed a discriminative autoencoder (DcAE) for speech recognition. DcAE combines two training schemes into one. First, since DcAE aims to learn encoder-decoder mappings, the squared error between the reconstructed speech and the input speech is minimized. Second, in the code layer, frame-based phonetic embeddings are obtained by minimizing the categorical cross-entropy between ground truth labels and predicted triphone-state scores. DcAE is developed based on the Kaldi toolkit by treating various TDNN models as encoders. In this paper, we further propose three new versions of DcAE. First, a new objective function that considers both categorical cross-entropy and mutual information between ground truth and predicted triphone-state sequences is used. The resulting DcAE is called a chain-based DcAE (c-DcAE). For application to robust speech recognition, we further extend c-DcAE to hierarchical and parallel structures, resulting in hc-DcAE and pc-DcAE. In these two models, both the error between the reconstructed noisy speech and the input noisy speech and the error between the enhanced speech and the reference clean speech are taken into the objective function. Experimental results on the WSJ and Aurora-4 corpora show that our DcAE models outperform baseline systems., Comment: Published in Interspeech 2022

Realizing van der Waals (vdW) epitaxy in the 80s represents a breakthrough that circumvents the stringent lattice matching and processing compatibility requirements in conventional covalent heteroepitaxy. However, due to the weak vdW interactions, there is little control over film qualities by the substrate. Typically, discrete domains with a spread of misorientation angles are formed, limiting the applicability of vdW epitaxy. Here we report the epitaxial growth of monocrystalline, covalent Cr5Te8 2D crystals on monolayer vdW WSe2 by chemical vapor deposition, driven by interfacial dative bond formation. The lattice of Cr5Te8, with a lateral dimension of a few ten microns, is fully commensurate with that of WSe2 via 3 x 3 (Cr5Te8)-7 x 7 (WSe2) supercell matching, forming a single crystalline moire superlattice. Our work has established a conceptually distinct paradigm of thin film epitaxy termed dative epitaxy, which takes full advantage of covalent epitaxy with chemical bonding for fixing the atomic registry and crystal orientation, while circumventing its stringent lattice matching and processing compatibility requirements; conversely, it ensures the full flexibility of vdW epitaxy, while avoiding its poor orientation control. Cr5Te8 2D crystals grown by dative epitaxy exhibit square magnetic hysteresis, suggesting minimized interfacial defects that can serve as pinning sites.

We present the state-of-the-art lattice QCD calculation of the pion and kaon light-cone distribution amplitudes (DAs) using large-momentum effective theory. The calculation is done at three lattice spacings $a\approx\{0.06,0.09,0.12\}$ fm and physical pion and kaon masses, with the meson momenta $P_z = \{1.29,1.72,2.15\}$ GeV. The result is non-perturbatively renormalized in a recently proposed hybrid scheme with self renormalization, and extrapolated to the continuum as well as the infinite momentum limit. We find a significant deviation of the pion and kaon DAs from the asymptotic form, and a large $SU(3)$ flavor breaking effect in the kaon DA., Comment: 11 pages,18 figures

Molybdenum Disulfide (MoS$_2$) has been recognized as a potential substitution of Platinum (Pt) for electrochemical hydrogen evolution reaction (HER). However, the broad adoption of MoS$_2$ is hindered by its limited number of active sites and relatively low inherent electrical conductivity. In this work, we demonstrated a synergistic enhancement of both active site exposure and electrical conductivity by a one-step solvothermal synthesis technique. The 1T-phase enriched MoS$_2$ was directly formed on the titanium carbide (Ti$_3$C$_2$Tx, MXene) with carbon nanotubes (CNTs) acting as crosslinks. The existence of edge-enriched metallic phase MoS$_2$, the conductive backbone of MXene along with the crosslinking function of CNTs clearly improved the overall electrical conductivity of the catalyst. Moreover, the integration of two-dimensional (2D) MoS$_2$ with MXene effectively suppressed the MXene oxidation and 2D layer restacking, leading to good catalytic stability. As a result, an overpotential of 169 mV and a low Tafel slope of 51 mV/dec was successfully achieved. This work provides a new route for 2D-based electrocatalyst engineering and sheds light on the development of the next-generation PGM-free HER electrocatalysts.