Search

This technical report presents our initial attempt to build a spoken large language model (LLM) for Taiwanese Mandarin, specifically tailored to enable real-time, speech-to-speech interaction in multi-turn conversations. Our end-to-end model incorporates a decoder-only transformer architecture and aims to achieve seamless interaction while preserving the conversational flow, including full-duplex capabilities allowing simultaneous speaking and listening. The paper also details the training process, including data preparation with synthesized dialogues and adjustments for real-time interaction. We also developed a platform to evaluate conversational fluency and response coherence in multi-turn dialogues. We hope the release of the report can contribute to the future development of spoken LLMs in Taiwanese Mandarin., Comment: Work in progress

Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of a comprehensive evaluation benchmark poses a significant challenge. We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models. Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks, making it the largest benchmark for speech and audio evaluation. While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio. Evaluation results indicate that none of the models performed well universally. SALMONN-13B excelled in English ASR, while WavLLM demonstrated high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks. We will soon open-source all task data and the evaluation pipeline.

Warning: This paper may contain texts with uncomfortable content. Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.

Speech Integrated Large Language Models (SILLMs) combine large language models with speech perception to perform diverse tasks, such as emotion recognition to speaker verification, demonstrating universal audio understanding capability. However, these models may amplify biases present in training data, potentially leading to biased access to information for marginalized groups. This work introduces a curated spoken bias evaluation toolkit and corresponding dataset. We evaluate gender bias in SILLMs across four semantic-related tasks: speech-to-text translation (STT), spoken coreference resolution (SCR), spoken sentence continuation (SSC), and spoken question answering (SQA). Our analysis reveals that bias levels are language-dependent and vary with different evaluation methods. Our findings emphasize the necessity of employing multiple approaches to comprehensively assess biases in SILLMs, providing insights for developing fairer SILLM systems.

Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. However, it is unclear whether these LLM-based evaluators can be applied in real-world classrooms to assess student assignments. This empirical report shares how we use GPT-4 as an automatic assignment evaluator in a university course with 1,028 students. Based on student responses, we find that LLM-based assignment evaluators are generally acceptable to students when students have free access to these LLM-based evaluators. However, students also noted that the LLM sometimes fails to adhere to the evaluation instructions. Additionally, we observe that students can easily manipulate the LLM-based evaluator to output specific strings, allowing them to achieve high scores without meeting the assignment rubric. Based on student feedback and our experience, we provide several recommendations for integrating LLM-based evaluators into future classrooms. Our observation also highlights potential directions for improving LLM-based evaluators, including their instruction-following ability and vulnerability to prompt hacking., Comment: EMNLP 2024 main conference paper. An empirical report of our course: Introduction to Generative AI 2024 Spring (https://speech.ee.ntu.edu.tw/~hylee/genai/2024-spring.php)

Metallic hydrogen and hydride materials stand as promising avenues to achieve room-temperature superconductivity. Characterized by their high phonon frequencies and moderate coupling strengths, several high-pressure hydrides were theoretically predicted to exhibit transition temperatures ($T_c$) exceeding 250\,K, a claim further substantiated by experimental evidence. In an effort to push $T_c$ beyond room temperature, we introduce a dynamical method that involves stimulating hydrides with mid-infrared lasers. Employing Floquet first-principles simulations, we observe that in a nonequilibrium state induced by light, both the electronic density of states and the coupling to high-energy phonons see notable enhancements. These simultaneous improvements collectively result in an estimated 20\%-30\% rise in $T_c$ in practical pump conditions. Our theoretical investigation, therefore, offers a novel strategy to potentially raise the $T_c$ of hydrides above room temperature., Comment: 11 pages, 6 figures

Motivated by properties-controlling potential of the strain, we investigate strain dependence of structure, electronic and magnetic properties of Sr$_2$IrO$_4$ using complementary theoretical tools: {\it ab-initio} calculations, analytical approaches (rigid octahedra picture, Slater-Koster integrals), and extended $t-{\mathcal{J}}$ model. We find that strain affects both Ir-Ir distance and Ir-O-Ir angle, and the rigid octahedra picture is not relevant. Second, we find fundamentally different behavior for compressive and tensile strain. One remarkable feature is the formation of two subsets of bond- and orbital- dependent carriers, a compass-like model, under compression. This originates from the strain-induced renormalization of the Ir-O-Ir superexchange and O on-site energy. We also show that under compressive (tensile) strain, Fermi surface becomes highly dispersive (relatively flat). Already at a tensile strain of $1.5\%$, we observe spectral weight redistribution, with the low-energy band acquiring almost purely singlet character. These results can be directly compared with future experiments., Comment: 28 pages, 6 figures, includes Supplemental Information

p x n-type materials refer to materials with a p-type Seebeck coefficient in one direction and a n-type coefficient in the orthogonal direction. This type of materials allows for a transverse thermoelectric response, which is highly desirable for energy applications. Here, we report the observation of p x n-type behavior in TaIrTe4, a type-II Weyl semimetal, with an in-plane thermopower anisotropy S$_{xx}$-S$_{yy}$ reaches a maximum value 40${\mu}$V/K at 200K. Intriguingly. we found that such a p x n-type behavior is absent in the similar compound NbIrTe4. The presence and absence of p x n-type behavior in these two materials are consistent with density functional theory calculations, which further predict that the thermopower anisotropy in both compounds can be enhanced up to 130${\mu}$V/K by electron doping. Such a strong thermopower anisotropy originates from the presence of both p-type and n-type carriers, each with high mobility in one direction. These results suggest that although type-II Weyl semimetal phase does not guarantee the existence of p x n-type behavior, its unique band structure provides the ingredient to engineer and optimize this phenomenon., Comment: 9 pages, 3 figures

We study the square-lattice extended Hubbard model with on-site $U$ and nearest-neighbor $V$ interactions by exact diagonalization. We show that non-equilibrium quench dynamics can help determine the equilibrium phase transition boundaries, which agree with the calculations of fidelity metric, dynamical structure factor, and correlation function. At half filling, the phase diagrams in the strong-coupling regime include spin density wave and $d_{x^2-y^2}$-wave superconductivity at large positive $U$, charge density wave (extended $s^*$-wave superconductivity) at large positive (negative) $V$, and $s$-wave superconductivity at large negative $U$ with vanishing $V$. The energies of different particle sectors also help determine the phase separation region. With carrier doping, charge fluctuation result in strong competition between different orders, making it more difficult to identify the leading instability on finite-size cluster. Nevertheless, the more exotic $p$-wave superconducting pairing is found to be enhanced when the system is heavily overdoped by $37.5\%-50\%$ holes, especially in interaction parameter range relevant to the cuprate superconductors., Comment: 15 pages, 15 figures

We search for new superhard B-N-O compounds with an iterative machine learning (ML) procedure, where ML models are trained using sample crystal structures from evolutionary algorithm. We first use cohesive energy to evaluate the thermodynamic stability of varying B$_x$N$_y$O$_z$ compositions, and then gradually focus on compositional regions with high cohesive energy and high hardness. The results converge quickly after a few iterations. Our resulting ML models show that B$_{x+2}$N$_{x}$O$_{3}$ compounds with $x \geq 3$ (like B$_5$N$_3$O$_3$, B$_6$N$_4$O$_3$, etc.) are potentially superhard and thermodynamically favorable. Our meta-GGA density functional theory calculations indicate that these materials are also wide bandgap ($\ge 4.4$ eV) insulators, with the valence band maximum related to the $p$-orbitals of nitrogen atoms near vacant sites. This study demonstrates that an iterative method combining ML and ab initio simulations provides a powerful tool for discovering novel materials., Comment: 8 pages, 5 figures

The mechanism of unconventional superconductivity in correlated materials remains a great challenge in condensed matter physics. The recent discovery of superconductivity in infinite-layer nickelates, as analog to high-Tc cuprates, has opened a new route to tackle this challenge. By growing 8 nm Pr0.8Sr0.2NiO2 films on the (LaAlO3)0.3(Sr2AlTaO6)0.7 substrate, we successfully raise the transition temperature Tc from 9 K in the widely studied SrTiO3-substrated nickelates into 15 K. By combining x-ray absorption spectroscopy with the first-principles and many-body simulations, we find a positive correlation between Tc and the pre-edge peak intensity, which can be attributed to the hybridization between Ni and O orbitals induced by the strain. Our result suggests that structural engineering can further enhance unconventional superconductivity, and the charge-transfer property plays a crucial role in the pairing strength., Comment: 8 pages, 4 figures

We study lanthanum mononitride LaN by first-principles calculations. The commonly reported rock-salt structure of $Fm\bar{3}m$ symmetry for rare-earth monopnictides is found dynamically unstable for LaN at zero temperature. Using density functional theory and evolutionary crystal prediction, we discover a new, dynamically stable structure with $P1$ symmetry at 0 K. This $P1$-LaN exhibits spontaneous electric polarization. Our ab initio molecular dynamics simulations of finite-temperature phonon spectra further suggest that LaN will undergo ferroelectric and structural transitions from $P1$ to $Fm\bar{3}m$ symmetry, when temperature is increased. Moreover, $P1$-LaN will transform to a tetragonal structure with $P4/nmm$ symmetry at a critical pressure $P=18$ GPa at 0 K. Electronic structures computed with an advanced hybrid functional show that the high-temperature rock-salt LaN can change from a trivial insulator to a strong topological insulator at $P \sim 14$ GPa. Together, our results indicate that when $P=14 - 18$ GPa, LaN can show simultaneous temperature-induced structural, ferroelectric, and topological transitions. Lanthanum monopnictides thereby provide a rich playground for exploring novel phases and phase transitions driven by temperature and pressure.

We build random forests models to predict elastic properties and mechanical hardness of a compound, using only its chemical formula as input. The model training uses over 10,000 target compounds and 60 features based on stoichiometric attributes, elemental properties, orbital occupations, and ionic bonding levels. Using the models, we construct triangular graphs for B-C-N compounds to map out their bulk and shear moduli, as well as hardness values. The graphs indicate that a 1:1 B-N ratio can lead to various superhard compositions. We also validate the machine learning results by evolutionary structure prediction and density functional theory. Our study shows that BC$_{10}$N, B$_4$C$_5$N$_3$, and B$_2$C$_3$N exhibit dynamically stable phases with hardness values $>40$GPa, which are potentially new superhard materials that could be synthesized by low-temperature plasma methods.

Rare-earth monopnictides have attracted much attention due to their unusual electronic and topological properties for potential device applications. Here, we study rock-salt structured lanthanum monopnictides LaX (X = P, As) by density functional theory (DFT) simulations. We show systematically that a meta-GGA functional combined with scissor correction can efficiently and accurately compute electronic structures on a fine DFT $k$-grid, which is necessary for converging thermoelectric calculations. We also show that strain engineering can effectively improve thermoelectric performance. Under the optimal condition of 2% tensile strain and carrier concentration $n=3\times10^{20}~\textrm{cm}^{-3}$, LaP at temperature 1200 K can achieve a figure of merit $ZT$ value $>2$, which is enhanced by 90% compared to the unstrained value. With carrier doping and strain engineering, lanthanum monopnictides thereby could be promising high-temperature thermoelectric materials., Comment: 8 pages, 7 figures, with supplemental materials

A phase transition between topologically distinct insulating phases involves closing and reopening of the bandgap. Close to this topological phase transition, the bulk energy spectrum is characterized by a massive Dirac dispersion, where the mass plays the role of bandgap. Here we report the observation of a non-monotonic strain dependence of resistivity and negative longitudinal magnetoresistance in ZrTe5, which is known to host massive Dirac Fermions in the bulk. This non-monotonic strain dependence is consistent with the closing and reopening of the bandgap at the Brillouin-zone center, indicative of a topological phase transition. This observation suggests that the topological state of ZrTe5 is highly sensitive to uniaxial stress. Our study presents a promising platform for continuous in-situ control of nontrivial topological properties of materials., Comment: 27 pages, 11 figures