Search

We analyze contextual representations in neural autoregressive language models, emphasizing long-range contexts that span several thousand tokens. Our methodology employs a perturbation setup and the metric \emph{Anisotropy-Calibrated Cosine Similarity}, to capture the degree of contextualization of long-range patterns from the perspective of representation geometry. We begin the analysis with a case study on standard decoder-only Transformers, demonstrating that similar perplexity can exhibit markedly different downstream task performance, which can be explained by the difference in contextualization of long-range content. Next, we extend the analysis to other models, covering recent novel architectural designs and various training configurations. The representation-level results illustrate a reduced capacity for high-complexity (i.e., less compressible) sequences across architectures, and that fully recurrent models rely heavily on local context, whereas hybrid models more effectively encode the entire sequence structure. Finally, preliminary analysis of model size and training configurations on the encoding of long-range context suggest potential directions for improving existing language models., Comment: 17 pages, 9 figures

We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs., Comment: COLM 2024; Code is available at https://github.com/hsiehjackson/RULER

Building machine learning models can be a time-consuming process that often takes several months to implement in typical business scenarios. To ensure consistent model performance and account for variations in data distribution, regular retraining is necessary. This paper introduces a solution for improving online customer service in e-commerce by presenting a universal model for predict-ing labels based on customer questions, without requiring training. Our novel approach involves using machine learning techniques to tag customer questions in transcripts and create a repository of questions and corresponding labels. When a customer requests assistance, an information retrieval model searches the repository for similar questions, and statistical analysis is used to predict the corresponding label. By eliminating the need for individual model training and maintenance, our approach reduces both the model development cycle and costs. The repository only requires periodic updating to maintain accuracy.

Electric LC resonances, occurring in metallic circuits, govern the motion of free electrons or electric signals. In this paper, optical LC-like resonances in high-index particles (HIPs) or dielectric LC resonances have been studied, which involve bounded electrons and optical fields instead. The resonance effect is dominated by optical analogues of inductance and capacitance, which can be determined according to the electromagnetic energy bounded near the particle. The viewpoint of dielectric optical circuit with equivalent parameters facilitates the understanding of dielectric resonance effect. The result also provides a method for studying the optical properties of the HIPs., Comment: 19 pages, 5 figures

Diode effects are of great interest for both fundamental physics and modern technologies. Electrical diode effects (nonreciprocal transport) have been observed in Weyl systems. Optical diode effects arising from the Weyl fermions have been theoretically considered but not probed experimentally. Here, we report the observation of a nonlinear optical diode effect (NODE) in the magnetic Weyl semimetal CeAlSi, where the magnetization introduces a pronounced directionality in the nonlinear optical second-harmonic generation (SHG). We show demonstrate a six-fold change of the measured SHG intensity between opposite propagation directions over a bandwidth exceeding 250 meV. Supported by density-functional theory, we establish the linearly dispersive bands emerging from Weyl nodes as the origin of this broadband effect. We further demonstrate current-induced magnetization switching and thus electrical control of the NODE. Our results advance ongoing research to identify novel nonlinear optical/transport phenomena in magnetic topological materials and further opens new pathways for the unidirectional manipulation of light., Comment: 20 pages, 4 figures, SI included

The bound state of a $\tau^+\tau^-$ pair by the electromagnetic force is the heaviest and smallest QED atom. Since the discovery of the two lightest QED atoms more than 60 years ago, no evidence for the third one has been found. We demonstrate that the $J_\tau$ ($\tau^+\tau^-$ atom with $J^{PC}=1^{--}$) resonance signal can be observed with a significance larger than $5\sigma$ including both statistical and systematic uncertainties, via the process $e^+e^-\to X^+ Y^- \slashed{E}$ ($X,\,Y=e$, $\mu$, $\pi$, $K$, or $\rho$, and $\slashed{E}$ is the missing energy) with $1.5~{\rm ab^{-1}}$ data taken around the $\tau$ pair production threshold. The $\tau$ lepton mass can be measured in a precision of 2 keV with the same data sample. This is within one year's running time of the proposed super tau-charm factory or super charm-tau factory., Comment: 7 pages, 4 figures

In this paper, we prove a large deviation principle for the empirical measures of a system of weakly interacting diffusion with reflection. We adopt the weak convergence approach. To make this approach work, we show that the sequence of empirical measures of the controlled reflected system will converge to the weak solution of an associated reflected McKean--Vlasov equation.

Berry curvature multipoles appearing in topological quantum materials have recently attracted much attention. Their presence can manifest in novel phenomena, such as nonlinear anomalous Hall effects (NLAHE). The notion of Berry curvature multipoles extends our understanding of Berry curvature effects on the material properties. Hence, research on this subject is of fundamental importance and may also enable future applications in energy harvesting and high-frequency technology. It was shown that a Berry curvature dipole can give rise to a 2nd order NLAHE in materials of low crystalline symmetry. Here, we demonstrate a fundamentally new mechanism for Berry curvature multipoles in antiferromagnets that are supported by the underlying magnetic symmetries. Carrying out electric transport measurements on the kagome antiferromagnet FeSn, we observe a 3rd order NLAHE, which appears as a transverse voltage response at the 3rd harmonic frequency when a longitudinal a.c. current drive is applied. Interestingly, this NLAHE is strongest at and above room temperature. We combine these measurements with a scaling law analysis, a symmetry analysis, model calculations, first-principle calculations, and magnetic Monte-Carlo simulations to show that the observed NLAHE is induced by a Berry curvature quadrupole appearing in the spin-canted state of FeSn. At a practical level, our study establishes NLAHE as a sensitive probe of antiferromagnetic phase transitions in other materials, such as moir\'e superlattices, two-dimensional van der Waal magnets, and quantum spin liquid candidates, that remain poorly understood to date. More broadly, Berry curvature multipole effects are predicted to exist for 90 magnetic point groups. Hence, our work opens a new research area to study a variety of topological magnetic materials through nonlinear measurement protocols.

Fine-tuning is a popular method for adapting text-to-speech (TTS) models to new speakers. However this approach has some challenges. Usually fine-tuning requires several hours of high quality speech per speaker. There is also that fine-tuning will negatively affect the quality of speech synthesis for previously learnt speakers. In this paper we propose an alternative approach for TTS adaptation based on using parameter-efficient adapter modules. In the proposed approach, a few small adapter modules are added to the original network. The original weights are frozen, and only the adapters are fine-tuned on speech for new speaker. The parameter-efficient fine-tuning approach will produce a new model with high level of parameter sharing with original model. Our experiments on LibriTTS, HiFi-TTS and VCTK datasets validate the effectiveness of adapter-based method through objective and subjective metrics., Comment: Submitted to ICASSP 2023

Multimodal learning is a recent challenge that extends unimodal learning by generalizing its domain to diverse modalities, such as texts, images, or speech. This extension requires models to process and relate information from multiple modalities. In Information Retrieval, traditional retrieval tasks focus on the similarity between unimodal documents and queries, while image-text retrieval hypothesizes that most texts contain the scene context from images. This separation has ignored that real-world queries may involve text content, image captions, or both. To address this, we introduce Multimodal Retrieval on Representation of ImaGe witH Text (Mr. Right), a novel and comprehensive dataset for multimodal retrieval. We utilize the Wikipedia dataset with rich text-image examples and generate three types of text-based queries with different modality information: text-related, image-related, and mixed. To validate the effectiveness of our dataset, we provide a multimodal training paradigm and evaluate previous text retrieval and image retrieval frameworks. The results show that proposed multimodal retrieval can improve retrieval performance, but creating a well-unified document representation with texts and images is still a challenge. We hope Mr. Right allows us to broaden current retrieval systems better and contributes to accelerating the advancement of multimodal learning in the Information Retrieval., Comment: Dataset available at https://github.com/hsiehjackson/Mr.Right

Recently, a quantum anomalous Hall (QAH) state was observed in AB stacked moir\'e MoTe$_2$/WSe$_2$ heterobilayers at half-filling. More recent layer-resolved magnetic circular dichroism (MCD) measurements revealed that spin-polarized moir\'e bands from both the MoTe$_2$ and the WSe$_2$ layers are involved at the formation of the QAH state. This scenario is not expected by existing theories. In this work, we suggest that the observed QAH state is a new state of matter, namely, a topological $p_x+ip_y$ inter-valley coherent state (TIVC). We point out that the massive Dirac spectrum of the MoTe$_2$ moir\'e bands, together with the Hund's interaction and the Coulomb interactions give rise to this novel QAH state. Through a self-consistent Hartree-Fock analysis, we find a wide range of interaction strengths and displacement fields that the $p_x+ip_y$-pairing phase is energetically favourable. Besides explaining several key features of the experiments, our theory predicts that the order parameter would involve the pairing of electrons and holes with a definite momentum mismatch such that the pairing would generate a new unit cell which is three times the size of the original moir\'e unit cell, due to the order parameter modulations., Comment: 13 pages, 6 figures

Recently, a SU(4) $\pi$-flux Hubbard model on the honeycomb lattice has been proposed to study the spin-orbit excitations of $\alpha$-ZrCl$_3$ [Phys.~Rev.~Lett. 121.097201~(2017)]. Based on this model with a zigzag edge, we show the edge defects can induce edge flat bands that result in a SU(4) edge ferromagnetism. We develop an effective one-dimensional interaction Hamiltonian to study the corresponding SU(4) spin excitations. Remarkably, SU(4) spin excitations of the edge ferromagnet appear as a continuum covering the entire energy region rather than usual magnons. Through further entanglement entropy analysis, we suggest that the continuum consists of fractionalized spin excitations from the disappeared magnons, except for that from the particle-hole Stoner excitations. Moreover, in ribbon systems with finite widths, the disappeared magnons can be restored in the gap formed by the finite-size effect and the optical branch of the restored magnons are found to be topological nontrivial.

Prompting has shown impressive success in enabling large pretrained language models (LMs) to perform diverse NLP tasks, especially when only few downstream data are available. Automatically finding the optimal prompt for each task, however, is challenging. Most existing work resorts to tuning soft prompt (e.g., embeddings) which falls short of interpretability, reusability across LMs, and applicability when gradients are not accessible. Discrete prompt, on the other hand, is difficult to optimize, and is often created by "enumeration (e.g., paraphrasing)-then-selection" heuristics that do not explore the prompt space systematically. This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL). RLPrompt formulates a parameter-efficient policy network that generates the desired discrete prompt after training with reward. To overcome the complexity and stochasticity of reward signals by the large LM environment, we incorporate effective reward stabilization that substantially enhances the training efficiency. RLPrompt is flexibly applicable to different types of LMs, such as masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both classification and generation tasks. Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods. Interestingly, the resulting optimized prompts are often ungrammatical gibberish text; and surprisingly, those gibberish prompts are transferrable between different LMs to retain significant performance, indicating LM prompting may not follow human language patterns., Comment: EMNLP 2022 Camera Ready. Code available at https://github.com/mingkaid/rl-prompt

It is well known that the optical properties of spherical metal particles can be described with the Rayleigh approximation or rigorous Mie theory. But for the single metallic nanorods, a theory well capturing the fundamental resonance and scattering features is still absent. In this study, an analytical model is developed for the metallic nanorod, considering the longitudinally non-uniform conduction current and surface charges. With the circuit parameters deduced from the kinetic and electromagnetic energy of the nanorod, a formula which agrees well with the simulations has been suggested for the resonance wavelength of the fundamental longitudinal mode. Moreover, by introducing the radiative resistance to the circuit theory, the dipole moment, extinction spectrum, and near-field enhancement of the nanorod have been derived analytically and confirmed numerically. The results are important for understanding the optical properties of the metallic nanorods and provide a guideline for designing the light scattering and absorption., Comment: 27 pages, 5 figures

In the framework of a non-relativistic quark model, the mass spectra of the ground and excited states of doubly heavy $\Xi$ and $\Omega$ baryons are calculated. We estimate the mass difference between the $ \Omega $ and corresponding $ \Xi $ baryons as $M_{\Omega}-M_{\Xi}\simeq178 $ MeV for all the states containing $ cc,~bc $, or $ bb $ quarks. A simple form of the universal Isgur-Wise function, as the transition form factor between the doubly heavy baryons, is introduced. Working in the close-to-zero recoil limit, we investigate the $ b \rightarrow c $ semileptonic decay widths and branching fractions of the doubly heavy baryons. The obtained results are compared with other theoretical predictions., Comment: 11 pages, 4 figures and 5 tables