Search

Large language models (LLMs) have achieved remarkable success across various NLP tasks, yet their focus has predominantly been on English due to English-centric pre-training and limited multilingual data. While some multilingual LLMs claim to support for hundreds of languages, models often fail to provide high-quality response for mid- and low-resource languages, leading to imbalanced performance heavily skewed in favor of high-resource languages like English and Chinese. In this paper, we prioritize quality over scaling number of languages, with a focus on multilingual machine translation task, and introduce X-ALMA, a model designed with a commitment to ensuring top-tier performance across 50 diverse languages, regardless of their resource levels. X-ALMA surpasses state-of-the-art open-source multilingual LLMs, such as Aya-101 and Aya-23, in every single translation direction on the FLORES and WMT'23 test datasets according to COMET-22. This is achieved by plug-and-play language-specific module architecture to prevent language conflicts during training and a carefully designed training regimen with novel optimization methods to maximize the translation performance. At the final stage of training regimen, our proposed Adaptive Rejection Preference Optimization (ARPO) surpasses existing preference optimization methods in translation tasks.

If dark matter (DM) accumulates inside neutron stars (NS), it changes their internal structure and causes a shift of the tidal deformability from the value predicted by the dense-matter equation of state (EOS). In principle, this shift could be observable in the gravitational-wave (GW) signal of binary neutron star (BNS) mergers. We investigate the effect of fermionic, non-interacting DM when observing a large number of GW events from DM-admixed BNSs with the precision of the proposed Einstein telescope (ET). Specifically, we study the impact on the recovery of the baryonic EOS and whether DM properties can be constrained. For this purpose, we create event catalogues of BNS mock events with DM fraction up to 1%, from which we reconstruct the posterior uncertainties with the Fisher matrix approach. Using this data, we perform joint Bayesian inference on the baryonic EOS, DM particle mass, and DM particle fraction in each event. Our results reveal that when falsely ignoring DM effects, the EOS posterior is biased towards softer EOSs, though the offset is rather small. Further, we find that within our assumptions of our DM model and population, ET will likely not be able to test the presence of DM in BNSs, even when combining many events and adding Cosmic Explorer (CE) to the next-generation detector network. Likewise, the potential constraints on the DM particle mass will remain weak because of degeneracies with the fraction and EOS., Comment: 15 pages, 6 figures, comments welcome

This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking and supersedes it. The purpose of this report is not to interpret any findings but only provide preliminary results to the participants of the General MT task that may be useful during the writing of the system submission.

Nuclear theory and experiments, alongside astrophysical observations, constrain the equation of state (EOS) of supranuclear-dense matter. Conversely, knowledge of the EOS allows an improved interpretation of nuclear or astrophysical data. In this article, we use several established constraints on the EOS and the new NICER measurement of PSR J0437-4715 to comment on the nature of the primary companion in GW230529 and the companion of PSR J0514-4002E. We find that, with a probability of $\gtrsim 84\%$ and $\gtrsim 68\%$, respectively, both objects are black holes. These likelihoods increase to above $95\%$ when one uses GW170817's remnant as an upper limit on the TOV mass. We also demonstrate that the current knowledge of the EOS substantially disfavors high masses and radii for PSR J0030+0451, inferred recently when combining NICER with XMM-Newton background data and using particular hot-spot models. Finally, we also use our obtained EOS knowledge to comment on measurements of the nuclear symmetry energy, finding that the large value predicted by the PREX-II measurement displays some mild tension with other constraints on the EOS., Comment: 14 pages, 5 figures, comments welcome

This paper investigates the propagation of harmful information in multilingual large language models (LLMs) and evaluates the efficacy of various unlearning methods. We demonstrate that fake information, regardless of the language it is in, once introduced into these models through training data, can spread across different languages, compromising the integrity and reliability of the generated content. Our findings reveal that standard unlearning techniques, which typically focus on English data, are insufficient in mitigating the spread of harmful content in multilingual contexts and could inadvertently reinforce harmful content across languages. We show that only by addressing harmful responses in both English and the original language of the harmful data can we effectively eliminate generations for all languages. This underscores the critical need for comprehensive unlearning strategies that consider the multilingual nature of modern LLMs to enhance their safety and reliability across diverse linguistic landscapes.

Data availability limits the scope of any given task. In machine translation, historical models were incapable of handling longer contexts, so the lack of document-level datasets was less noticeable. Now, despite the emergence of long-sequence methods, we remain within a sentence-level paradigm and without data to adequately approach context-aware machine translation. Most large-scale datasets have been processed through a pipeline that discards document-level metadata. In this work, we reconstruct document-level information for three (ParaCrawl, News Commentary, and Europarl) large datasets in German, French, Spanish, Italian, Polish, and Portuguese (paired with English). We then introduce a document-level filtering technique as an alternative to traditional bitext filtering. We present this filtering with analysis to show that this method prefers context-consistent translations rather than those that may have been sentence-level machine translated. Last we train models on these longer contexts and demonstrate improvement in document-level translation without degradation of sentence-level translation. We release our dataset, ParaDocs, and resulting models as a resource to the community., Comment: ACL 2024 Findings

Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from them. However, there is currently no standard for evaluating the use of LLMs in astronomy. Therefore, we present the experimental design for an evaluation study on how astronomy researchers interact with LLMs. We deploy a Slack chatbot that can answer queries from users via Retrieval-Augmented Generation (RAG); these responses are grounded in astronomy papers from arXiv. We record and anonymize user questions and chatbot answers, user upvotes and downvotes to LLM responses, user feedback to the LLM, and retrieved documents and similarity scores with the query. Our data collection method will enable future dynamic evaluations of LLM tools for astronomy., Comment: 7 pages, 3 figures. Code available at https://github.com/jsalt2024-evaluating-llms-for-astronomy/astro-arxiv-bot

Predicting cancer dynamics under treatment is challenging due to high inter-patient heterogeneity, lack of predictive biomarkers, and sparse and noisy longitudinal data. Mathematical models can summarize cancer dynamics by a few interpretable parameters per patient. Machine learning methods can then be trained to predict the model parameters from baseline covariates, but do not account for uncertainty in the parameter estimates. Instead, hierarchical Bayesian modeling can model the relationship between baseline covariates to longitudinal measurements via mechanistic parameters while accounting for uncertainty in every part of the model. The mapping from baseline covariates to model parameters can be modeled in several ways. A linear mapping simplifies inference but fails to capture nonlinear covariate effects and scale poorly for interaction modeling when the number of covariates is large. In contrast, Bayesian neural networks can potentially discover interactions between covariates automatically, but at a substantial cost in computational complexity. In this work, we develop a hierarchical Bayesian model of subpopulation dynamics that uses baseline covariate information to predict cancer dynamics under treatment, inspired by cancer dynamics in multiple myeloma (MM), where serum M protein is a well-known proxy of tumor burden. As a working example, we apply the model to a simulated dataset and compare its ability to predict M protein trajectories to a model with linear covariate effects. Our results show that the Bayesian neural network covariate effect model predicts cancer dynamics more accurately than a linear covariate effect model when covariate interactions are present. The framework can also be applied to other types of cancer or other time series prediction problems that can be described with a parametric model., Comment: 22 pages, 10 figures

Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about +7 ASR-BLEU for English-Spanish (En-Es) and +2 ASR-BLEU for English-French (En-Fr) translations on the CVSS benchmark, while attaining over 14x speedup for En-Es and 5x speedup for En-Fr translations compared to autoregressive baselines., Comment: Accepted at NeurIPS 2024

The properties of relativistic jets, their interaction with the environment, and their emission of radiation can be self-consistently studied by using particle-in-cell (PIC) numerical simulations. Using three-dimensional (3D), relativistic PIC simulations, we present the first self-consistently calculated synthetic spectra of head-on and off-axis emission from electrons accelerated in cylindrical, relativistic plasma jets containing an initial toroidal magnetic field. The jet particles are initially accelerated during the linear stage of growing plasma instabilities, which are the Weibel instability (WI), kinetic Kelvin-Helmholtz instability (kKHI), and mushroom instability (MI). In the nonlinear stage, these instabilities are dissipated and generate turbulent magnetic fields which accelerate particles further. We calculate the synthetic spectra by tracing a large number of jet electrons in the nonlinear stage, near the jet head where the magnetic fields are turbulent. Our results show the basic properties of jitter-like radiation emitted by relativistic electrons when they travel through a magnetized plasma with the plasma waves driven by kinetic instabilities (WI, kKHI, and MI) growing into the nonlinear regime. At low frequencies, the slope of the spectrum is ~ 0.94, which is similar to that of the jitter radiation. The results are relevant to active galactic nuclei/blazars and gamma-ray burst jet emission and set the ground for future studies on synthetic spectra from relativistic jets., Comment: 9 pages, 8 figures, submitted to MNRAS

While Transformer-based neural machine translation (NMT) is very effective in high-resource settings, many languages lack the necessary large parallel corpora to benefit from it. In the context of low-resource (LR) MT between two closely-related languages, a natural intuition is to seek benefits from structural "shortcuts", such as copying subwords from the source to the target, given that such language pairs often share a considerable number of identical words, cognates, and borrowings. We test Pointer-Generator Networks for this purpose for six language pairs over a variety of resource ranges, and find weak improvements for most settings. However, analysis shows that the model does not show greater improvements for closely-related vs. more distant language pairs, or for lower resource ranges, and that the models do not exhibit the expected usage of the mechanism for shared subwords. Our discussion of the reasons for this behaviour highlights several general challenges for LR NMT, such as modern tokenization strategies, noisy real-world conditions, and linguistic complexities. We call for better scrutiny of linguistically motivated improvements to NMT given the blackbox nature of Transformer models, as well as for a focus on the above problems in the field., Comment: 5 pages, Accepted at Workshop on Insights from Negative Results in NLP (NAACL) 2024

Through continuous progress in nuclear theory and experiment and an increasing number of neutron-star observations, a multitude of information about the equation of state (EOS) for matter at extreme densities is available. Here, we apply these different pieces of data individually to a broad set of physics-agnostic candidate EOSs and analyze the resulting constraints. Specifically, we make use of information from chiral effective field theory, perturbative quantum chromodynamics, as well as data from heavy-ion collisions and the PREX-II and CREX experiments. We also investigate the impact of current mass and radius measurements of neutron stars, such as radio timing measurements of heavy pulsars, NICER data, and other X-ray observations. We augment these by reanalyses of the gravitational-wave (GW) signal GW170817, its associated kilonova AT2017gfo and gamma-ray burst afterglow, the GW signal GW190425, and the GRB211211A afterglow, where we use improved models for the tidal waveform and kilonova light curves. Additionally, we consider the postmerger fate of GW170817 and its consequences for the EOS. This large and diverse set of constraints is eventually combined in numerous ways to explore limits on quantities such as the typical neutron-star radius, the maximum neutron-star mass, the nuclear symmetry-energy parameters, and the speed of sound. Based on the priors from our EOS candidate set, we find the radius of the canonical 1.4 M$_\odot$ neutron star to be $R_{1.4}= 12.27_{-0.94}^{+0.83}$ km and the TOV mass $M_{\rm TOV}= 2.26_{-0.22}^{+0.45}$ M$_\odot$ at 95% credibility, when including those constraints where systematic uncertainties are deemed small. A less conservative approach, combining all the presented constraints, similarly yields $R_{1.4}= 12.20_{-0.50}^{+0.53}$ km and $M_{\rm TOV}= 2.31_{-0.20}^{+0.08}$ M$_\odot$., Comment: 49 pages, 32 figures, webinterface for custom constraint combinations in https://enlil.gw.physik.uni-potsdam.de/eos_constraints

We introduce STAR (Stream Transduction with Anchor Representations), a novel Transformer-based model designed for efficient sequence-to-sequence transduction over streams. STAR dynamically segments input streams to create compressed anchor representations, achieving nearly lossless compression (12x) in Automatic Speech Recognition (ASR) and outperforming existing methods. Moreover, STAR demonstrates superior segmentation and latency-quality trade-offs in simultaneous speech-to-text tasks, optimizing latency, memory footprint, and quality.

As the influence of large language models (LLMs) spans across global communities, their safety challenges in multilingual settings become paramount for alignment research. This paper examines the variations in safety challenges faced by LLMs across different languages and discusses approaches to alleviating such concerns. By comparing how state-of-the-art LLMs respond to the same set of malicious prompts written in higher- vs. lower-resource languages, we observe that (1) LLMs tend to generate unsafe responses much more often when a malicious prompt is written in a lower-resource language, and (2) LLMs tend to generate more irrelevant responses to malicious prompts in lower-resource languages. To understand where the discrepancy can be attributed, we study the effect of instruction tuning with reinforcement learning from human feedback (RLHF) or supervised finetuning (SFT) on the HH-RLHF dataset. Surprisingly, while training with high-resource languages improves model alignment, training in lower-resource languages yields minimal improvement. This suggests that the bottleneck of cross-lingual alignment is rooted in the pretraining stage. Our findings highlight the challenges in cross-lingual LLM safety, and we hope they inform future research in this direction.

Mechanistic learning, the synergistic combination of knowledge-driven and data-driven modeling, is an emerging field. In particular, in mathematical oncology, the application of mathematical modeling to cancer biology and oncology, the use of mechanistic learning is growing. This review aims to capture the current state of the field and provide a perspective on how mechanistic learning may further progress in mathematical oncology. We highlight the synergistic potential of knowledge-driven mechanistic mathematical modeling and data-driven modeling, such as machine and deep learning. We point out similarities and differences regarding model complexity, data requirements, outputs generated, and interpretability of the algorithms and their results. Then, organizing combinations of knowledge- and data-driven modeling into four categories (sequential, parallel, intrinsic, and extrinsic mechanistic learning), we summarize a variety of approaches at the interface between purely data- and knowledge-driven models. Using examples predominantly from oncology, we discuss a range of techniques including physics-informed neural networks, surrogate model learning, and digital twins. We see that mechanistic learning, with its intentional leveraging of the strengths of both knowledge and data-driven modeling, can greatly impact the complex problems of oncology. Given the increasing ubiquity and impact of machine learning, it is critical to incorporate it into the study of mathematical oncology with mechanistic learning providing a path to that end. As the field of mechanistic learning advances, we aim for this review and proposed categorization framework to foster additional collaboration between the data- and knowledge-driven modeling fields. Further collaboration will help address difficult issues in oncology such as limited data availability, requirements of model transparency, and complex input data

We propose a machine learning method to model molecular tensorial quantities, namely the magnetic anisotropy tensor, based on the Gaussian-moment neural-network approach. We demonstrate that the proposed methodology can achieve an accuracy of 0.3--0.4 cm$^{-1}$ and has excellent generalization capability for out-of-sample configurations. Moreover, in combination with machine-learned interatomic potential energies based on Gaussian moments, our approach can be applied to study the dynamic behavior of magnetic anisotropy tensors and provide a unique insight into spin-phonon relaxation.

Translating literary works has perennially stood as an elusive dream in machine translation (MT), a journey steeped in intricate challenges. To foster progress in this domain, we hold a new shared task at WMT 2023, the first edition of the Discourse-Level Literary Translation. First, we (Tencent AI Lab and China Literature Ltd.) release a copyrighted and document-level Chinese-English web novel corpus. Furthermore, we put forth an industry-endorsed criteria to guide human evaluation process. This year, we totally received 14 submissions from 7 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. In addition, our extensive analysis reveals a series of interesting findings on literary and discourse-aware MT. We release data, system outputs, and leaderboard at http://www2.statmt.org/wmt23/literary-translation-task.html., Comment: WMT2023 Discourse-Level Literary Translation Shared Task Overview Paper

Large language models trained primarily in a monolingual setting have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-context learning. However, even though zero-shot translations are relatively good, there remains a discernible gap comparing their performance with the few-shot setting. In this paper, we investigate the factors contributing to this gap and find that this gap can largely be closed (for about 70%) by matching the writing styles of the target corpus. Additionally, we explore potential approaches to enhance zero-shot baselines without the need for parallel demonstration examples, providing valuable insights into how these methods contribute to improving translation metrics.

Text generation models are notoriously vulnerable to errors in the training data. With the wide-spread availability of massive amounts of web-crawled data becoming more commonplace, how can we enhance the robustness of models trained on a massive amount of noisy web-crawled text? In our work, we propose Error Norm Truncation (ENT), a robust enhancement method to the standard training objective that truncates noisy data. Compared to methods that only uses the negative log-likelihood loss to estimate data quality, our method provides a more accurate estimation by considering the distribution of non-target tokens, which is often overlooked by previous work. Through comprehensive experiments across language modeling, machine translation, and text summarization, we show that equipping text generation models with ENT improves generation quality over standard training and previous soft and hard truncation methods. Furthermore, we show that our method improves the robustness of models against two of the most detrimental types of noise in machine translation, resulting in an increase of more than 2 BLEU points over the MLE baseline when up to 50% of noise is added to the data., Comment: ICLR 2024

For a pair of supermassive black holes (SMBHs) in the remnant of a dual galaxy merger, well-known models exist to describe their dynamical evolution until the final coalescence accompanied by the emission of a low-frequency gravitational wave (GW) signal. In this article, we investigate the dynamical evolution of three SMBH triple systems recovered from the ROMULUS25 cosmological simulation to explore common dynamical evolution patterns and assess typical coalescence times. For this purpose, we construct initial conditions from the ROMULUS25 data and perform high-resolution gravitodynamical \N-body simulations. We track the orbital evolution from the galactic inspiral to the formation of hard binaries at sub-parsec separation and use the observed hardening rates to project the time of coalescence. In all cases, the two heaviest black holes form an efficiently hardening binary that merges within fractions of the Hubble time. The lightest SMBH either gets ejected, forms a stable hierarchical triple system with the heavier binary, forms a hardening binary with the previously merged binary's remnant, or remains on a wide galactic orbit. The coalescence times of the lighter black holes are thus significantly longer than for the heavier binary, as they experience lower dynamical friction and stellar hardening rates. We observe the formation of hierarchical triples when the density profile of the galactic nucleus is sufficiently steep., Comment: 11 pages, 6 figures, 4 tables, accepted for publication in A&A

Incorporating language-specific (LS) modules is a proven method to boost performance in multilingual machine translation. This approach bears similarity to Mixture-of-Experts (MoE) because it does not inflate FLOPs. However, the scalability of this approach to hundreds of languages (experts) tends to be unmanageable due to the prohibitive number of parameters introduced by full-rank matrices in fully-connected layers. In this work, we introduce the Language-Specific Matrix Synthesis (LMS) method. This approach constructs LS modules by generating low-rank matrices from two significantly smaller matrices to approximate the full-rank matrix. Furthermore, we condense multilingual knowledge from multiple LS modules into a single shared module with the Fuse Distillation (FD) technique to improve the efficiency of inference and model serialization. We show that our LMS method significantly outperforms previous LS methods and MoE methods with the same amount of extra parameters, e.g., 1.73 BLEU points over the Switch Transformer on many-to-many multilingual machine translation. Importantly, LMS is able to have comparable translation performance with much fewer parameters., Comment: Accepted at the main conference of EMNLP 2023

We introduce and demonstrate how to effectively train multilingual machine translation models with pixel representations. We experiment with two different data settings with a variety of language and script coverage, demonstrating improved performance compared to subword embeddings. We explore various properties of pixel representations such as parameter sharing within and across scripts to better understand where they lead to positive transfer. We observe that these properties not only enable seamless cross-lingual transfer to unseen scripts, but make pixel representations more data-efficient than alternatives such as vocabulary expansion. We hope this work contributes to more extensible multilingual models for all languages and scripts., Comment: EMNLP 2023

Single-atom imaging resolution of many-body quantum systems in optical lattices is routinely achieved with quantum-gas microscopes. Key to their great versatility as quantum simulators is the ability to use engineered light potentials at the microscopic level. Here, we employ dynamically varying microscopic light potentials in a quantum-gas microscope to study commensurate and incommensurate 1D systems of interacting bosonic Rb atoms. Such incommensurate systems are analogous to doped insulating states that exhibit atom transport and compressibility. Initially, a commensurate system with unit filling and fixed atom number is prepared between two potential barriers. We deterministically create an incommensurate system by dynamically changing the position of the barriers such that the number of available lattice sites is reduced while retaining the atom number. Our systems are characterised by measuring the distribution of particles and holes as a function of the lattice filling, and interaction strength, and we probe the particle mobility by applying a bias potential. Our work provides the foundation for preparation of low-entropy states with controlled filling in optical-lattice experiments., Comment: 12 pages, 10 figures

We present a validation of a three-dimensional, two-fluid simulation of plasma turbulence in the TJ-K stellarator, a low temperature plasma experiment ideally suited for turbulence measurements. The simulation is carried out by the GBS code, recently adapted to simulate 3D magnetic fields. The comparison shows that GBS retrieves the main turbulence properties observed in the device, namely the fact that transport is dominated by fluctuations with low poloidal mode number. The poloidal dependence of the radial $\text{E}\times\text{B}$ turbulent flux is compared on a poloidal plane with elliptical flux surfaces, where a very good agreement between experiment and simulation is observed, and on another with triangular flux surfaces, which shows a poorer comparison. The fluctuation levels in both cases are underestimated in the simulations. The equilibrium density profile is well retrieved by the simulation, while the electron temperature and the electrostatic potential profiles, which are very sensitive to the strength and localization of the sources, do not agree well with the experimental measurements.

Tumor heterogeneity is a complex and widely recognized trait that poses significant challenges in developing effective cancer therapies. In particular, many tumors harbor a variety of subpopulations with distinct therapeutic response characteristics. Characterizing this heterogeneity by determining the subpopulation structure within a tumor enables more precise and successful treatment strategies. In our prior work, we developed PhenoPop, a computational framework for unravelling the drug-response subpopulation structure within a tumor from bulk high-throughput drug screening data. However, the deterministic nature of the underlying models driving PhenoPop restricts the model fit and the information it can extract from the data. As an advancement, we propose a stochastic model based on the linear birth-death process to address this limitation. Our model can formulate a dynamic variance along the horizon of the experiment so that the model uses more information from the data to provide a more robust estimation. In addition, the newly proposed model can be readily adapted to situations where the experimental data exhibits a positive time correlation. We test our model on simulated data (in silico) and experimental data (in vitro), which supports our argument about its advantages., Comment: 36 pages, 14 figures. v2: 1. Rearranged paper and figures. 2. Modified the figures to make them easier to access; results unchanged. 3. Revised the argument in section 3 and section 4; results unchanged. 4. Revised the abstract

Estrogen receptor positive breast cancer is frequently treated with anti-hormonal treatment such as aromatase inhibitors (AI). Interestingly, a high body mass index has been shown to have a negative impact on AI efficacy, most likely due to disturbances in steroid metabolism and adipokine production. Here, we propose a mathematical model based on a system of ordinary differential equations to investigate the effect of high-fat diet on tumor growth. We inform the model with data from mouse experiments, where the animals are fed with high-fat or control (normal) diet. By incorporating AI treatment with drug resistance into the model and by solving optimal control problems we found differential responses for control and high-fat diet. To the best of our knowledge, this is the first attempt to model optimal anti-hormonal treatment for breast cancer in the presence of drug resistance. Our results underline the importance of considering high-fat diet and obesity as factors influencing clinical outcomes during anti-hormonal therapies in breast cancer patients., Comment: 47 pages, 22 figures