Search

Physics-informed machine learning (PIML) has emerged as a promising alternative to classical methods for predicting dynamical systems, offering faster and more generalizable solutions. However, existing models, including recurrent neural networks (RNNs), transformers, and neural operators, face challenges such as long-time integration, long-range dependencies, chaotic dynamics, and extrapolation, to name a few. To this end, this paper introduces state-space models implemented in Mamba for accurate and efficient dynamical system operator learning. Mamba addresses the limitations of existing architectures by dynamically capturing long-range dependencies and enhancing computational efficiency through reparameterization techniques. To extensively test Mamba and compare against another 11 baselines, we introduce several strict extrapolation testbeds that go beyond the standard interpolation benchmarks. We demonstrate Mamba's superior performance in both interpolation and challenging extrapolation tasks. Mamba consistently ranks among the top models while maintaining the lowest computational cost and exceptional extrapolation capabilities. Moreover, we demonstrate the good performance of Mamba for a real-world application in quantitative systems pharmacology for assessing the efficacy of drugs in tumor growth under limited data scenarios. Taken together, our findings highlight Mamba's potential as a powerful tool for advancing scientific machine learning in dynamical systems modeling. (The code will be available at https://github.com/zheyuanhu01/State_Space_Model_Neural_Operator upon acceptance.), Comment: 34 pages

In this paper we explore the potential of utilizing Virtual Reality (VR) as a therapeutic tool for supporting individuals in the LGBTQ+ community, who often face elevated risks of mental health issues. Specifically, we investigated the effectiveness of using pre-existing avatars compared to allowing individuals to create their own avatars through a website, and their experience in a VR space when using these avatars. We conducted a user study (n=10) measuring heart rate variability (HRV) and gathering subjective feedback through semi-structured interviews conducted in VR. Avatar creation was facilitated using an online platform, and conversations took place within a two-user VR space developed in a commercially available VR application. Our findings suggest that users significantly prefer creating their own avatars in the context of therapy sessions, and while there was no statistically significant difference, there was a consistent trend of enhanced physiological response when using self-made avatars in VR. This study provides initial empirical support for the importance of custom avatar creation in utilizing VR for therapy within the LGBTQ+ community.

We discuss the effect of the quark anomalous magnetic moment (AMM) on the neutral dense quark matter under magnetic fields based on the Nambu-Jona-Lasinio (NJL) model at finite baryon density. To address its correlation with the chiral symmetry, we consider a simplified situation: the model includes the two-quark flavors under constant magnetic fields, and incorporates the effective interaction of the quark AMM linked to the spontaneous chiral symmetry breaking. We then examine the equation of state (EoS) in cases with and without magnetization for anatomizing the thermodynamic quantities. Without the magnetization, a small magnetic field stiffens the EoS, but with increasing the magnetic field, the EoS tends to soften. The stiffness of the EoS is found to be influenced by the magnetic effect on the critical chemical potential of the chiral phase transition and the quark number density at this critical point. As a result, the mass and radius of the neutral dense quark matter increase with the small magnetic field but turn to decrease as the magnetic field further increases. By including the quark AMM, the critical chemical potential is decreased and the quark number density takes a smaller value. Thus, for the stronger magnetic fields, the quark AMM suppresses the softening effect of the magnetic field on the EoS, leading to increased mass and radius compared to when the quark AMM is absent. In contrast, for the small magnetic field, the contribution of the quark AMM to the EoS is marginal. When the magnetization is taken into account, the magnetic effect on the stiffness of the EoS is overshadowed by the contribution of the magnetization, and the significance of the quark AMM also becomes invisible in the mass-radius relation., Comment: 16 pages, 7 figures

Increase in data, size, or compute can lead to sudden learning of specific capabilities by a neural network -- a phenomenon often called "emergence". Beyond scientific understanding, establishing the causal factors underlying such emergent capabilities is crucial to enable risk regulation frameworks for AI. In this work, we seek inspiration from study of emergent properties in other fields and propose a phenomenological definition for the concept in the context of neural networks. Our definition implicates the acquisition of specific structures underlying the data-generating process as a cause of sudden performance growth for specific, narrower tasks. We empirically investigate this definition by proposing an experimental system grounded in a context-sensitive formal language and find that Transformers trained to perform tasks on top of strings from this language indeed exhibit emergent capabilities. Specifically, we show that once the language's underlying grammar and context-sensitivity inducing structures are learned by the model, performance on narrower tasks suddenly begins to improve. We then analogize our network's learning dynamics with the process of percolation on a bipartite graph, establishing a formal phase transition model that predicts the shift in the point of emergence observed in experiment when changing the data structure. Overall, our experimental and theoretical frameworks yield a step towards better defining, characterizing, and predicting emergence in neural networks., Comment: Preprint

The dependence of the final fate of supermassive star (SMS) cores on their mass and angular momentum is studied with simple modeling. SMS cores in the hydrogen burning phase encounter the general relativistic instability during the stellar evolution if the mass is larger than $\sim 3 \times 10^4M_\odot$. Spherical SMS cores in the helium burning phase encounter the general relativistic instability prior to the onset of the electron-positron pair instability if the mass is larger than $\sim 1\times 10^4M_\odot$. For rapidly rotating SMS cores, these values for the threshold mass are enhanced by up to a factor of $\sim 5$, and thus, for SMSs with mass smaller than $\sim 10^4M_\odot$ the collapse is triggered by the pair-instability, irrespective of the rotation. After the onset of the general relativistic instability, SMS cores in the hydrogen burning phase with reasonable metallicity are likely to collapse to a black hole irrespective of the degree of rotation, whereas the SMS cores in the helium burning phase could explode via nuclear burning with no black hole formation, as previous works demonstrate., Comment: 13 pages, 6 figures. Submitted to ApJ

We perform new general relativistic hydrodynamics simulations for collapses of rotating supermassive star cores with an approximate nuclear burning up to carbon and a detailed equation of state. For all the models we investigate, the energy generation by nuclear burning plays only a minor role, leading to the formation of a black hole without a nuclear-powered explosion. For rotating models, however, the stellar explosion associated with shock heating is driven from a torus, which forms after the black hole formation. The explosion energy is up to $10^{-4}$ of the mass energy of the supermassive star cores ($\sim 10^{55}-10^{56}$ erg). We find that, even if we increase the rotational angular momentum of the progenitor, the ejecta mass saturates at $\sim 1$\% of the total mass of the initial stellar core. The average ejecta velocity also saturates at $\approx 20\%$ of the speed of light. As a result, the ejecta kinetic energy is approximately proportional to the initial mass of the supermassive star core for the rapidly rotating case. We also perform viscous hydrodynamics simulations for exploring the evolution of the remnant torus. Although the viscous heating drives an outflow from the torus, we find that its effect is subdominant in terms of the kinetic energy because of the small velocity ($\approx 0.07c$) of the ejecta component., Comment: 21 pages, 11 figures, submitted to ApJ

We present the first systematic evaluation examining format bias in performance of large language models (LLMs). Our approach distinguishes between two categories of an evaluation metric under format constraints to reliably and accurately assess performance: one measures performance when format constraints are adhered to, while the other evaluates performance regardless of constraint adherence. We then define a metric for measuring the format bias of LLMs and establish effective strategies to reduce it. Subsequently, we present our empirical format bias evaluation spanning four commonly used categories -- multiple-choice question-answer, wrapping, list, and mapping -- covering 15 widely-used formats. Our evaluation on eight generation tasks uncovers significant format bias across state-of-the-art LLMs. We further discover that improving the format-instruction following capabilities of LLMs across formats potentially reduces format bias. Based on our evaluation findings, we study prompting and fine-tuning with synthesized format data techniques to mitigate format bias. Our methods successfully reduce the variance in ChatGPT's performance among wrapping formats from 235.33 to 0.71 (%$^2$).

This paper introduces an active learning (AL) framework for anomalous sound detection (ASD) in machine condition monitoring system. Typically, ASD models are trained solely on normal samples due to the scarcity of anomalous data, leading to decreased accuracy for unseen samples during inference. AL is a promising solution to solve this problem by enabling the model to learn new concepts more effectively with fewer labeled examples, thus reducing manual annotation efforts. However, its effectiveness in ASD remains unexplored. To minimize update costs and time, our proposed method focuses on updating the scoring backend of ASD system without retraining the neural network model. Experimental results on the DCASE 2023 Challenge Task 2 dataset confirm that our AL framework significantly improves ASD performance even with low labeling budgets. Moreover, our proposed sampling strategy outperforms other baselines in terms of the partial area under the receiver operating characteristic score., Comment: Accepted as a conference paper in INTERSPEECH 2024

We publicly release OpenUAS, a dataset of area embeddings based on urban usage patterns, including embeddings for over 1.3 million 50-meter square meshes covering a total area of 3,300 square kilometers. This dataset is valuable for analyzing area functions in fields such as market analysis, urban planning, transportation infrastructure, and infection prediction. It captures the characteristics of each area in the city, such as office districts and residential areas, by employing an area embedding technique that utilizes location information typically obtained by GPS. Numerous area embedding techniques have been proposed, and while the public release of such embedding datasets is technically feasible, it has not been realized. One of the obstacles has been the integration of data from different cities and periods into a unified space without sharing raw location data. We address this issue by developing an anchoring method that establishes anchors within a shared embedding space. We publicly release this anchor dataset along with area embedding datasets from several periods in eight major Japanese cities. This dataset allows users to analyze urban usage patterns in Japanese cities and embed their urban dataset into the same embedding space using the anchoring method. Our key contributions include the development of the anchoring method, releasing area embedding datasets for Japanese cities, and providing tools for effective data utilization.

We revisit one-loop corrections to the power spectrum of curvature perturbations $\zeta$ in an inflationary scenario containing a transient ultra-slow-roll (USR) period. In Ref.[1], it was argued that one-loop corrections to the power spectrum of $\zeta$ can be larger than the tree-level one within the parameter region generating the seeds of primordial black holes during the USR epoch, which implies the breakdown of perturbation theory. We prove that this is not the case by using a master formula for one-loop corrections to the power spectrum obtained in Ref.[2]. We derive the same formula within the path-integral formalism, which is simpler than the original derivation in [2]. To show the smallness of one-loop corrections, the consistency relations and the effective constancy of tree-level mode functions of $\zeta$ for super-Hubble modes play essential roles, with which the master formula gives a simple expression for one-loop corrections. For concreteness, we provide a reduced set of interactions including the leading-order one, while establishing the consistency relations in a self-consistent manner. We also show how the consistency relations of various operators hold explicitly, which plays a key role in proving the absence of large one-loop corrections., Comment: 46 pages, 11 figures

Dynamical phase transitions (DPTs) arise from qualitative changes in the long-time behavior of stochastic trajectories, often observed in systems with kinetic constraints or driven out of equilibrium. Here we demonstrate that first-order DPTs can occur even in the large deviations of a single Brownian particle without drift, but only when the system's dimensionality exceeds four. These DPTs are accompanied by temporal phase separations in the trajectories and exhibit dimension-dependent order due to the threshold behavior for bound state formation in Schr\"{o}dinger operators. We also discover second-order DPTs in one-dimensional Brownian motion, characterized by universal exponents in the rate function of dynamical observables. Our results establish a novel framework linking classical DPTs to quantum phase transitions., Comment: 6 pages, 3 figures. arXiv admin note: substantial text overlap with arXiv:2407.14090

This paper investigates the relationship between extensional and shear viscosity of low-viscosity power-law fluids. We showed the first experimental evidence of the conditions satisfying the same power exponents for extensional and shear viscosity, as indicated by the Carreau model. The extensional and shear viscosity are respectively measured by capillary breakup extensional rheometry dripping-onto-substrate (CaBER-DoS) and by a shear rheometer for various Ohnesorge number Oh. The viscosity ranges measured were about O(10^0) to O(10^4) mPas for shear viscosity and O(10^1) to O(10^3) mPas for extensional viscosity. Our experimental results show that, at least for the range of Oh > 1, the power-law expression for the liquid filament radius, extensional viscosity, and shear viscosity holds, even for low-viscosity fluids.

Self-propulsion is a quintessential aspect of biological systems, which can induce nonequilibrium phenomena that have no counterparts in passive systems. Motivated by biophysical interest together with recent advances in experimental techniques, active matter has been a rapidly developing field in physics. Meanwhile, over the past few decades, topology has played a crucial role to understand certain robust properties appearing in condensed matter systems. For instance, the nontrivial topology of band structures leads to the notion of topological insulators, where one can find robust gapless edge modes protected by the bulk band topology. We here review recent progress in an interdisciplinary area of research at the intersection of these two fields. Specifically, we give brief introductions to active matter and band topology in Hermitian systems, and then explain how the notion of band topology can be extended to nonequilibrium (and thus non-Hermitian) systems including active matter. We review recent studies that have demonstrated the intimate connections between active matter and topological materials, where exotic topological phenomena that are unfeasible in passive systems have been found. A possible extension of the band topology to nonlinear systems is also briefly discussed. Active matter can thus provide an ideal playground to explore topological phenomena in qualitatively new realms beyond conservative linear systems., Comment: 37 pages, 23 figures

We study the dynamical phase transitions (DPTs) appearing for a single Brownian particle without drift. We first explore how first-order DPTs in large deviations can be found even for a single Brownian particle without any force upon raising the dimension to higher than four. The DPTs accompany temporal phase separations in their dynamical paths, which we numerically confirm by fitting to scaling functions. We next investigate how second-order DPTs can appear in one-dimensional free Brownian motion by choosing the observable, which essentially captures the localization transition of the trajectories. We discuss and confirm that the DPTs predicted for high dimensions can also be found when considering many Brownian particles at lower dimensions., Comment: 16 pages, 8 figures

We present a study of the molecular gas in five closely-spaced ($R_{\perp}<20$ kpc) dual quasars ($L_{\rm bol}\gtrsim10^{44}~\mathrm{erg~s}^{-1}$) at redshifts $0.4

It is pointed out that millimeter-wave vortex may contribute an efficient plasma heating since it was found that the millimeter-wave vortex can propagate in magnetized plasma even in which the normal plane wave is in cut-off condition. Then, it was assumed that the vortex field was the Laguerre-Gaussian (L-G) mode which is free-space solution, but the generation and stable propagation of the L-G mode vortex are not easy in the millimeter frequency range. On the other hand, it is known that millimeter-wave hybrid mode of cylindrical corrugated waveguide has also vortex property. In this paper, we investigate propagation characteristics of millimeter-wave vortex of a hybrid mode of cylindrical corrugated waveguide in the magnetized plasma by using three dimensional numerical simulations with finite-difference time-domain (FDTD) method., Comment: 9 pages, 5 figures

When LLMs are deployed in sensitive, human-facing settings, it is crucial that they do not output unsafe, biased, or privacy-violating outputs. For this reason, models are both trained and instructed to refuse to answer unsafe prompts such as "Tell me how to build a bomb." We find that, despite these safeguards, it is possible to break model defenses simply by appending a space to the end of a model's input. In a study of eight open-source models, we demonstrate that this acts as a strong enough attack to cause the majority of models to generate harmful outputs with very high success rates. We examine the causes of this behavior, finding that the contexts in which single spaces occur in tokenized training data encourage models to generate lists when prompted, overriding training signals to refuse to answer unsafe requests. Our findings underscore the fragile state of current model alignment and promote the importance of developing more robust alignment methods. Code and data will be available at https://github.com/hannah-aught/space_attack., Comment: 8 pages, 6 figures

We introduce a defense against adversarial attacks on LLMs utilizing self-evaluation. Our method requires no model fine-tuning, instead using pre-trained models to evaluate the inputs and outputs of a generator model, significantly reducing the cost of implementation in comparison to other, finetuning-based methods. Our method can significantly reduce the attack success rate of attacks on both open and closed-source LLMs, beyond the reductions demonstrated by Llama-Guard2 and commonly used content moderation APIs. We present an analysis of the effectiveness of our method, including attempts to attack the evaluator in various settings, demonstrating that it is also more resilient to attacks than existing methods. Code and data will be made available at https://github.com/Linlt-leon/self-eval., Comment: 8 pages, 7 figures

The molecular dynamics of regioregulated poly(3-hexylthiophene) P3HT is investigated using muon spin relaxation ($\mu$SR). The response of the $\mu$SR spectra to a longitudinal magnetic field ($B_{\rm LF}$, parallel to the initial muon spin direction) indicates that the implanted muons form both muonated radicals localized on the thiophene ring and diamagnetic states with comparable yields. Moreover, the unpaired electron in the radical undergoes hyperfine interactions with muon bound to thiophene and with neighboring protons, whose fluctuations can serve as a measure for the molecular dynamics. The $B_{\rm LF}$ dependence of the longitudinal muon spin relaxation rate ($1/T_{1\mu}$) measured in detail at several temperatures is found to be well reproduced by the spectral density function $J(\omega)$ derived from the local susceptibility that incorporates the Havriliak-Negami (H-N) function used in the analysis of dielectric relaxation, $\chi(\omega)\propto1/[1-i(\omega/\tilde{\nu})^\delta]^\gamma$ (where $\tilde{\nu}$ is the mean fluctuation rate, and $0<\gamma, \delta\le1$). The magnitude of $\tilde{\nu}$ and its temperature dependence deduced from the analysis of $1/T_{1\mu}$ are found to be consistent with the motion of hexyl chains and thiophene rings suggested by $^{13}$C-NMR. The present result marks a methodological milestone in the application of $\mu$SR to the dynamics of complex systems with coexisting fluctuations over a wide range of time scales, such as polymers., Comment: 9 pages, 5 figures

The recent Gamma-Ray Burst (GRB) GRB 211211A provides the earliest ($\sim 5$ h) data of a kilonova (KN) event, displaying bright ($\sim10^{42}$ erg s$^{-1}$) and blue early emission. Previously, this KN has been explained using simplistic multi-component fitting methods. Here, in order to understand the physical origin of the KN emission in GRB 211211A, we employ an analytic multi-zone model for r-process powered KN. We find that r-process powered KN models alone cannot explain the fast temporal evolution and the spectral energy distribution (SED) of the observed emission. Specifically, i) r-process models require high ejecta mass to match early luminosity, which overpredicts late-time emission, while ii) red KN models that reproduce late emission underpredict early luminosity. We propose an alternative scenario involving early contributions from the GRB central engine via a late low-power jet, consistent with plateau emission in short GRBs and GeV emission detected by Fermi-LAT at $\sim10^4$ s after GRB 211211A. Such late central engine activity, with an energy budget of $\sim \text{a few }\%$ of that of the prompt jet, combined with a single red-KN ejecta component, can naturally explain the light curve and SED of the observed emission; with the late-jet -- ejecta interaction reproducing the early blue emission and r-process heating reproducing the late red emission. This supports claims that late low-power engine activity after prompt emission may be common. We encourage very early follow-up observations of future nearby GRBs, and compact binary merger events, to reveal more about the central engine of GRBs and r-process events., Comment: 18 pages, 6 figures, and 1 tables. Accepted for publication in the Astrophysical Journal Letters

Bi-level optimization (BO) has become a fundamental mathematical framework for addressing hierarchical machine learning problems. As deep learning models continue to grow in size, the demand for scalable bi-level optimization solutions has become increasingly critical. Traditional gradient-based bi-level optimization algorithms, due to their inherent characteristics, are ill-suited to meet the demands of large-scale applications. In this paper, we introduce $\textbf{F}$orward $\textbf{G}$radient $\textbf{U}$nrolling with $\textbf{F}$orward $\textbf{F}$radient, abbreviated as $(\textbf{FG})^2\textbf{U}$, which achieves an unbiased stochastic approximation of the meta gradient for bi-level optimization. $(\text{FG})^2\text{U}$ circumvents the memory and approximation issues associated with classical bi-level optimization approaches, and delivers significantly more accurate gradient estimates than existing large-scale bi-level optimization approaches. Additionally, $(\text{FG})^2\text{U}$ is inherently designed to support parallel computing, enabling it to effectively leverage large-scale distributed computing systems to achieve significant computational efficiency. In practice, $(\text{FG})^2\text{U}$ and other methods can be strategically placed at different stages of the training process to achieve a more cost-effective two-phase paradigm. Further, $(\text{FG})^2\text{U}$ is easy to implement within popular deep learning frameworks, and can be conveniently adapted to address more challenging zeroth-order bi-level optimization scenarios. We provide a thorough convergence analysis and a comprehensive practical discussion for $(\text{FG})^2\text{U}$, complemented by extensive empirical evaluations, showcasing its superior performance in diverse large-scale bi-level optimization tasks.

Fractional and tempered fractional partial differential equations (PDEs) are effective models of long-range interactions, anomalous diffusion, and non-local effects. Traditional numerical methods for these problems are mesh-based, thus struggling with the curse of dimensionality (CoD). Physics-informed neural networks (PINNs) offer a promising solution due to their universal approximation, generalization ability, and mesh-free training. In principle, Monte Carlo fractional PINN (MC-fPINN) estimates fractional derivatives using Monte Carlo methods and thus could lift CoD. However, this may cause significant variance and errors, hence affecting convergence; in addition, MC-fPINN is sensitive to hyperparameters. In general, numerical methods and specifically PINNs for tempered fractional PDEs are under-developed. Herein, we extend MC-fPINN to tempered fractional PDEs to address these issues, resulting in the Monte Carlo tempered fractional PINN (MC-tfPINN). To reduce possible high variance and errors from Monte Carlo sampling, we replace the one-dimensional (1D) Monte Carlo with 1D Gaussian quadrature, applicable to both MC-fPINN and MC-tfPINN. We validate our methods on various forward and inverse problems of fractional and tempered fractional PDEs, scaling up to 100,000 dimensions. Our improved MC-fPINN/MC-tfPINN using quadrature consistently outperforms the original versions in accuracy and convergence speed in very high dimensions., Comment: 15 pages

We introduce an innovative approach for solving high-dimensional Fokker-Planck-L\'evy (FPL) equations in modeling non-Brownian processes across disciplines such as physics, finance, and ecology. We utilize a fractional score function and Physical-informed neural networks (PINN) to lift the curse of dimensionality (CoD) and alleviate numerical overflow from exponentially decaying solutions with dimensions. The introduction of a fractional score function allows us to transform the FPL equation into a second-order partial differential equation without fractional Laplacian and thus can be readily solved with standard physics-informed neural networks (PINNs). We propose two methods to obtain a fractional score function: fractional score matching (FSM) and score-fPINN for fitting the fractional score function. While FSM is more cost-effective, it relies on known conditional distributions. On the other hand, score-fPINN is independent of specific stochastic differential equations (SDEs) but requires evaluating the PINN model's derivatives, which may be more costly. We conduct our experiments on various SDEs and demonstrate numerical stability and effectiveness of our method in dealing with high-dimensional problems, marking a significant advancement in addressing the CoD in FPL equations., Comment: 16 pages, 1 figure

We present the task description of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2024 Challenge Task 2: First-shot unsupervised anomalous sound detection (ASD) for machine condition monitoring. Continuing from last year's DCASE 2023 Challenge Task 2, we organize the task as a first-shot problem under domain generalization required settings. The main goal of the first-shot problem is to enable rapid deployment of ASD systems for new kinds of machines without the need for machine-specific hyperparameter tunings. This problem setting was realized by (1) giving only one section for each machine type and (2) having completely different machine types for the development and evaluation datasets. For the DCASE 2024 Challenge Task 2, data of completely new machine types were newly collected and provided as the evaluation dataset. In addition, attribute information such as the machine operation conditions were concealed for several machine types to mimic situations where such information are unavailable. We will add challenge results and analysis of the submissions after the challenge submission deadline., Comment: anomaly detection, acoustic condition monitoring, domain shift, first-shot problem, DCASE Challenge. arXiv admin note: text overlap with arXiv:2305.07828

Despite the recent advancements in offline RL, no unified algorithm could achieve superior performance across a broad range of tasks. Offline \textit{value function learning}, in particular, struggles with sparse-reward, long-horizon tasks due to the difficulty of solving credit assignment and extrapolation errors that accumulates as the horizon of the task grows.~On the other hand, models that can perform well in long-horizon tasks are designed specifically for goal-conditioned tasks, which commonly perform worse than value function learning methods on short-horizon, dense-reward scenarios. To bridge this gap, we propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals. Our experimental results suggest that PlanDQ can achieve superior or competitive performance on D4RL continuous control benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks.

In-Context Learning (ICL) has been a powerful emergent property of large language models that has attracted increasing attention in recent years. In contrast to regular gradient-based learning, ICL is highly interpretable and does not require parameter updates. In this paper, we show that, for linearized transformer networks, ICL can be made explicit and permanent through the inclusion of bias terms. We mathematically demonstrate the equivalence between a model with ICL demonstration prompts and the same model with the additional bias terms. Our algorithm (ICLCA) allows for exact conversion in an inexpensive manner. Existing methods are not exact and require expensive parameter updates. We demonstrate the efficacy of our approach through experiments that show the exact incorporation of ICL tokens into a linear transformer. We further suggest how our method can be adapted to achieve cheap approximate conversion of ICL tokens, even in regular transformer networks that are not linearized. Our experiments on GPT-2 show that, even though the conversion is only approximate, the model still gains valuable context from the included bias terms., Comment: Accepted to ICML 2024

Overparametrized transformer networks are the state-of-the-art architecture for Large Language Models (LLMs). However, such models contain billions of parameters making large compute a necessity, while raising environmental concerns. To address these issues, we propose FinerCut, a new form of fine-grained layer pruning, which in contrast to prior work at the transformer block level, considers all self-attention and feed-forward network (FFN) layers within blocks as individual pruning candidates. FinerCut prunes layers whose removal causes minimal alternation to the model's output -- contributing to a new, lean, interpretable, and task-agnostic pruning method. Tested across 9 benchmarks, our approach retains 90% performance of Llama3-8B with 25% layers removed, and 95% performance of Llama3-70B with 30% layers removed, all without fine-tuning or post-pruning reconstruction. Strikingly, we observe intriguing results with FinerCut: 42% (34 out of 80) of the self-attention layers in Llama3-70B can be removed while preserving 99% of its performance -- without additional fine-tuning after removal. Moreover, FinerCut provides a tool to inspect the types and locations of pruned layers, allowing to observe interesting pruning behaviors. For instance, we observe a preference for pruning self-attention layers, often at deeper consecutive decoder layers. We hope our insights inspire future efficient LLM architecture designs., Comment: 22 pages

Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs). Developing effective protection against many modes of attack prompts requires discovering diverse attacks. Automated red-teaming typically uses reinforcement learning to fine-tune an attacker language model to generate prompts that elicit undesirable responses from a target LLM, as measured, for example, by an auxiliary toxicity classifier. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. As a flexible and probabilistically principled alternative, we propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts. We find that the attacks generated by our method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer well between target LLMs. Finally, we demonstrate that models safety-tuned using a dataset of red-teaming prompts generated by our method are robust to attacks from other RL-based red-teaming approaches.

In [13], for any continuous self-map of a compact metric space with s-limit shadowing, by using a $G_\delta$-partition of the phase space, a global description of Li-Yorke type chaos (with respect to several particular Furstenberg families) is obtained. In this paper, we complement the results in [13] by some results concerning the s-limit shadowing and general Furstenberg families. An example is also given to illustrate the results., Comment: 18 pages. arXiv admin note: substantial text overlap with arXiv:2308.11163

Analogous to the charged electron-electron pair condensation in superconductors, an excitonic insulator (EI) represents Fermi surface instability due to spontaneous formation and condensation of charge-neutral electron-hole pair (exciton). Unlike in superconductors, however, the charge-neutral nature of exciton makes probing emergent EI phase via macroscopic physical properties generally difficult. Here, we propose a van der Waals coupled antiferromagnetic semiconductor GdGaI (GGI) as a new material category leading to emergent multi-q magnet intertwined with spontaneous exciton formation/condensation. Before excitonic band hybridization, a simple picture for the parent electronic state consists of electron (Gd-derived 5d) and hole (Ga-derived 4p) delocalized bands, together with Gd-derived 4f localized antiferromagnets with S = 7/2 classical nature. Through intra Gd atom 4f-5d Hund's coupling, a notable finding is the emergent minimum length scale (2a) Skyrmion-like spin texture resulting from spontaneous condensation/formation of spin-polarized exciton with BCS-BEC crossover phenomenology. This discovered platform is promising for realizing valuable quantum matter on the nanoscale; our finding will provide significant insight into designing the atomic scale topological magnetism out of itinerant systems., Comment: 23 pages, 10 figures

Molecule-text modeling, which aims to facilitate molecule-relevant tasks with a textual interface and textual knowledge, is an emerging research direction. Beyond single molecules, studying reaction-text modeling holds promise for helping the synthesis of new materials and drugs. However, previous works mostly neglect reaction-text modeling: they primarily focus on modeling individual molecule-text pairs or learning chemical reactions without texts in context. Additionally, one key task of reaction-text modeling -- experimental procedure prediction -- is less explored due to the absence of an open-source dataset. The task is to predict step-by-step actions of conducting chemical experiments and is crucial to automating chemical synthesis. To resolve the challenges above, we propose a new pretraining method, ReactXT, for reaction-text modeling, and a new dataset, OpenExp, for experimental procedure prediction. Specifically, ReactXT features three types of input contexts to incrementally pretrain LMs. Each of the three input contexts corresponds to a pretraining task to improve the text-based understanding of either reactions or single molecules. ReactXT demonstrates consistent improvements in experimental procedure prediction and molecule captioning and offers competitive results in retrosynthesis. Our code is available at https://github.com/syr-cn/ReactXT., Comment: ACL 2024 Findings, 9 pages

Language Models (LMs) excel in understanding textual descriptions of proteins, as evident in biomedical question-answering tasks. However, their capability falters with raw protein data, such as amino acid sequences, due to a deficit in pretraining on such data. Conversely, Protein Language Models (PLMs) can understand and convert protein data into high-quality representations, but struggle to process texts. To address their limitations, we introduce ProtT3, a framework for Protein-to-Text Generation for Text-based Protein Understanding. ProtT3 empowers an LM to understand protein sequences of amino acids by incorporating a PLM as its protein understanding module, enabling effective protein-to-text generation. This collaboration between PLM and LM is facilitated by a cross-modal projector (i.e., Q-Former) that bridges the modality gap between the PLM's representation space and the LM's input space. Unlike previous studies focusing on protein property prediction and protein-text retrieval, we delve into the largely unexplored field of protein-to-text generation. To facilitate comprehensive benchmarks and promote future research, we establish quantitative evaluations for protein-text modeling tasks, including protein captioning, protein question-answering, and protein-text retrieval. Our experiments show that ProtT3 substantially surpasses current baselines, with ablation studies further highlighting the efficacy of its core components. Our code is available at https://github.com/acharkq/ProtT3., Comment: ACL 2024, 9 pages

The truncated Wigner approximation (TWA) enables us to calculate bosonic quantum many-body dynamics while accounting for the effects of quantum fluctuations. In this work, we formulate the TWA for bosonic Markovian open quantum systems described by the Gorini-Kossakowski-Sudarshan-Lindblad (GKSL) equation from the coherent-state path-integral approach using the Wigner function. We derive an analytical expression for the GKSL equation in the TWA where we consider a bosonic system with an arbitrary Hamiltonian with jump operators that do not couple different states. We numerically confirm that the time evolution of physical quantities and the non-equal time correlation functions obtained in our formulation agree well with the exact ones in the numerically solvable models., Comment: 33 pages, 4 figures

We present ALMA [CII] 158 $\mu$m line and underlying far-infrared (FIR) continuum emission observations ($0''.57 \times 0''.46$ resolution) toward a quasar-quasar pair system recently discovered at $z = 6.05$ (Matsuoka et al. 2024). The quasar nuclei (C1 and C2) are faint ($M_{\rm 1450} \gtrsim -23$ mag), but we detect very bright [CII] emission bridging the 12 kpc between the two objects and extending beyond them (total luminosity $L_{\rm [CII]} \simeq 6 \times 10^9~L_\odot$). The [CII]-based total star formation rate of the system is $\sim 550~M_\odot$ yr$^{-1}$ (IR-based dust-obscured SFR is $\sim 100~M_\odot$ yr$^{-1}$), with a [CII]-based total gas mass of $\sim 10^{11}~M_\odot$. The dynamical masses of the two galaxies are large ($\sim 9 \times 10^{10}~M_\odot$ for C1 and $\sim 5 \times 10^{10}~M_\odot$ for C2). There is a smooth velocity gradient in [CII], indicating that these quasars are a tidally interacting system. We identified a dynamically distinct, fast [CII] component around C1: detailed inspection of the line spectrum there reveals the presence of a broad wing component, which we interpret as the indication of fast outflows with a velocity of $\sim 600$ km s$^{-1}$. The expected mass loading factor of the outflows, after accounting for multiphase gas, is $\gtrsim 2-3$, which is intermediate between AGN-driven and starburst-driven outflows. Hydrodynamic simulations in the literature predicted that this pair will evolve to a luminous ($M_{\rm 1450} \lesssim -26$ mag), starbursting ($\gtrsim 1000~M_\odot$ yr$^{-1}$) quasar after coalescence, one of the most extreme populations in the early universe., Comment: 16 pages incl. Appendix, 4 main figures, 1 table, 2 appendix figures. Accepted for publication in the Astrophysical Journal on 2024 June 11

We report the discovery of two quasars at a redshift of $z$ = 6.05, in the process of merging. They were serendipitously discovered from the deep multi-band imaging data collected by the Hyper Suprime-Cam (HSC) Subaru Strategic Program survey. The quasars, HSC $J$121503.42$-$014858.7 (C1) and HSC $J$121503.55$-$014859.3 (C2), both have luminous ($>$10$^{43}$ erg s$^{-1}$) Ly$\alpha$ emission with a clear broad component (full width at half maximum $>$1000 km s$^{-1}$). The rest-frame ultraviolet (UV) absolute magnitudes are $M_{1450} = -23.106 \pm 0.017$ (C1) and $-22.662 \pm 0.024$ (C2). Our crude estimates of the black hole masses provide $\log (M_{\rm BH}/M_\odot) = 8.1 \pm 0.3$ in both sources. The two quasars are separated by 12 kpc in projected proper distance, bridged by a structure in the rest-UV light suggesting that they are undergoing a merger. This pair is one of the most distant merging quasars reported to date, providing crucial insight into galaxy and black hole build-up in the hierarchical structure formation scenario. A companion paper will present the gas and dust properties captured by Atacama Large Millimeter/submillimeter Array observations, which provide additional evidence for and detailed measurements of the merger and also demonstrate that the two sources are not gravitationally-lensed images of a single quasar., Comment: Published from ApJL. A companion paper (T. Izumi et al.) presents the associated ALMA observations

We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process inspired by the successful strategy employed by AlphaZero. Our work leverages Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. To enhance consistency in intermediate steps, we combine outcome validation and stepwise self-evaluation, continually updating the quality assessment of newly generated data. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data. Theoretical analysis reveals the importance of using on-policy sampled data for successful self-improving. Extensive evaluations on various arithmetic and commonsense reasoning tasks demonstrate remarkable performance improvements over existing models. For instance, our approach outperforms the Mistral-7B Supervised Fine-Tuning (SFT) baseline on GSM8K, MATH, and ARC-C, with substantial increases in accuracy to $81.8\%$ (+$5.9\%$), $34.7\%$ (+$5.8\%$), and $76.4\%$ (+$15.8\%$), respectively. Additionally, our research delves into the training and inference compute tradeoff, providing insights into how our method effectively maximizes performance gains. Our code is publicly available at https://github.com/YuxiXie/MCTS-DPO., Comment: 10 pages, 4 figures, 4 tables (24 pages, 9 figures, 9 tables including references and appendices)

We study the merger of black hole-neutron star (BH-NS) binaries in numerical relativity, focusing on the properties of the remnant disk and the ejecta, varying the mass of compactness of the NS and the mass and spin of the BH. We find that within the precision of our numerical simulations, the remnant disk mass and ejecta mass normalized by the NS baryon mass ($\hat{M}_{\rm{rem}}$ and $\hat{M}_{\rm{eje}}$, respectively), and the cutoff frequency $f_{\rm{cut}}$ normalized by the initial total gravitational mass of the system at infinite separation approximately agree among the models with the same NS compactness $C_{\rm{NS}}=M_{\rm{NS}}/R_{\rm{NS}}$, mass ratio $Q=M_{\rm{BH}}/M_{\rm{NS}}$, and dimensionless BH spin $\chi_{\rm{BH}}$ irrespective of the NS mass $M_{\rm{NS}}$ in the range of $1.092$--$1.691\,M_\odot$. This result shows that the merger outcome depends sensitively on $Q$, $\chi_{\rm BH}$, and $C_{\rm{NS}}$ but only weekly on $M_{\rm{NS}}$. This justifies the approach of studying the dependence of NS tidal disruptions on the NS compactness by fixing the NS mass but changing the EOS. We further perform simulations with massive NSs of $M_{\rm{NS}}=1.8M_{\odot}$, and compare our results of $\hat{M}_{\rm{rem}}$ and $\hat{M}_{\rm{eje}}$ with those given by existing fitting formulas to test their robustness for more compact NSs. We find that the fitting formulas obtained in the previous studies are accurate within the numerical errors assumed, while our results also suggest that further improvement is possible by systematically performing more precise numerical simulations., Comment: 20 pages, 14 figures

We study long-term evolution of the matter ejected in a black-hole neutron-star (BH-NS) merger employing the results of a long-term numerical-relativity simulation and nucleosynthesis calculation, in which both dynamical and post-merger ejecta formation are consistently followed. In particular, we employ the results for the merger of a $1.35\,M_\odot$ NS and a $5.4\,M_\odot$ BH with the dimensionless spin of 0.75. We confirm the finding in the previous studies that thermal pressure induced by radioactive heating in the ejecta significantly modifies the morphology of the ejecta. We then compute the kilonova (KN) light curves employing the ejecta profile obtained by the long-term evolution. We find that our present BH-NS model results in a KN light curve that is fainter yet more enduring than that observed in AT2017gfo. This is due to the fact that the emission is primarily powered by the lanthanide-rich dynamical ejecta, in which a long photon diffusion time scale is realized by the large mass and high opacity. While the peak brightness of the KN emission in both the optical and near-infrared bands is fainter than or comparable to those of binary NS models, the time-scale maintaining the peak brightness is much longer in the near-infrared band for the BH-NS KN model. Our result indicates that a BH-NS merger with massive ejecta can observationally be identified by the bright and long lasting ($>$two weeks) near-infrared emission., Comment: 20 pages, 12 figures, submitted to MNRAS

Most works studying representation learning focus only on classification and neglect regression. Yet, the learning objectives and, therefore, the representation topologies of the two tasks are fundamentally different: classification targets class separation, leading to disconnected representations, whereas regression requires ordinality with respect to the target, leading to continuous representations. We thus wonder how the effectiveness of a regression representation is influenced by its topology, with evaluation based on the Information Bottleneck (IB) principle. The IB principle is an important framework that provides principles for learning effective representations. We establish two connections between it and the topology of regression representations. The first connection reveals that a lower intrinsic dimension of the feature space implies a reduced complexity of the representation Z. This complexity can be quantified as the conditional entropy of Z on the target Y, and serves as an upper bound on the generalization error. The second connection suggests a feature space that is topologically similar to the target space will better align with the IB principle. Based on these two connections, we introduce PH-Reg, a regularizer specific to regression that matches the intrinsic dimension and topology of the feature space with the target space. Experiments on synthetic and real-world regression tasks demonstrate the benefits of PH-Reg. Code: https://github.com/needylove/PH-Reg., Comment: ICML 2024

Clinical trials are pivotal in the drug discovery process to determine the safety and efficacy of a drug candidate. The high failure rates of these trials are attributed to deficiencies in clinical model development and protocol design. Improvements in the clinical drug design process could therefore yield significant benefits for all stakeholders involved. This paper examines the current challenges faced in clinical trial design and optimization, reviews established classical computational approaches, and introduces quantum algorithms aimed at enhancing these processes. Specifically, the focus is on three critical aspects: clinical trial simulations, site selection, and cohort identification. This study aims to provide a comprehensive framework that leverages quantum computing to innovate and refine the efficiency and effectiveness of clinical trials.

We solve high-dimensional steady-state Fokker-Planck equations on the whole space by applying tensor neural networks. The tensor networks are a tensor product of one-dimensional feedforward networks or a linear combination of several selected radial basis functions. The use of tensor feedforward networks allows us to efficiently exploit auto-differentiation in major Python packages while using radial basis functions can fully avoid auto-differentiation, which is rather expensive in high dimensions. We then use the physics-informed neural networks and stochastic gradient descent methods to learn the tensor networks. One essential step is to determine a proper truncated bounded domain or numerical support for the Fokker-Planck equation. To better train the tensor radial basis function networks, we impose some constraints on parameters, which lead to relatively high accuracy. We demonstrate numerically that the tensor neural networks in physics-informed machine learning are efficient for steady-state Fokker-Planck equations from two to ten dimensions.

Moir\'e superlattices consisting of two-dimensional materials have attracted immense attention because of emergent phenomena such as flat band-induced Mott insulating states and unconventional superconductivity. However, the effects of spin-orbit coupling on these materials have not yet been fully explored. Here, we show that single- and double-bilayer antimony honeycomb lattices, referred to as antimonene, form moir\'e superlattices on a Bi(111) substrate due to lattice mismatch. Scanning tunnelling microscopy (STM) measurements reveal the presence of spectral peaks near the Fermi level, which are spatially modulated with the moir\'e period. Angle-resolved photoemission spectroscopy (ARPES) combined with density functional theory calculations clarify the surface band structure with saddle points near the Fermi level, which allows us to attribute the observed STM spectral peaks to the van Hove singularity. Moreover, spin-resolved ARPES measurements reveal that the observed surface states are Rashba-type spin-polarized. The present work has significant implications in that Fermi surface instability and symmetry breaking may emerge at low temperatures, where the spin degree of freedom and electron correlation also play important roles., Comment: 37 pages including Supplementary Information