Search

Large language models rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks. Cross Entropy (CE) loss is the de facto choice in SFT, but it often leads to overfitting and limited output diversity due to its aggressive updates to the data distribution. This paper aim to address these issues by introducing the maximum entropy principle, which favors models with flatter distributions that still effectively capture the data. Specifically, we develop a new distribution matching method called GEM, which solves reverse Kullback-Leibler divergence minimization with an entropy regularizer. For the SFT of Llama-3-8B models, GEM outperforms CE in several aspects. First, when applied to the UltraFeedback dataset to develop general instruction-following abilities, GEM exhibits reduced overfitting, evidenced by lower perplexity and better performance on the IFEval benchmark. Furthermore, GEM enhances output diversity, leading to performance gains of up to 7 points on math reasoning and code generation tasks using best-of-n sampling, even without domain-specific data. Second, when fine-tuning with domain-specific datasets for math reasoning and code generation, GEM also shows less overfitting and improvements of up to 10 points compared with CE.

Structure and dynamics of an active polymer on a smooth cylindrical surface are studied by Brownian dynamics simulations. The effect of active force on the polymer adsorption behavior and the combined effect of chain mobility, length N, rigidity \k{appa}, and cylinder radius, R, on phase diagrams are systemically investigated. We find that complete adsorption is replaced by irregular alternative adsorption/desorption process at a large driving force. Three typical (spiral, helix-like, rod-like) conformations of the active polymer are observed, dependent on N, \k{appa}, and R. Dynamically, the polymer shows rotational motion in spiral state, snake-like motion in the intermediate state, and straight translational motion without turning back in the rod-like state. In the spiral state, we find that rotation velocity {\omega} and chain length follows a power-law relation {\omega}~N^(-0.42), consistent with the torque-balance theory of general Archimedean spirals. And the polymer shows super-diffusive behavior along the cylinder at long time in the helix-like and rod-like states. Our results highlight the mobility, rigidity, as well as curvature of surface can be used to regulate the polymer behavior.

The modeling and manipulation of 3D scenes captured from the real world are pivotal in various applications, attracting growing research interest. While previous works on editing have achieved interesting results through manipulating 3D meshes, they often require accurately reconstructed meshes to perform editing, which limits their application in 3D content generation. To address this gap, we introduce a novel single-image-driven 3D scene editing approach based on 3D Gaussian Splatting, enabling intuitive manipulation via directly editing the content on a 2D image plane. Our method learns to optimize the 3D Gaussians to align with an edited version of the image rendered from a user-specified viewpoint of the original scene. To capture long-range object deformation, we introduce positional loss into the optimization process of 3D Gaussian Splatting and enable gradient propagation through reparameterization. To handle occluded 3D Gaussians when rendering from the specified viewpoint, we build an anchor-based structure and employ a coarse-to-fine optimization strategy capable of handling long-range deformation while maintaining structural stability. Furthermore, we design a novel masking strategy to adaptively identify non-rigid deformation regions for fine-scale modeling. Extensive experiments show the effectiveness of our method in handling geometric details, long-range, and non-rigid deformation, demonstrating superior editing flexibility and quality compared to previous approaches., Comment: 10 pages, 12 figures

It has been supposed that the interplay of elasticity and activity plays a key role in triggering the non-equilibrium behaviors in biological systems. However, the experimental model system is missing to investigate the spatiotemporally dynamical phenomena. Here, a model system of an active chain, where active eccentric-disks are linked by a spring, is designed to study the interplay of activity, elasticity, and friction. Individual active chain exhibits longitudinal and transverse motion, however, it starts to self-rotate when pinning one end, and self-beats when clamping one end. Additionally, our eccentric-disk model can qualitatively reproduce such behaviors and explain the unusual self-rotation of the first disk around its geometric center. Further, the structure and dynamics of long chains were studied via simulations without steric interactions. It was found that hairpin conformation emerges in free motion, while in the constrained motions, the rotational and beating frequencies scale with the flexure number (the ratio of self-propelling force to bending rigidity), ~4/3. Scaling analysis suggests that it results from the balance between activity and energy dissipation. Our findings show that topological constraints play a vital role in non-equilibrium synergy behavior.

In this study, we introduce a novel approach in quantum field theories to estimate the action using the artificial neural networks (ANNs). The estimation is achieved by learning on system configurations governed by the Boltzmann factor, $e^{-S}$ at different temperatures within the imaginary time formalism of thermal field theory. We focus on 0+1 dimensional quantum field with kink/anti-kink configurations to demonstrate the feasibility of the method. The continuous-mixture autoregressive networks (CANs) enable the construction of accurate effective actions with tractable probability density estimation. Our numerical results demonstrate that this methodology not only facilitates the construction of effective actions at specified temperatures but also adeptly estimates the action at intermediate temperatures using data from both lower and higher temperature ensembles. This capability is especially valuable for the detailed exploration of phase diagrams., Comment: 8 pages, 4 figures. Comments welcome!

In this paper we investigate the existence of singular solutions to the conformal Dirac-Einstein system. Because of its conformal invariance, there are many similarities with the classical construction of singular solutions for the Yamabe problem. We construct here a family of singular solutions, on the three-dimensional sphere, having exactly two singularities., Comment: 22 pages

3D Gaussian splatting, emerging as a groundbreaking approach, has drawn increasing attention for its capabilities of high-fidelity reconstruction and real-time rendering. However, it couples the appearance and geometry of the scene within the Gaussian attributes, which hinders the flexibility of editing operations, such as texture swapping. To address this issue, we propose a novel approach, namely Texture-GS, to disentangle the appearance from the geometry by representing it as a 2D texture mapped onto the 3D surface, thereby facilitating appearance editing. Technically, the disentanglement is achieved by our proposed texture mapping module, which consists of a UV mapping MLP to learn the UV coordinates for the 3D Gaussian centers, a local Taylor expansion of the MLP to efficiently approximate the UV coordinates for the ray-Gaussian intersections, and a learnable texture to capture the fine-grained appearance. Extensive experiments on the DTU dataset demonstrate that our method not only facilitates high-fidelity appearance editing but also achieves real-time rendering on consumer-level devices, e.g. a single RTX 2080 Ti GPU.

Natural Medicinal Materials (NMMs) have a long history of global clinical applications and a wealth of records and knowledge. Although NMMs are a major source for drug discovery and clinical application, the utilization and sharing of NMM knowledge face crucial challenges, including the standardized description of critical information, efficient curation and acquisition, and language barriers. To address these, we developed ShennongAlpha, an AI-driven sharing and collaboration platform for intelligent knowledge curation, acquisition, and translation. For standardized knowledge curation, the platform introduced a Systematic Nomenclature to enable accurate differentiation and identification of NMMs. More than fourteen thousand Chinese NMMs have been curated into the platform along with their knowledge. Furthermore, the platform pioneered chat-based knowledge acquisition, standardized machine translation, and collaborative knowledge updating. Together, our study represents the first major advance in leveraging AI to empower NMM knowledge sharing, which not only marks a novel application of AI for Science, but also will significantly benefit the global biomedical, pharmaceutical, physician, and patient communities., Comment: 53 pages, 6 figures, 10 supplementary figures, 2 supplementary tables

Aligning intelligent agents with human preferences and values is important. This paper examines two popular alignment methods: Direct Preference Optimization (DPO) and Reward-Model-Based Policy Optimization (RMB-PO). A variant of RMB-PO, referred to as RMB-PO+ is also considered. These methods, either explicitly or implicitly, learn a reward model from preference data and differ in the data used for policy optimization to unlock the generalization ability of the reward model. In particular, compared with DPO, RMB-PO additionally uses policy-generated data, and RMB-PO+ further leverages new, preference-free data. We examine the impact of such out-of-preference data. Our study, conducted through controlled and synthetic experiments, demonstrates that DPO performs poorly, whereas RMB-PO+ performs the best. In particular, even when providing the policy model with a good feature representation, we find that policy optimization with adequate out-of-preference data significantly improves performance by harnessing the reward model's generalization capabilities.

In the realm of quantum control, reinforcement learning, a prominent branch of machine learning, emerges as a competitive candidate for computer-assisted optimal design for experiments. This study investigates the extent to which guidance from human experts is necessary for the effective implementation of reinforcement learning in designing quantum control protocols. Specifically, we focus on the engineering of a robust two-qubit gate within a nuclear magnetic resonance system, utilizing a combination of analytical solutions as prior knowledge and techniques from the field of computer science. Through extensive benchmarking of different models, we identify dropout, a widely-used method for mitigating overfitting in machine learning, as an especially robust approach. Our findings demonstrate the potential of incorporating computer science concepts to propel the development of advanced quantum technologies., Comment: 11 pages, 4 figures

Reinforcement Learning from Human Feedback (RLHF) is key to aligning Large Language Models (LLMs), typically paired with the Proximal Policy Optimization (PPO) algorithm. While PPO is a powerful method designed for general reinforcement learning tasks, it is overly sophisticated for LLMs, leading to laborious hyper-parameter tuning and significant computation burdens. To make RLHF efficient, we present ReMax, which leverages 3 properties of RLHF: fast simulation, deterministic transitions, and trajectory-level rewards. These properties are not exploited in PPO, making it less suitable for RLHF. Building on the renowned REINFORCE algorithm, ReMax does not require training an additional value model as in PPO and is further enhanced with a new variance reduction technique. ReMax offers several benefits over PPO: it is simpler to implement, eliminates more than 4 hyper-parameters in PPO, reduces GPU memory usage, and shortens training time. ReMax can save about 46% GPU memory than PPO when training a 7B model and enables training on A800-80GB GPUs without the memory-saving offloading technique needed by PPO. Applying ReMax to a Mistral-7B model resulted in a 94.78% win rate on the AlpacaEval leaderboard and a 7.739 score on MT-bench, setting a new SOTA for open-source 7B models. These results show the effectiveness of ReMax while addressing the limitations of PPO in LLMs.

Learning a precise dynamics model can be crucial for offline reinforcement learning, which, unfortunately, has been found to be quite challenging. Dynamics models that are learned by fitting historical transitions often struggle to generalize to unseen transitions. In this study, we identify a hidden but pivotal factor termed dynamics reward that remains consistent across transitions, offering a pathway to better generalization. Therefore, we propose the idea of reward-consistent dynamics models: any trajectory generated by the dynamics model should maximize the dynamics reward derived from the data. We implement this idea as the MOREC (Model-based Offline reinforcement learning with Reward Consistency) method, which can be seamlessly integrated into previous offline model-based reinforcement learning (MBRL) methods. MOREC learns a generalizable dynamics reward function from offline data, which is subsequently employed as a transition filter in any offline MBRL method: when generating transitions, the dynamics model generates a batch of transitions and selects the one with the highest dynamics reward value. On a synthetic task, we visualize that MOREC has a strong generalization ability and can surprisingly recover some distant unseen transitions. On 21 offline tasks in D4RL and NeoRL benchmarks, MOREC improves the previous state-of-the-art performance by a significant margin, i.e., 4.6% on D4RL tasks and 25.9% on NeoRL tasks. Notably, MOREC is the first method that can achieve above 95% online RL performance in 6 out of 12 D4RL tasks and 3 out of 9 NeoRL tasks.

Imitation learning (IL) has proven to be an effective method for learning good policies from expert demonstrations. Adversarial imitation learning (AIL), a subset of IL methods, is particularly promising, but its theoretical foundation in the presence of unknown transitions has yet to be fully developed. This paper explores the theoretical underpinnings of AIL in this context, where the stochastic and uncertain nature of environment transitions presents a challenge. We examine the expert sample complexity and interaction complexity required to recover good policies. To this end, we establish a framework connecting reward-free exploration and AIL, and propose an algorithm, MB-TAIL, that achieves the minimax optimal expert sample complexity of $\widetilde{O} (H^{3/2} |S|/\varepsilon)$ and interaction complexity of $\widetilde{O} (H^{3} |S|^2 |A|/\varepsilon^2)$. Here, $H$ represents the planning horizon, $|S|$ is the state space size, $|A|$ is the action space size, and $\varepsilon$ is the desired imitation gap. MB-TAIL is the first algorithm to achieve this level of expert sample complexity in the unknown transition setting and improves upon the interaction complexity of the best-known algorithm, OAL, by $O(H)$. Additionally, we demonstrate the generalization ability of MB-TAIL by extending it to the function approximation setting and proving that it can achieve expert sample and interaction complexity independent of $|S|$, Comment: arXiv admin note: text overlap with arXiv:2106.10424

Let $(M,\textit{g},\sigma)$ be an $m$-dimensional closed spin manifold, with a fixed Riemannian metric $\textit{g}$ and a fixed spin structure $\sigma$; let $\mathbb{S}(M)$ be the spinor bundle over $M$. The spinorial Yamabe-type problems address the solvability of the following equation \[ D_{\textit{g}}\psi = f(x)|\psi|_{\textit{g}}^{\frac2{m-1}}\psi, \quad \psi:M\to\mathbb{S}(M), \ x\in M \] where $D_{\textit{g}}$ is the associated Dirac operator and $f:M\to\mathbb{R}$ is a given function. The study of such nonlinear equation is motivated by its important applications in Spin Geometry: when $m=2$, a solution corresponds to a conformal isometric immersion of the universal covering $\widetilde M$ into $\mathbb{R}^3$ with prescribed mean curvature $f$; meanwhile, for general dimensions and $f\equiv constant\neq0$, a solution provides an upper bound estimate for the B\"ar-Hijazi-Lott invariant. The aim of this paper is to establish non-compactness results related to the spinorial Yamabe-type problems. Precisely, concrete analysis is made for two specific models on the manifold $(S^m,\textit{g})$ where the solution set of the spinorial Yamabe-type problem is not compact: $1).$ the geometric potential $f$ is constant (say $f\equiv1$) with the background metric $\textit{g}$ being a $C^k$ perturbation of the canonical round metric $\textit{g}_{S^m}$, which is not conformally flat somewhere on $S^m$; $2).$ $f$ is a perturbation from constant and is of class $C^2$, while the background metric $\textit{g}\equiv\textit{g}_{S^m}$., Comment: 45 pages. arXiv admin note: text overlap with arXiv:2304.02807

This paper is part of a program to establish the existence theory for the conformally invariant Dirac equation \[ D_{\textit{g}}\psi=f(x)|\psi|_{\textit{g}}^{\frac2{m-1}}\psi \] on a closed spin manifold $(M,\textit{g})$ of dimension $m\geq2$ with a fixed spin structure, where $f:M\to\mathbb{R}$ is a given function. The study on such nonlinear equation is motivated by its important applications in Spin Geometry: when $m=2$, a solution corresponds to an isometric immersion of the universal covering $\widetilde M$ into $\mathbb{R}^3$ with prescribed mean curvature $f$; meanwhile, for general dimensions and $f\equiv constant$, a solution provides an upper bound estimate for the B\"ar-Hijazi-Lott invariant.