Search

Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training self-correction typically depend on either multiple models, a more advanced model, or additional forms of supervision. To address these shortcomings, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are often insufficient for instilling self-correction behavior. In particular, we observe that training via SFT falls prey to either a distribution mismatch between mistakes made by the data-collection policy and the model's own responses, or to behavior collapse, where learning implicitly prefers only a certain mode of correction behavior that is often not effective at self-correction on test problems. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction behavior that is effective at test time as opposed to fitting high-reward responses for a given prompt. This regularization process includes an initial phase of multi-turn RL on a base model to generate a policy initialization that is less susceptible to collapse, followed by using a reward bonus to amplify self-correction. With Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.

While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms, and there is no universally accepted definition. We thus focus on studying only those hallucinations where a correct answer appears verbatim in the training set. To fully control the training data content, we construct a knowledge graph (KG)-based dataset, and use it to train a set of increasingly large LMs. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on $\leq5$% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoffmann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size improves performance on fixed LM's outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations., Comment: Published at COLM 2024. 16 pages, 11 figures

We study the cosmological evolution of a light QCD axion and identify the parameter space to obtain the correct relic dark matter abundance. The axion potential is flattened at the origin, corresponding to the only minimum, while it is unsuppressed at $\pi$. These potential features arise by assuming a mirror sector with the strong CP phase $\bar\theta$ shifted by $\pi$ compared to the SM sector, which allows the mirror axion potential to be tuned against the usual QCD axion potential. Before the QCD phase transition, assuming the mirror sector is decoupled and much colder than the SM thermal bath, the mirror sector potential dominates, causing the axion to initially roll to a temporary minimum at $\pi$. However, after the QCD phase transition, the potential minimum changes, and the axion relaxes from the newly created "hilltop" near $\pi$ to the CP-conserving minimum at the origin. As the axion adiabatically tracks this shift in the potential minimum through the QCD phase transition, with non-adiabatic evolution near $\pi$ and 0, it alters the usual prediction of the dark matter abundance. Consequently, this "hilltop" misalignment mechanism opens new regions of axion parameter space, with the correct relic abundance while still solving the strong CP problem, that could be explored in future experiments., Comment: 28 pages, 9 figures. Match to the published version in JHEP

Understanding the rules underlying organismal development is a major unsolved problem in biology. Each cell in a developing organism responds to signals in its local environment by dividing, excreting, consuming, or reorganizing, yet how these individual actions coordinate over a macroscopic number of cells to grow complex structures with exquisite functionality is unknown. Here we use recent advances in automatic differentiation to discover local interaction rules and genetic networks that yield emergent, systems-level characteristics in a model of development. We consider a growing tissue with cellular interactions mediated by morphogen diffusion, cell adhesion and mechanical stress. Each cell has an internal genetic network that is used to make decisions based on the cell's local environment. We show that one can learn the parameters governing cell interactions in the form of interpretable genetic networks for complex developmental scenarios, including directed axial elongation, cell type homeostasis via chemical signaling and homogenization of growth via mechanical stress. When combined with recent experimental advances measuring spatio-temporal dynamics and gene expression of cells in a growing tissue, the methodology outlined here offers a promising path to unraveling the cellular bases of development., Comment: 12 pages, 4 figures