Search

Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety., Comment: updated to add missing acknowledgements

Human feedback can prevent overtly harmful utterances in conversational models, but may not automatically mitigate subtle problematic behaviors such as a stated desire for self-preservation or power. Constitutional AI offers an alternative, replacing human feedback with feedback from AI models conditioned only on a list of written principles. We find this approach effectively prevents the expression of such behaviors. The success of simple principles motivates us to ask: can models learn general ethical behaviors from only a single written principle? To test this, we run experiments using a principle roughly stated as "do what's best for humanity". We find that the largest dialogue models can generalize from this short constitution, resulting in harmless assistants with no stated interest in specific motivations like power. A general principle may thus partially avoid the need for a long list of constitutions targeting potentially harmful behaviors. However, more detailed constitutions still improve fine-grained control over specific types of harms. This suggests both general and specific principles have value for steering AI safely.

We test the hypothesis that language models trained with reinforcement learning from human feedback (RLHF) have the capability to "morally self-correct" -- to avoid producing harmful outputs -- if instructed to do so. We find strong evidence in support of this hypothesis across three different experiments, each of which reveal different facets of moral self-correction. We find that the capability for moral self-correction emerges at 22B model parameters, and typically improves with increasing model size and RLHF training. We believe that at this level of scale, language models obtain two capabilities that they can use for moral self-correction: (1) they can follow instructions and (2) they can learn complex normative concepts of harm like stereotyping, bias, and discrimination. As such, they can follow instructions to avoid certain kinds of morally harmful outputs. We believe our results are cause for cautious optimism regarding the ability to train language models to abide by ethical principles.

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors., Comment: for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks., Comment: v2 fixes a few typos from v1

"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning in transformer models of any size. For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing., Comment: 23+17 pages; refs added, typos fixed

Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or unintentionally because data deduplication is not perfect and the model is exposed to repeated data at the sentence, paragraph, or document level. Some works have reported substantial negative performance effects of this repeated data. In this paper we attempt to study repeated data systematically and to understand its effects mechanistically. To do this, we train a family of models where most of the data is unique but a small fraction of it is repeated many times. We find a strong double descent phenomenon, in which repeated data can lead test loss to increase midway through training. A predictable range of repetition frequency leads to surprisingly severe degradation in performance. For instance, performance of an 800M parameter model can be degraded to that of a 2x smaller model (400M params) by repeating 0.1% of the data 100 times, despite the other 90% of the training tokens remaining unique. We suspect there is a range in the middle where the data can be memorized and doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs. Finally, we connect these observations to recent mechanistic interpretability work - attempting to reverse engineer the detailed computations performed by the model - by showing that data repetition disproportionately damages copying and internal structures associated with generalization, such as induction heads, providing a possible mechanism for the shift from generalization to memorization. Taken together, these results provide a hypothesis for why repeating a relatively small fraction of data in large language models could lead to disproportionately large harms to performance., Comment: 23 pages, 22 figures

We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work., Comment: Data available at https://github.com/anthropics/hh-rlhf

Large-scale pre-training has recently emerged as a technique for creating capable, general purpose, generative models such as GPT-3, Megatron-Turing NLG, Gopher, and many others. In this paper, we highlight a counterintuitive property of such models and discuss the policy implications of this property. Namely, these generative models have an unusual combination of predictable loss on a broad training distribution (as embodied in their "scaling laws"), and unpredictable specific capabilities, inputs, and outputs. We believe that the high-level predictability and appearance of useful capabilities drives rapid development of such models, while the unpredictable qualities make it difficult to anticipate the consequences of model deployment. We go through examples of how this combination can lead to socially harmful behavior with examples from the literature and real world observations, and we also perform two novel experiments to illustrate our point about harms from unpredictability. Furthermore, we analyze how these conflicting properties combine to give model developers various motivations for deploying these models, and challenges that can hinder deployment. We conclude with a list of possible interventions the AI community may take to increase the chance of these models having a beneficial impact. We intend this paper to be useful to policymakers who want to understand and regulate AI systems, technologists who care about the potential policy impact of their work, and academics who want to analyze, critique, and potentially develop large generative models., Comment: Updated to reflect the version submitted (and accepted) to ACM FAccT '22. This update incorporates feedback from peer-review and fixes minor typos. See open access FAccT conference version at: https://dl.acm.org/doi/abs/10.1145/3531146.3533229

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences., Comment: 26+19 pages; v2 typos fixed, refs added, figure scale / colors fixed; v3 correct very non-standard TruthfulQA formatting and metric, alignment implications slightly improved

We propose that retinal-based phototrophy arose early in the evolution of life on Earth, profoundly impacting the development of photosynthesis and creating implications for the search for life beyond our planet. While the early evolutionary history of phototrophy is largely in the realm of the unknown, the onset of oxygenic photosynthesis in primitive cyanobacteria significantly altered the Earth's atmosphere by contributing to the rise of oxygen ~2.3 billion years ago. However, photosynthetic chlorophyll and bacteriochlorophyll pigments lack appreciable absorption at wavelengths about 500-600 nm, an energy-rich region of the solar spectrum. By contrast, simpler retinal-based light-harvesting systems such as the haloarchaeal purple membrane protein bacteriorhodopsin show a strong well-defined peak of absorbance centered at 568 nm, which is complementary to that of chlorophyll pigments. We propose a scenario where simple retinal-based light-harvesting systems like that of the purple chromoprotein bacteriorhodopsin, originally discovered in halophilic Archaea, may have dominated prior to the development of photosynthesis. We explore this hypothesis, termed the 'Purple Earth,' and discuss how retinal photopigments may serve as remote biosignatures for exoplanet research., Comment: Published Open Access in the International Journal of Astrobiology; 10 pages, 6 figures

For the first time in human history, we will soon be able to apply the scientific method to the question "Are We Alone?" The rapid advance of exoplanet discovery, planetary systems science, and telescope technology will soon allow scientists to search for life beyond our Solar System through direct observation of extrasolar planets. This endeavor will occur alongside searches for habitable environments and signs of life within our Solar System. While the searches are thematically related and will inform each other, they will require separate observational techniques. The search for life on exoplanets holds potential through the great diversity of worlds to be explored beyond our Solar System. However, there are also unique challenges related to the relatively limited data this search will obtain on any individual world. This white paper reviews the scientific community's ability to use data from future telescopes to search for life on exoplanets. This material summarizes products from the Exoplanet Biosignatures Workshop Without Walls (EBWWW). The EBWWW was constituted by a series of online and in person activities, with participation from the international exoplanet and astrobiology communities, to assess state of the science and future research needs for the remote detection of life on planets outside our Solar System., Comment: This is a white paper that was submitted to the National Academies of Sciences Study: Astrobiology Science Strategy for the Search for Life in the Universe

Exoplanet science promises a continued rapid accumulation of new observations in the near future, energizing a drive to understand and interpret the forthcoming wealth of data to identify signs of life beyond our Solar System. The large statistics of exoplanet samples, combined with the ambiguity of our understanding of universal properties of life and its signatures, necessitate a quantitative framework for biosignature assessment Here, we introduce a Bayesian framework for guiding future directions in life detection, which permits the possibility of generalizing our search strategy beyond biosignatures of known life. The Bayesian methodology provides a language to define quantitatively the conditional probabilities and confidence levels of future life detection and, importantly, may constrain the prior probability of life with or without positive detection. We describe empirical and theoretical work necessary to place constraints on the relevant likelihoods, including those emerging from stellar and planetary context, the contingencies of evolutionary history and the universalities of physics and chemistry. We discuss how the Bayesian framework can guide our search strategies, including determining observational wavelengths or deciding between targeted searches or larger, lower resolution surveys. Our goal is to provide a quantitative framework not entrained to specific definitions of life or its signatures, which integrates the diverse disciplinary perspectives necessary to confidently detect alien life., Comment: 125 pages (double spaced), 12 figures; The paper is the fourth in a series of 5 review manuscripts of the NExSS Exoplanet Biosignatures Workshop. Community comment was solicited at this url: https://nexss.info/groups/ebwww

Here we review how environmental context can be used to interpret whether O2 is a biosignature in extrasolar planetary observations. This paper builds on the overview of current biosignature research discussed in Schwieterman et al. (2017), and provides an in-depth, interdisciplinary example of biosignature identification and observation that serves as a basis for the development of the general framework for biosignature assessment described in Catling et al., (2017). O2 is a potentially strong biosignature that was originally thought to be an unambiguous indicator for life at high-abundance. We describe the coevolution of life with the early Earth's environment, and how the interplay of sources and sinks in the planetary environment may have resulted in suppression of O2 release into the atmosphere for several billion years, a false negative for biologically generated O2. False positives may also be possible, with recent research showing potential mechanisms in exoplanet environments that may generate relatively high abundances of atmospheric O2 without a biosphere being present. These studies suggest that planetary characteristics that may enhance false negatives should be considered when selecting targets for biosignature searches. Similarly our ability to interpret O2 observed in an exoplanetary atmosphere is also crucially dependent on environmental context to rule out false positive mechanisms. We describe future photometric, spectroscopic and time-dependent observations of O2 and the planetary environment that could increase our confidence that any observed O2 is a biosignature, and help discriminate it from potential false positives. By observing and understanding O2 in its planetary context we can increase our confidence in the remote detection of life, and provide a model for biosignature development for other proposed biosignatures., Comment: 55 pages. The paper is the second in a series of 5 review manuscripts of the NExSS Exoplanet Biosignatures Workshop. Community commenting is solicited at https://nexss.info/groups/ebwww

Finding life on exoplanets from telescopic observations is an ultimate goal of exoplanet science. Life produces gases and other substances, such as pigments, which can have distinct spectral or photometric signatures. Whether or not life is found with future data must be expressed with probabilities, requiring a framework of biosignature assessment. We present a framework in which we advocate using biogeochemical "Exo-Earth System" models to simulate potential biosignatures in spectra or photometry. Given actual observations, simulations are used to find the Bayesian likelihoods of those data occurring for scenarios with and without life. The latter includes "false positives" where abiotic sources mimic biosignatures. Prior knowledge of factors influencing planetary inhabitation, including previous observations, is combined with the likelihoods to give the Bayesian posterior probability of life existing on a given exoplanet. Four components of observation and analysis are necessary. 1) Characterization of stellar (e.g., age and spectrum) and exoplanetary system properties, including "external" exoplanet parameters (e.g., mass and radius) to determine an exoplanet's suitability for life. 2) Characterization of "internal" exoplanet parameters (e.g., climate) to evaluate habitability. 3) Assessment of potential biosignatures within the environmental context (components 1-2) and any corroborating evidence. 4) Exclusion of false positives. The resulting posterior Bayesian probabilities of life's existence map to five confidence levels, ranging from "very likely" (90-100%) to "very unlikely" ($\le$10%) inhabited., Comment: Part of a NASA Nexus for Exoplanet System Science (NExSS) series of 5 papers. V.3 published as an Open Access paper in Astrobiology

In the coming years and decades, advanced space- and ground-based observatories will allow an unprecedented opportunity to probe the atmospheres and surfaces of potentially habitable exoplanets for signatures of life. Life on Earth, through its gaseous products and reflectance and scattering properties, has left its fingerprint on the spectrum of our planet. Aided by the universality of the laws of physics and chemistry, we turn to Earth's biosphere, both in the present and through geologic time, for analog signatures that will aid in the search for life elsewhere. Considering the insights gained from modern and ancient Earth, and the broader array of hypothetical exoplanet possibilities, we have compiled a state-of-the-art overview of our current understanding of potential exoplanet biosignatures including gaseous, surface, and temporal biosignatures. We additionally survey biogenic spectral features that are well-known in the specialist literature but have not yet been robustly vetted in the context of exoplanet biosignatures. We briefly review advances in assessing biosignature plausibility, including novel methods for determining chemical disequilibrium from remotely obtainable data and assessment tools for determining the minimum biomass required for a given atmospheric signature. We focus particularly on advances made since the seminal review by Des Marais et al. (2002). The purpose of this work is not to propose new biosignatures strategies, a goal left to companion papers in this series, but to review the current literature, draw meaningful connections between seemingly disparate areas, and clear the way for a path forward., Comment: Open Access Article. 46 pages, 13 figures

The identification of a universal biosignature that could be sensed remotely is critical to the prospects for success in the search for life elsewhere in the universe. A candidate universal biosignature is homochirality, which is likely to be a generic property of all biochemical life. Due to the optical activity of chiral molecules, it has been hypothesized that this unique characteristic may provide a suitable remote sensing probe using circular polarization spectroscopy. Here, we report the detection of circular polarization in light scattered by photosynthetic microbes. We show that the circular polarization appears to arise from circular dichroism of the strong electronic transitions of photosynthetic absorption bands. We conclude that circular polarization spectroscopy could provide a powerful remote sensing technique for generic life searches., Comment: 15 pages, 4 figures; Proceedings of the National Academy of Sciences of the United States of America, in press

Time dependent STM has been used to evaluate step fluctuations as a function of temperature (300-450 K) on Ag(111) films grown on mica. The temporal correlation function scales as a power law in time, t^1/n with measured values of 1/n varying over a range of 0.19 pm 0.04 to 0.29 pm 0.04 with no dependence on temperature. The average value of 1/n = 0.24 pm 0.01 is consistent with step-edge diffusion limited fluctuations (n = z = 4, conserved noise). The magnitude of the time correlation function and the width of the fluctuations both scale with temperature with the same apparent activation energy of Eeff = 0.21 pm 0.02 eV, indicating that the correlation time is at most weakly temperature dependent. Direct analysis of the autocorrelation function confirms that the correlation time is at most weakly temperature dependent, and thus the apparent correlation length is strongly temperature dependent. This behavior can be reproduced by assuming that the apparent correlation length is governed by the longest wavelength of step fluctuations that can be sampled in the measurement time interval. Evaluation of the correlation time for previous measurements for Al/Si(111) (z =2) yields the same conclusion about measurement time interval. In both cases the ratio of the measurement time to the effective correlation time is on the order of 10.

Stochastic simulation results, appropriate for Molecular Beam Epitaxy, involving ballistic deposition and thermally activated Arrhenius diffusion of adatoms are presented for one- and two-dimensional substrates, allowing for overhangs and bulk vacancies. The asymptotic Kardar-Parisi- Zhang universality is found to be triggered by a sudden nucleation of large-scale defect formation in the growing film that shows a distinct dependence on dimensionality. The pre-nucleation transient behavior, which may be of experimental relevance due to the low defect content, is associated with standard solid-on-solid universality classes., Comment: 20 pages (+4 figures - not included), Latex 2.95, PP-1133