Search

The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as simple chatbots. Meanwhile, LLM agents -- which use external tools and can execute multi-stage tasks -- may pose a greater risk if misused, but their robustness remains underexplored. To facilitate research on LLM agent misuse, we propose a new benchmark called AgentHarm. The benchmark includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. In addition to measuring whether models refuse harmful agentic requests, scoring well on AgentHarm requires jailbroken agents to maintain their capabilities following an attack to complete a multi-step task. We evaluate a range of leading LLMs, and find (1) leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking, (2) simple universal jailbreak templates can be adapted to effectively jailbreak agents, and (3) these jailbreaks enable coherent and malicious multi-step agent behavior and retain model capabilities. To enable simple and reliable evaluation of attacks and defenses for LLM-based agents, we publicly release AgentHarm at https://huggingface.co/datasets/ai-safety-institute/AgentHarm.

When solving challenging problems, language models (LMs) are able to identify relevant information from long and complicated contexts. To study how LMs solve retrieval tasks in diverse situations, we introduce ORION, a collection of structured retrieval tasks spanning six domains, from text understanding to coding. Each task in ORION can be represented abstractly by a request (e.g. a question) that retrieves an attribute (e.g. the character name) from a context (e.g. a story). We apply causal analysis on 18 open-source language models with sizes ranging from 125 million to 70 billion parameters. We find that LMs internally decompose retrieval tasks in a modular way: middle layers at the last token position process the request, while late layers retrieve the correct entity from the context. After causally enforcing this decomposition, models are still able to solve the original task, preserving 70% of the original correct token probability in 98 of the 106 studied model-task pairs. We connect our macroscopic decomposition with a microscopic description by performing a fine-grained case study of a question-answering task on Pythia-2.8b. Building on our high-level understanding, we demonstrate a proof of concept application for scalable internal oversight of LMs to mitigate prompt-injection while requiring human supervision on only a single input. Our solution improves accuracy drastically (from 15.5% to 97.5% on Pythia-12b). This work presents evidence of a universal emergent modular processing of tasks across varied domains and models and is a pioneering effort in applying interpretability for scalable internal oversight of LMs.

Mechanistic interpretability aims to explain what a neural network has learned at a nuts-and-bolts level. What are the fundamental primitives of neural network representations? Previous mechanistic descriptions have used individual neurons or their linear combinations to understand the representations a network has learned. But there are clues that neurons and their linear combinations are not the correct fundamental units of description: directions cannot describe how neural networks use nonlinearities to structure their representations. Moreover, many instances of individual neurons and their combinations are polysemantic (i.e. they have multiple unrelated meanings). Polysemanticity makes interpreting the network in terms of neurons or directions challenging since we can no longer assign a specific feature to a neural unit. In order to find a basic unit of description that does not suffer from these problems, we zoom in beyond just directions to study the way that piecewise linear activation functions (such as ReLU) partition the activation space into numerous discrete polytopes. We call this perspective the polytope lens. The polytope lens makes concrete predictions about the behavior of neural networks, which we evaluate through experiments on both convolutional image classifiers and language models. Specifically, we show that polytopes can be used to identify monosemantic regions of activation space (while directions are not in general monosemantic) and that the density of polytope boundaries reflect semantic boundaries. We also outline a vision for what mechanistic interpretability might look like through the polytope lens., Comment: 22/11/22 initial upload

We obtain the first complete classification result for algebraically primitive Teichm\"{u}ller curves in a stratum component in genus greater than 2. Specifically, we show that the Veech 14-gon generates the unique algebraically primitive Teichm\"{u}ller curve in the hyperelliptic component of the stratum $\Omega\mathcal{M}_3(2,2)$., Comment: 19 pages, comments welcome

We give a new proof of the classification of ${\rm GL}^+(2,\mathbb{R})$-orbit closures that are saturated for the absolute period foliation of the Hodge bundle. As a consequence, we obtain a short proof of the classification of closures of leaves of the absolute period foliation of the Hodge bundle. Our approach is based on a method for classifying ${\rm GL}^+(2,\mathbb{R})$-orbit closures using deformations of flat pairs of pants., Comment: 19 pages, 1 additional result, comments welcome

We show the existence of a dense orbit for real Rel flows on the area-1 locus of every connected component of every stratum of holomorphic 1-forms with at least 2 distinct zeros. For this purpose, we establish a general density criterion for ${\rm SL}(2,\mathbb{R})$-orbit closures, based on finding an orbit of a real Rel flow whose closure contains a horocycle. This criterion can be verified using explicit constructions of holomorphic 1-forms with a periodic horizontal foliation. Our constructions also provide explicit examples of dense leaves of the absolute period foliation and many subfoliations of these loci., Comment: 32 pages, comments welcome

Recent advances in efficient Transformers have exploited either the sparsity or low-rank properties of attention matrices to reduce the computational and memory bottlenecks of modeling long sequences. However, it is still challenging to balance the trade-off between model quality and efficiency to perform a one-size-fits-all approximation for different tasks. To better understand this trade-off, we observe that sparse and low-rank approximations excel in different regimes, determined by the softmax temperature in attention, and sparse + low-rank can outperform each individually. Inspired by the classical robust-PCA algorithm for sparse and low-rank decomposition, we propose Scatterbrain, a novel way to unify sparse (via locality sensitive hashing) and low-rank (via kernel feature map) attention for accurate and efficient approximation. The estimation is unbiased with provably low error. We empirically show that Scatterbrain can achieve 2.1x lower error than baselines when serving as a drop-in replacement in BigGAN image generation and pre-trained T2T-ViT. On a pre-trained T2T Vision transformer, even without fine-tuning, Scatterbrain can reduce 98% of attention memory at the cost of only 1% drop in accuracy. We demonstrate Scatterbrain for end-to-end training with up to 4 points better perplexity and 5 points better average accuracy than sparse or low-rank efficient transformers on language modeling and long-range-arena tasks., Comment: NeurIPS 2021

Let $\mathcal{C}$ be a connected component of a stratum of the moduli space of holomorphic $1$-forms of genus $g$. We show that the absolute period foliation of $\mathcal{C}$ is ergodic on the area-$1$ locus, and that the non-dense leaves lie in an explicit countable union of suborbifolds, subject to a mild assumption on $\mathcal{C}$. We show similar results for subspaces of $\mathcal{C}$ defined by topological restrictions on the absolute periods. We obtain these dynamical results by showing that for a typical positive cohomology class in $H^1(S_g;\mathbb{C})$, the associated space of isoperiodic forms in $\mathcal{C}$ is connected. Lastly, we show that certain covering constructions provide examples of spaces of isoperiodic forms with positive dimension and infinitely many connected components., Comment: 68 pages, typo corrected in statements of Theorems 1.1-1.2 and in Sections 5-6

Let $\mathcal E: y^2 = x^3 + A(T)x + B(T)$ be a nontrivial one-parameter family of elliptic curves over $\mathbb{Q}(T)$, with $A(T), B(T) \in \mathbb Z(T)$, and consider the $k$\textsuperscript{th} moments $A_{k,\mathcal{E}}(p) := \sum_{t (p)} a_{\mathcal{E}_t}(p)^k$ of the Dirichlet coefficients $a_{\mathcal{E}_t}(p) := p + 1 - |\mathcal{E}_t (\mathbb{F}_p)|$. Rosen and Silverman proved a conjecture of Nagao relating the first moment $A_{1,\mathcal{E}}(p)$ to the rank of the family over $\mathbb{Q}(T)$, and Michel proved that if $j(T)$ is not constant then the second moment is equal to $A_{2,\mathcal{E}}(p) = p^2 + O(p^{3/2})$. Cohomological arguments show that the lower order terms are of sizes $p^{3/2}, p, p^{1/2}$, and $1$. In every case we are able to analyze in closed form, the largest lower order term in the second moment expansion that does not average to zero is on average negative, though numerics suggest this may fail for families of moderate rank. We prove this Bias Conjecture for several large classes of families, including families with rank, complex multiplication, and constant $j(T)$-invariant. We also study the analogous Bias Conjecture for families of Dirichlet characters, holomorphic forms on GL$(2)/\mathbb{Q}$, and their symmetric powers and Rankin-Selberg convolutions. We identify all lower order terms in large classes of families, shedding light on the arithmetic objects controlling these terms. The negative bias in these lower order terms has implications toward the excess rank conjecture and the behavior of zeros near the central point in families of $L$-functions., Comment: Version 1.0, 40 pages, 2 appendices

This brief is the first in a series focusing on Employment First implementation as it relates to one of the seven elements within the High-Performing States in Integrated Employment model. It examines the background of circumstances under which Employment First efforts began in seven states, and introduces each state's values, mission, and goals around increasing employment opportunities for people with disabilities. States may use the lessons in this brief to develop an Employment First policy, or to evolve existing efforts. [This is a publication of Access to Integrated Employment, a project of ThinkWork! at the Institute for Community Inclusion.]

This brief is the third in a series focusing on Employment First implementation as it relates to one of the seven elements within the High-Performing States in Integrated Employment model. It examines the background of circumstances under which Employment First efforts began in seven states, and introduces each state's values, mission, and goals around increasing employment opportunities for people with disabilities. States may use the lessons in this brief to develop an Employment First policy, or to evolve existing efforts. [This is a publication of Access to Integrated Employment, a project of ThinkWork at the Institute for Community Inclusion.]

A generalized lexicographic order on words is a lexicographic order where the total order of the alphabet depends on the position of the comparison. A generalized Lyndon word is a finite word which is strictly smallest among its class of rotations with respect to a generalized lexicographic order. This notion can be extended to infinite words: an infinite generalized Lyndon word is an infinite word which is strictly smallest among its class of suffixes. We prove a conjecture of Dolce, Restivo, and Reutenauer: every infinite word has a unique nonincreasing factorization into finite and infinite generalized Lyndon words. When this factorization has finitely many terms, we characterize the last term of the factorization. Our methods also show that the infinite generalized Lyndon words are precisely the words with infinitely many generalized Lyndon prefixes., Comment: 14 pages, 1 figure

We derive a refined conjecture for the variance of Gaussian primes across sectors, with a power saving error term, by applying the L-functions Ratios Conjecture. We observe a bifurcation point in the main term, consistent with the Random Matrix Theory (RMT) heuristic previously proposed by Rudnick and Waxman. Our model also identifies a second bifurcation point, undetected by the RMT model, that emerges upon taking into account lower order terms. For sufficiently small sectors, we moreover prove an unconditional result that is consistent with our conjecture down to lower order terms., Comment: 47 pages. Minor revisions. Appendix with relevant Mathematica code, included. Accepted for publication in Experimental Mathematics

An equivalent definition of the Fibonacci numbers is that they are the unique sequence such that every integer can be written uniquely as a sum of non-adjacent terms. We can view this as we have bins of length 1, we can take at most one element from a bin, and if we choose an element from a bin we cannot take one from a neighboring bin. We generalize to allowing bins of varying length and restrictions as to how many elements may be used in a decomposition. We derive conditions on when the resulting sequences have uniqueness of decomposition, and (similar to the Fibonacci case) when the number of summands converges to a Gaussian; the main tool in the proofs here is the Lyaponuv Central Limit Theorem., Comment: Version 1.0, 18 pages

Recently Burkhardt et. al. introduced the $k$-checkerboard random matrix ensembles, which have a split limiting behavior of the eigenvalues (in the limit all but $k$ of the eigenvalues are on the order of $\sqrt{N}$ and converge to semi-circular behavior, with the remaining $k$ of size $N$ and converging to hollow Gaussian ensembles). We generalize their work to consider non-Hermitian ensembles with complex eigenvalues; instead of a blip new behavior is seen, ranging from multiple satellites to annular rings. These results are based on moment method techniques adapted to the complex plane as well as analysis of singular values, and we further isolate the singular value joint density formula for the Complex Symmetric Gaussian Ensemble., Comment: Version 1.0, 35 pages, 5 figures

This brief is the second in a series focusing on Employment First implementation as it relates to one of the seven elements within the High-Performing States in Integrated Employment model. It examines the background of circumstances under which Employment First efforts began in seven states, and introduces each state's values, mission, and goals around increasing employment opportunities for people with disabilities. States may use the lessons in this brief to develop an Employment First policy, or to evolve existing efforts. [This is a publication of Access to Integrated Employment, a project of ThinkWork! at the Institute for Community Inclusion. For the first brief, "State Employment First Policies: State Definitions, Goals, and Values," see ED605611.]

Due to its proximity, the Milky Way nuclear star cluster provides us with a wealth of data not available in other galactic nuclei. In particular, with adaptive optics, we can observe the detailed properties of individual stars, which can offer insight into the origin and evolution of the cluster. We summarize work on the central parsec of the Galactic center based on imaging and spectroscopic observations at the Keck and Gemini telescopes. These observations include stellar positions in two dimension and the velocity in three dimensions. Spectroscopic observations also enable measurements of the physical properties of individual stars, such as the spectral type and in some cases the effective temperature, metallicity, and surface gravity. We present a review of our latest measurements of the density profiles and luminosity functions of the young and old stars in this region. These observations show a complex stellar population with a young (4-6 Myr) compact star cluster in the central 0.5 pc embedded in an older and much more massive nuclear star cluster. Surprisingly, the old late-type giants do not show a cusp profile as long been expected from theoretical work. The solution to the missing cusp problem may offer us insight into the dynamical evolution of the cluster. Finally, we also discuss recent work on the metallicity of stars in this region and how they might be used to trace their origin. The nuclear star cluster shows a large variation in metallicity ([M/Fe]). The majority of the stars have higher than solar metallicity, with about 6\% having [M/Fe] $< -0.5$. These observations indicate that the NSC was not built from the globular clusters that we see today. The formation of the nuclear star cluster is more likely from the inward migration of gas originating in the disk of the Milky Way., Comment: 9 pages, 5 figures, 2 tables, IAU 322 The Multi-Messenger Astrophysics of the Galactic Centre

While Random Matrix Theory has successfully modeled many quantities of families of L-functions, it frequently cannot see the family's arithmetic. In some situations this requires an extended theory that inserts arithmetic factors depending on the family, while in other cases these factors result in contributions which vanish in the limit, and are thus not detected. We review the general theory associated to one of the most important statistics, the n-level density of zeros near the central point. According to the Katz-Sarnak density conjecture, to each family of L-functions there is a corresponding symmetry group such that the behavior of zeros near the central point as the conductors tend to infinity agrees with the behavior of eigenvalues near 1 as the matrix size tends to infinity. We show how these calculations are done, emphasizing the techniques, methods and obstructions to improving the results, by considering in full detail a family of Dirichlet characters. We then describe how we may associate a symmetry constant to each family, and how to determine the symmetry group of a compound family in terms of the symmetries of the constituents. These calculations explain the remarkable universality of behavior, where the main terms are independent of the arithmetic (only the first two moments of the Satake parameters contribute in the limit; similar to the Central Limit Theorem, the higher moments are only felt in the rate of convergence). We end by exploring lower order terms in families of elliptic curves. We present evidence supporting a conjecture that the average second moment in one-parameter families without complex multiplication has, when appropriately viewed, a negative bias, and end with a discussion of the consequences of this bias on the distribution of low-lying zeros, in particular relations between such a bias and the observed excess rank in families., Comment: Version 1.0, 43 pages

We present a metallicity analysis of 83 late-type giants within the central 1 pc of the Milky Way. K-band spectroscopy of these stars were obtained with the medium-spectral resolution integral-field spectrograph NIFS on Gemini North using laser-guide star adaptive optics. Using spectral template fitting with the MARCS synthetic spectral grid, we find that there is large variation in metallicity, with stars ranging from [M/H] $<$ -1.0 to above solar metallicity. About 6\% of the stars have [M/H] $<$ -0.5. This result is in contrast to previous observations, with smaller samples, that show stars at the Galactic center have approximately solar metallicity with only small variations. Our current measurement uncertainties are dominated by systematics in the model, especially at [M/H] $>$ 0, where there are stellar lines not represented in the model. However, the conclusion that there are low metallicity stars, as well as large variations in metallicity is robust. The metallicity may be an indicator of the origin of these stars. The low-metallicity population is consistent with that of globular clusters in the Milky Way, but their small fraction likely means that globular cluster infall is not the dominant mechanism for forming the Milky Way nuclear star cluster. The majority of stars are at or above solar metallicity, which suggests they were formed closer to the Galactic center or from the disk. In addition, our results indicate that it will be important for star formation history analyses using red giants at the Galactic center to consider the effect of varying metallicity., Comment: 11 pages, 10 figures, ApJ Accepted

Consider the ensembles of real symmetric Toeplitz matrices and real symmetric Hankel matrices whose entries are i.i.d. random variables chosen from a fixed probability distribution p of mean 0, variance 1, and finite higher moments. Previous work on real symmetric Toeplitz matrices shows that the spectral measures, or densities of normalized eigenvalues, converge almost surely to a universal near-Gaussian distribution, while previous work on real symmetric Hankel matrices shows that the spectral measures converge almost surely to a universal non-unimodal distribution. Real symmetric Toeplitz matrices are constant along the diagonals, while real symmetric Hankel matrices are constant along the skew diagonals. We generalize the Toeplitz and Hankel matrices to study matrices that are constant along some curve described by a real-valued bivariate polynomial. Using the Method of Moments and an analysis of the resulting Diophantine equations, we show that the spectral measures associated with linear bivariate polynomials converge in probability and almost surely to universal non-semicircular distributions. We prove that these limiting distributions approach the semicircle in the limit of large values of the polynomial coefficients. We then prove that the spectral measures associated with the sum or difference of any two real-valued polynomials with different degrees converge in probability and almost surely to a universal semicircular distribution., Comment: 24 pages, 8 figures

Let $L(s,f)$ be an $L$-function associated to a primitive (holomorphic or Maass) cusp form $f$ on GL(2) over $\mathbb{Q}$. Combining mean-value estimates of Montgomery and Vaughan with a method of Ramachandra, we prove a formula for the mixed second moments of derivatives of $L(1/2+it,f)$ and, via a method of Hall, use it to show that there are infinitely many gaps between consecutive zeros of $L(s,f)$ along the critical line that are at least $\sqrt 3 = 1.732...$ times the average spacing. Using general pair correlation results due to Murty and Perelli in conjunction with a technique of Montgomery, we also prove the existence of small gaps between zeros of any primitive $L$-function of the Selberg class. In particular, when $f$ is a primitive holomorphic cusp form on GL(2) over $\mathbb{Q}$, we prove that there are infinitely many gaps between consecutive zeros of $L(s,f)$ along the critical line that are at most $< 0.823$ times the average spacing., Comment: This article is a product of the 2014 SMALL REU at Williams College. Version 2 Comments: 25 pages, typos fixed, and references updated. We thank Micah Milinovich for his feedback. Version 3 Comment: to appear in the Journal of Mathematical Analysis and Applications. Version 4 Comments: typo in equation (1.6) fixed, in press at the Journal of Mathematical Analysis and Applications

We derived similar to Bo et al. (2010) results but in the case when the dynamics of the FX rate is driven by a general Merton jump-diffusion process. The main results of our paper are as follows: 1) formulas for the Esscher transform parameters which ensure that the martingale condition for the discounted foreign exchange rate is a martingale for a general Merton jump--diffusion process are derived; using the values of these parameters we proceeded to a risk-neural measure and provide new formulas for the distribution of jumps, the mean jump size, and the Poisson process intensity with respect to the measure; pricing formulas for European call foreign exchange options have been given as well; 2) obtained formulas are applied to the case of the exponential processes; 3) numerical simulations of European call foreign exchange option prices for different parameters are also provided; 4) codes for Matlab functions used in numerical simulations of option prices are given., Comment: 17 pages, 3 figures. arXiv admin note: substantial text overlap with arXiv:1402.1953

In this Data Note, the authors look at the average number of young adults with intellectual disabilities (ID) who between 2010 and 2014 exited vocational rehabilitation (VR) programs in each of the 50 states and the District of Columbia. To account for the different population sizes of the states, the authors divided these average numbers by the population of young adults with disabilities in the corresponding states, and then multiplied the result by 1,000. These figures are named "VR engagement rates." Compared to an earlier Data Note that examined the years 2002-2011 (ED587202), five new states reported substantially higher engagement rates in the period 2010-2014: Indiana, Mississippi, Nebraska, Tennessee, and Vermont. Also, Alaska, Arkansas, and New Mexico exited the group of states that, during the years 2002-2011, reported substantially lower engagement rates. [This is a publication of the Partnerships in Employment project and StateData.info--both projects of ThinkWork.org at the Institute for Community Inclusion, University of Massachusetts Boston.]