Search

Models that rely on subword tokenization have significant drawbacks, such as sensitivity to character-level noise like spelling errors and inconsistent compression rates across different languages and scripts. While character- or byte-level models like ByT5 attempt to address these concerns, they have not gained widespread adoption -- processing raw byte streams without tokenization results in significantly longer sequence lengths, making training and inference inefficient. This work introduces MrT5 (MergeT5), a more efficient variant of ByT5 that integrates a token deletion mechanism in its encoder to dynamically shorten the input sequence length. After processing through a fixed number of encoder layers, a learnt delete gate determines which tokens are to be removed and which are to be retained for subsequent layers. MrT5 effectively ``merges'' critical information from deleted tokens into a more compact sequence, leveraging contextual information from the remaining tokens. In continued pre-training experiments, we find that MrT5 can achieve significant gains in inference runtime with minimal effect on performance. When trained on English text, MrT5 demonstrates the capability to transfer its deletion feature zero-shot across several languages, with significant additional improvements following multilingual training. Furthermore, MrT5 shows comparable accuracy to ByT5 on downstream evaluations such as XNLI and character-level tasks while reducing sequence lengths by up to 80%. Our approach presents a solution to the practical limitations of existing byte-level models.

Characterizing the host galaxies of astrophysical transients is important to many areas of astrophysics, including constraining the progenitor systems of core-collapse supernovae, correcting Type Ia supernova distances, and probabilistically classifying transients without photometric or spectroscopic data. Given the increasing transient discovery rate in the coming years, there is substantial utility in providing public, transparent, reproducible, and automatic characterization for large samples of transient host galaxies. Here we present Blast, a web application that ingests live streams of transient alerts, matches transients to their host galaxies, and performs photometry on coincident archival imaging data of the host galaxy. The photometry is then used to infer both global host-galaxy properties and galaxy properties within 2 kpc of the transient location by using the Prospector Bayesian inference framework, with an acceleration in evaluation speed achieved via simulation-based inference. Blast provides host-galaxy properties to users via a web browser or an application program interface. The software can be extended to support alternative photometric or SED-fitting algorithms, and can be scaled via an asynchronous worker queue across multiple compute nodes to handle the processing of large volumes of transient alerts for upcoming transient surveys. Blast has been ingesting newly discovered transients from the Transient Name Server since mid-2024, and has currently measured SED parameters for more than 6000 transients. The service is publicly available at https://blast.scimma.org/., Comment: submitted to PASP

We use the Taylor-Wiles-Kisin patching method to investigate the multiplicities with which Hecke eigensystems appear in the mod-$\ell$ cohomology of unitary Shimura sets, associated to central simple algebras of the form $B=M_2(D)$, for $D$ a nonsplit quaternion algebra over a $CM$ field. We follow a similar strategy to the one used in our prior work for Shimura curves, exploiting the natural self-duality in this setting. Our method requires a careful analysis of certain irreducible components of local deformation rings. We introduce and analyze a new local model for local deformation rings specific to the case of a banal prime, which is significantly better behaved than the standard local models for local deformation rings. Our main result is a "multiplicity $2^a$ result", in the case where quaternion algebra $D$ ramifies only at banal primes, where $a$ is the number of places in the discriminant of $D$ which satisfy a certain Galois theoretic condition. We also prove an additional statement about the endomorphism ring of the cohomology of the Shimura set, which has an application to the study of congruence modules., Comment: 77 pages, comments welcome

The dearth of high quality spectroscopy of dusty star-forming galaxies (DSFGs) -- the main drivers of the assembly of dust and stellar mass at the peak of activity in the Universe -- greatly hinders our ability to interpret their physical processes and evolutionary pathways. We present JWST/NIRSpec observations from RUBIES of four submillimeter-selected, ALMA-detected DSFGs at cosmic noon, $z\sim2.3-2.7$. While photometry uniformly suggests vigorous ongoing star formation for the entire sample in line with canonical DSFGs, the spectra differ: one source has spectroscopic evidence of an evolved stellar population, indicating a recent transition to a post-starburst phase, while the remainder show strong spectroscopic signatures of ongoing starbursts. All four galaxies are infrared-luminous (log$_{10}$$L_{\rm{IR}}$/L$_{\rm \odot}$ $>12.4$), massive (log$_{10}\,M_\star$/M$_{\rm \odot}$ $>11$), and very dust-obscured ($A_V\sim3-4$ ABmag). Leveraging detections of multiple Balmer and Paschen lines, we derive an optical attenuation curve consistent with Calzetti overall, yet an optical extinction ratio $R_V\sim2.5$, potentially indicating smaller dust grains or differences in star-dust geometry. This case study provides some of the first detailed spectroscopic evidence that the DSFGs encompass a heterogeneous sample spanning a range of star formation properties and evolutionary stages, and illustrates the advantages of synergistic JWST and ALMA analysis of DSFGs., Comment: 20 pages, 4 figures; submitted to ApJ

Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality., Comment: Code, docs, weight, benchmark and training data are all avaliable at \href{https://rese1f.github.io/aurora-web/}{website}

We introduce NNetscape Navigator (NNetnav), a method for training web agents entirely through synthetic demonstrations. These demonstrations are collected by first interacting with a browser to generate trajectory rollouts, which are then retroactively labeled into instructions using a language model. Most work on training browser agents has relied on expensive human supervision, and the limited previous work on such interaction-first synthetic data techniques has failed to provide effective search through the exponential space of exploration. In contrast, NNetnav exploits the hierarchical structure of language instructions to make this search more tractable: complex instructions are typically decomposable into simpler subtasks, allowing NNetnav to automatically prune interaction episodes when an intermediate trajectory cannot be annotated with a meaningful sub-task. We use NNetnav demonstrations from a language model for supervised fine-tuning of a smaller language model policy, and find improvements of 6 points on WebArena and over 20 points on MiniWoB++, two popular environments for web-agents. Notably, on WebArena, we observe that language model policies can be further enhanced when fine-tuned with NNetnav demonstrations derived from the same language model. Finally, we collect and release a dataset of over 6k NNetnav demonstrations on WebArena, spanning a diverse and complex set of instructions., Comment: Preprint. Under Review

We present the PANORAMIC survey, a pure parallel extragalactic imaging program with NIRCam observed during JWST Cycle 1. The survey obtained $\sim$530 sq arcmin of NIRCam imaging from 1-5$\mu$m, totaling $\sim$192 hours of science integration time. This represents the largest on-sky time investment of any Cycle 1 GO extragalactic NIRCam imaging program by nearly a factor of 2. The survey includes $\sim$432 sq arcmin of novel sky area not yet observed with JWST using at least $6$ NIRCam broad-band filters, increasing the existing area covered by similar Cycle 1 data by $\sim$60%. 70 square arcmin was also covered by a 7th filter (F410M). A fraction of PANORAMIC data ($\sim$200 sq arcmin) was obtained in or around extragalactic deep-fields, enhancing their legacy value. Pure parallel observing naturally creates a wedding cake survey with both wide and ultra-deep tiers, with 5$\sigma$ point source depths at F444W ranging from 27.8-29.4 (ABmag), and with minimized cosmic variance. The 6+ filter observing setup yields remarkably good photometric redshift performance, achieving similar median scatter and outlier fraction as CANDELS ($\sigma_{\rm NMAD}\sim0.07$; $\eta\sim0.2$), which enables a wealth of science across redshift without the need for followup or ancillary data. We overview the proposed survey, the data obtained as part of this program, and document the science-ready data products in the first data release. PANORAMIC has delivered wide-area and deep imaging with excellent photometric performance, demonstrating that pure parallel observations with JWST are a highly efficient observing mode that is key to acquiring a complete picture of galaxy evolution from rare bright galaxies to fainter, more abundant sources at all redshifts., Comment: Submitted to AAS Journals; comments welcome. Initial data release available at https://panoramic-jwst.github.io/

The expensive fine-grained annotation and data scarcity have become the primary obstacles for the widespread adoption of deep learning-based Whole Slide Images (WSI) classification algorithms in clinical practice. Unlike few-shot learning methods in natural images that can leverage the labels of each image, existing few-shot WSI classification methods only utilize a small number of fine-grained labels or weakly supervised slide labels for training in order to avoid expensive fine-grained annotation. They lack sufficient mining of available WSIs, severely limiting WSI classification performance. To address the above issues, we propose a novel and efficient dual-tier few-shot learning paradigm for WSI classification, named FAST. FAST consists of a dual-level annotation strategy and a dual-branch classification framework. Firstly, to avoid expensive fine-grained annotation, we collect a very small number of WSIs at the slide level, and annotate an extremely small number of patches. Then, to fully mining the available WSIs, we use all the patches and available patch labels to build a cache branch, which utilizes the labeled patches to learn the labels of unlabeled patches and through knowledge retrieval for patch classification. In addition to the cache branch, we also construct a prior branch that includes learnable prompt vectors, using the text encoder of visual-language models for patch classification. Finally, we integrate the results from both branches to achieve WSI classification. Extensive experiments on binary and multi-class datasets demonstrate that our proposed method significantly surpasses existing few-shot classification methods and approaches the accuracy of fully supervised methods with only 0.22$\%$ annotation costs. All codes and models will be publicly available on https://github.com/fukexue/FAST., Comment: Accepted to NeurIPS 2024

Instruction tuning commonly means finetuning a language model on instruction-response pairs. We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning. We first find that instruction-response pairs are not necessary: training solely on responses, without any corresponding instructions, yields instruction following. This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses. However, we then find it's not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation. In particular, when instructions are very different from those in the narrow finetuning domain, models' responses do not adhere to the style of the finetuning domain. To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model's distribution yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words' probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.

In this paper we explore independently for the first time three chemical potentials (baryon $\mu_B$, charged $\mu_Q$, and strange $\mu_S$) in the Chiral mean-field (CMF) model. We designed and implemented \texttt{CMF++}, a new version of the CMF model rewritten in \texttt{C++} that is optimized, modular, and well-documented. \texttt{CMF++} has been integrated into the MUSES Calculation Engine as a free and open source software module. The runtime improved in more than 4 orders of magnitude across all 3 chemical potentials, when compared to the legacy code. Here we focus on the zero temperature case and study stable, as well as metastable and unstable, vacuum, hadronic, and quark phases, showing how phase boundaries vary with the different chemical potentials. Due to the significant numerical improvements in \texttt{CMF++}, we can calculate for the first time high-order susceptibilities within the CMF framework to study the properties of the quark deconfinement phase transition. We found phases of matter that include a light hadronic phase, strangeness-dominated hadronic phase, and quark deconfinement within our $\mu_B$, $\mu_S$, $\mu_Q$ phase space. The phase transitions are of first, second (quantum critical point), and third order between these phases and we even identified a tricritical point., Comment: 63 pages, 22 figures

One of the greatest challenges in galaxy evolution over the last decade has been constraining the prevalence of heavily dust-obscured galaxies in the early Universe. At $z>3$, these galaxies are increasingly rare, and difficult to identify as they are interspersed among the more numerous dust-obscured galaxy population at $z=1-3$, making efforts to secure confident spectroscopic redshifts expensive, and sometimes unsuccessful. In this work, we present the Extended Mapping Obscuration to Reionization with ALMA (Ex-MORA) Survey -- a 2mm blank-field survey in the COSMOS-Web field, and the largest ever ALMA blank-field survey to-date covering 577 arcmin$^2$. Ex-MORA is an expansion of the MORA survey designed to identify primarily $z>3$ dusty, star-forming galaxies while simultaneously filtering out the more numerous $z<3$ population by leveraging the very negative $K$-correction at observed-frame 2mm. We identify 37 significant ($>$5$\sigma$) sources, 33 of which are robust thermal dust emitters. We measure a median redshift of $\langle z \rangle = 3.6^{+0.1}_{-0.2}$, with two-thirds of the sample at $z>3$, and just under half at $z>4$, demonstrating the overall success of the 2mm-selection technique. The integrated $z>3$ volume density of Ex-MORA sources is $\sim1-3\times10^{-5}$ Mpc$^{-3}$, consistent with other surveys of infrared luminous galaxies at similar epochs. We also find that techniques using rest-frame optical emission (or lack thereof) to identify $z>3$ heavily dust-obscured galaxies miss at least half of Ex-MORA galaxies. This supports the idea that the dusty galaxy population is heterogeneous, and that synergies across observatories spanning multiple energy regimes are critical to understanding their formation and evolution at $z>3$., Comment: 13 pages, 5 figures, submitted to ApJ; fully reduced mosaic will be shared upon publication

The Linear Representation Hypothesis (LRH) states that neural networks learn to encode concepts as directions in activation space, and a strong version of the LRH states that models learn only such encodings. In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction. These representations have layered features that are impossible to locate in distinct linear subspaces. To show this, we train interventions to predict and manipulate tokens by learning the scaling factor corresponding to each sequence position. These interventions indicate that the smallest RNNs find only this magnitude-based solution, while larger RNNs have linear representations. These findings strongly indicate that interpretability research should not be confined by the LRH.

JWST has enabled detecting and spatially resolving the heavily dust-attenuated stellar populations of sub-millimeter galaxies, revealing detail that was previously inaccessible. In this work we construct a sample of 289 sub-millimeter galaxies with detailed joint ALMA and JWST constraints in the COSMOS field. Sources are originally selected using the SCUBA-2 instrument and have archival ALMA observations from various programs. Their JWST NIRCam imaging is from COSMOS-Web and PRIMER. We extract multi-wavelength photometry in a manner that leverages the unprecedented near-infrared spatial resolution of JWST, and fit the data with spectral energy distribution models to derive photometric redshifts, stellar masses, star-formation rates and optical attenuation. The sample has an average z=2.6, A_V=2.5, SFR=270 and log(M*)=11.1. There are 81 (30%) galaxies that have no previous optical/near-infrared detections, including 75% of the z>4 sub-sample (n=28). The faintest observed near-infrared sources have the highest redshifts and largest A_V=4. In a preliminary morphology analysis we find that ~10% of our sample exhibit spiral arms and 5% host stellar bars, with one candidate bar found at z>3. Finally, we find that the clustering of JWST galaxies within 10 arcseconds of a sub-mm galaxy is a factor of 2 greater than what is expected based on either random clustering or the distribution of sources around any red galaxy irrespective of a sub-mm detection., Comment: 37 pages (15 for RGBs + references), 14 figures, submitted to ApJ

Prior coarticulation studies focus mainly on limited phonemic sequences and specific articulators, providing only approximate descriptions of the temporal extent and magnitude of coarticulation. This paper is an initial attempt to comprehensively investigate coarticulation. We leverage existing Electromagnetic Articulography (EMA) datasets to develop and train a phoneme-to-articulatory (P2A) model that can generate realistic EMA for novel phoneme sequences and replicate known coarticulation patterns. We use model-generated EMA on 9K minimal word pairs to analyze coarticulation magnitude and extent up to eight phonemes from the coarticulation trigger, and compare coarticulation resistance across different consonants. Our findings align with earlier studies and suggest a longer-range coarticulation effect than previously found. This model-based approach can potentially compare coarticulation between adults and children and across languages, offering new insights into speech production., Comment: To be published in Interspeech 2024

Results are presented for the measurement of large-scale anisotropies in the arrival directions of ultra-high-energy cosmic rays detected at the Pierre Auger Observatory during 19 years of operation, prior to AugerPrime, the upgrade of the Observatory. The 3D dipole amplitude and direction are reconstructed above $4\,$EeV in four energy bins. Besides the established dipolar anisotropy in right ascension above $8\,$EeV, the Fourier amplitude of the $8$ to $16\,$EeV energy bin is now also above the $5\sigma$ discovery level. No time variation of the dipole moment above $8\,$EeV is found, setting an upper limit to the rate of change of such variations of $0.3\%$ per year at the $95\%$ confidence level. Additionally, the results for the angular power spectrum are shown, demonstrating no other statistically significant multipoles. The results for the equatorial dipole component down to $0.03\,$EeV are presented, using for the first time a data set obtained with a trigger that has been optimized for lower energies. Finally, model predictions are discussed and compared with observations, based on two source emission scenarios obtained in the combined fit of spectrum and composition above $0.6\,$EeV., Comment: Minor modifications, accepted for publication in Astrophysical Journal

The safety of Large Language Models (LLMs) remains a critical concern due to a lack of adequate benchmarks for systematically evaluating their ability to resist generating harmful content. Previous efforts towards automated red teaming involve static or templated sets of illicit requests and adversarial prompts which have limited utility given jailbreak attacks' evolving and composable nature. We propose a novel dynamic benchmark of composable jailbreak attacks to move beyond static datasets and taxonomies of attacks and harms. Our approach consists of three components collectively called h4rm3l: (1) a domain-specific language that formally expresses jailbreak attacks as compositions of parameterized prompt transformation primitives, (2) bandit-based few-shot program synthesis algorithms that generate novel attacks optimized to penetrate the safety filters of a target black box LLM, and (3) open-source automated red-teaming software employing the previous two components. We use h4rm3l to generate a dataset of 2656 successful novel jailbreak attacks targeting 6 state-of-the-art (SOTA) open-source and proprietary LLMs. Several of our synthesized attacks are more effective than previously reported ones, with Attack Success Rates exceeding 90% on SOTA closed language models such as claude-3-haiku and GPT4-o. By generating datasets of jailbreak attacks in a unified formal representation, h4rm3l enables reproducible benchmarking and automated red-teaming, contributes to understanding LLM safety limitations, and supports the development of robust defenses in an increasingly LLM-integrated world. Warning: This paper and related research artifacts contain offensive and potentially disturbing prompts and model-generated content.

Multiple instance learning (MIL) problem is currently solved from either bag-classification or instance-classification perspective, both of which ignore important information contained in some instances and result in limited performance. For example, existing methods often face difficulty in learning hard positive instances. In this paper, we formulate MIL as a semi-supervised instance classification problem, so that all the labeled and unlabeled instances can be fully utilized to train a better classifier. The difficulty in this formulation is that all the labeled instances are negative in MIL, and traditional self-training techniques used in semi-supervised learning tend to degenerate in generating pseudo labels for the unlabeled instances in this scenario. To resolve this problem, we propose a weakly-supervised self-training method, in which we utilize the positive bag labels to construct a global constraint and a local constraint on the pseudo labels to prevent them from degenerating and force the classifier to learn hard positive instances. It is worth noting that easy positive instances are instances are far from the decision boundary in the classification process, while hard positive instances are those close to the decision boundary. Through iterative optimization, the pseudo labels can gradually approach the true labels. Extensive experiments on two MNIST synthetic datasets, five traditional MIL benchmark datasets and two histopathology whole slide image datasets show that our method achieved new SOTA performance on all of them. The code will be publicly available.

Teaching Computer Science (CS) by having students write programs by hand on paper has key pedagogical advantages: It allows focused learning and requires careful thinking compared to the use of Integrated Development Environments (IDEs) with intelligent support tools or "just trying things out". The familiar environment of pens and paper also lessens the cognitive load of students with no prior experience with computers, for whom the mere basic usage of computers can be intimidating. Finally, this teaching approach opens learning opportunities to students with limited access to computers. However, a key obstacle is the current lack of teaching methods and support software for working with and running handwritten programs. Optical character recognition (OCR) of handwritten code is challenging: Minor OCR errors, perhaps due to varied handwriting styles, easily make code not run, and recognizing indentation is crucial for languages like Python but is difficult to do due to inconsistent horizontal spacing in handwriting. Our approach integrates two innovative methods. The first combines OCR with an indentation recognition module and a language model designed for post-OCR error correction without introducing hallucinations. This method, to our knowledge, surpasses all existing systems in handwritten code recognition. It reduces error from 30\% in the state of the art to 5\% with minimal hallucination of logical fixes to student programs. The second method leverages a multimodal language model to recognize handwritten programs in an end-to-end fashion. We hope this contribution can stimulate further pedagogical research and contribute to the goal of making CS education universally accessible. We release a dataset of handwritten programs and code to support future research at https://github.com/mdoumbouya/codeocr

In an online experiment with N = 1893 participants, we collected and analyzed over 18,000 prompts and over 300,000 images to explore how the importance of prompting will change as the capabilities of generative AI models continue to improve. Each participant in our experiment was randomly and blindly assigned to use one of three text-to-image diffusion models: DALL-E 2, its more advanced successor DALL-E 3, or a version of DALL-E 3 with automatic prompt revision. Participants were then asked to write prompts to reproduce a target image as closely as possible in 10 consecutive tries. We find that task performance was higher for participants using DALL-E 3 than for those using DALL-E 2. This performance gap corresponds to a noticeable difference in the similarity of participants' images to their target images, and was caused in equal measure by: (1) the increased technical capabilities of DALL-E 3, and (2) endogenous changes in participants' prompting in response to these increased capabilities. More specifically, despite being blind to the model they were assigned, participants assigned to DALL-E 3 wrote longer prompts that were more semantically similar to each other and contained a greater number of descriptive words. Furthermore, while participants assigned to DALL-E 3 with prompt revision still outperformed those assigned to DALL-E 2, automatic prompt revision reduced the benefits of using DALL-E 3 by 58\%. Taken together, our results suggest that as models continue to progress, people will continue to adapt their prompts to take advantage of new models' capabilities.

Our goal is to develop a design framework for multifunctional mechanical metamaterials that can tune their rigidity while optimizing other desired properties. Towards this goal, we first demonstrate that underconstrained central force networks possess a critical rigidity manifold of codimension one in the space of their physical constraints. We describe how the geometry of this manifold generates a natural parameterization in terms of the states of self-stress, and then use this parameterization to numerically generate disordered network structures that are on the critical rigidity manifold and also optimize various objective functions, such as maximizing the bulk stiffness under dilation, or minimizing length variance to find networks that can be self-assembled from equal-length parts. This framework can be used to design mechanical metamaterials that can tune their rigidity and also exhibit other desired properties., Comment: 19 pages, 8 figures

Ultra-high-energy cosmic rays are known to be mainly of extragalactic origin, and their propagation is limited by energy losses, so their arrival directions are expected to correlate with the large-scale structure of the local Universe. In this work, we investigate the possible presence of intermediate-scale excesses in the flux of the most energetic cosmic rays from the direction of the supergalactic plane region using events with energies above 20 EeV recorded with the surface detector array of the Pierre Auger Observatory up to 31 December 2022, with a total exposure of 135,000 km^2 sr yr. The strongest indication for an excess that we find, with a post-trial significance of 3.1{\sigma}, is in the Centaurus region, as in our previous reports, and it extends down to lower energies than previously studied. We do not find any strong hints of excesses from any other region of the supergalactic plane at the same angular scale. In particular, our results do not confirm the reports by the Telescope Array collaboration of excesses from two regions in the Northern Hemisphere at the edge of the field of view of the Pierre Auger Observatory. With a comparable exposure, our results in those regions are in good agreement with the expectations from an isotropic distribution., Comment: submitted to ApJ

JWST has revealed a ubiquitous population of ``little red dots'' (LRDs) at $z\gtrsim4$, selected via their red rest-frame optical emission and compact morphologies. They are thought to be reddened by dust, whether in tori of active galactic nuclei or the interstellar medium (ISM), though none have direct dust detections to date. Informed by the average characteristics of 675 LRDs drawn from the literature, we provide ballpark constraints on the dust characteristics of the LRD population and estimate they have average dust masses of $\langle M_{\rm dust}\rangle=(1.6^{+4.8}_{-0.9})\times10^{4} M_\odot$, luminosities of $\langle L_{\rm IR}\rangle=(8^{+3}_{-5})\times10^{10} L_\odot$ and temperatures of $\langle T_{\rm dust}\rangle=110^{+21}_{-36}$ K. Notably, the spectral energy distributions are thought to peak at $\sim$100 K (rest-frame 20-30 $\mu$m) regardless of heating mechanism, whether AGN or star formation. LRDs' compact sizes $R_{\rm eff}\sim100$ pc are the dominant factor contributing to their low estimated dust masses. Our predictions likely mean LRDs have, on average, submillimeter emission a factor of $\sim$100$\times$ fainter than current ALMA limits provide. The star-to-dust ratio is a factor $\sim$100$\times$ larger than expected from dust formation models if one assumes the rest-optical light is dominated by stars; this suggests stars do not dominate. Despite their high apparent volume density, LRDs contribute negligibly (0.1%) to the cosmic dust budget at $z\gtrsim4$ due to their low dust masses., Comment: 11 pages, 3 figures; accepted to ApJL

Given a hyperbolic group $G$ and a maximal infinite cyclic subgroup $\langle g \rangle$, we define a drilling of $G$ along $g$, which is a relatively hyperbolic group pair $(\widehat{G}, P)$. This is inspired by the well-studied procedure of drilling a hyperbolic 3-manifold along an embedded geodesic. We prove that, under suitable conditions, a hyperbolic group with $2$-sphere boundary admits a drilling where the resulting relatively hyperbolic group pair $(\widehat{G}, P)$ has relatively hyperbolic boundary $S^2$. This allows us to reduce the Cannon Conjecture (in the residually finite case) to a relative version., Comment: 83 pages, 2 figures. Comments welcome

Static word embeddings are ubiquitous in computational social science applications and contribute to practical decision-making in a variety of fields including law and healthcare. However, assessing the statistical uncertainty in downstream conclusions drawn from word embedding statistics has remained challenging. When using only point estimates for embeddings, researchers have no streamlined way of assessing the degree to which their model selection criteria or scientific conclusions are subject to noise due to sparsity in the underlying data used to generate the embeddings. We introduce a method to obtain approximate, easy-to-use, and scalable reconstruction error variance estimates for GloVe (Pennington et al., 2014), one of the most widely used word embedding models, using an analytical approximation to a multivariate normal model. To demonstrate the value of embeddings with variance (GloVe-V), we illustrate how our approach enables principled hypothesis testing in core word embedding tasks, such as comparing the similarity between different word pairs in vector space, assessing the performance of different models, and analyzing the relative degree of ethnic or gender bias in a corpus using different word lists.

The Pierre Auger Observatory is the most sensitive instrument to detect photons with energies above $10^{17}$ eV. It measures extensive air showers generated by ultra high energy cosmic rays using a hybrid technique that exploits the combination of a fluorescence detector with a ground array of particle detectors. The signatures of a photon-induced air shower are a larger atmospheric depth of the shower maximum ($X_{max}$) and a steeper lateral distribution function, along with a lower number of muons with respect to the bulk of hadron-induced cascades. In this work, a new analysis technique in the energy interval between 1 and 30 EeV (1 EeV = $10^{18}$ eV) has been developed by combining the fluorescence detector-based measurement of $X_{max}$ with the specific features of the surface detector signal through a parameter related to the air shower muon content, derived from the universality of the air shower development. No evidence of a statistically significant signal due to photon primaries was found using data collected in about 12 years of operation. Thus, upper bounds to the integral photon flux have been set using a detailed calculation of the detector exposure, in combination with a data-driven background estimation. The derived 95% confidence level upper limits are 0.0403, 0.01113, 0.0035, 0.0023, and 0.0021 km$^{-2}$ sr$^{-1}$ yr$^{-1}$ above 1, 2, 3, 5, and 10 EeV, respectively, leading to the most stringent upper limits on the photon flux in the EeV range. Compared with past results, the upper limits were improved by about 40% for the lowest energy threshold and by a factor 3 above 3 EeV, where no candidates were found and the expected background is negligible. The presented limits can be used to probe the assumptions on chemical composition of ultra-high energy cosmic rays and allow for the constraint of the mass and lifetime phase space of super-heavy dark matter particles., Comment: 19 pages, 22 figures

We report an investigation of the mass composition of cosmic rays with energies from 3 to 100 EeV (1 EeV=$10^{18}$ eV) using the distributions of the depth of shower maximum $X_\mathrm{max}$. The analysis relies on ${\sim}50,000$ events recorded by the Surface Detector of the Pierre Auger Observatory and a deep-learning-based reconstruction algorithm. Above energies of 5 EeV, the data set offers a 10-fold increase in statistics with respect to fluorescence measurements at the Observatory. After cross-calibration using the Fluorescence Detector, this enables the first measurement of the evolution of the mean and the standard deviation of the $X_\mathrm{max}$ distributions up to 100 EeV. Our findings are threefold: (1.) The evolution of the mean logarithmic mass towards a heavier composition with increasing energy can be confirmed and is extended to 100 EeV. (2.) The evolution of the fluctuations of $X_\mathrm{max}$ towards a heavier and purer composition with increasing energy can be confirmed with high statistics. We report a rather heavy composition and small fluctuations in $X_\mathrm{max}$ at the highest energies. (3.) We find indications for a characteristic structure beyond a constant change in the mean logarithmic mass, featuring three breaks that are observed in proximity to the ankle, instep, and suppression features in the energy spectrum., Comment: submitted to Phys. Rev. D, 28 pages, 18 figures, 5 tables

We present measurements of the atmospheric depth of the shower maximum $X_\mathrm{max}$, inferred for the first time on an event-by-event level using the Surface Detector of the Pierre Auger Observatory. Using deep learning, we were able to extend measurements of the $X_\mathrm{max}$ distributions up to energies of 100 EeV ($10^{20}$ eV), not yet revealed by current measurements, providing new insights into the mass composition of cosmic rays at extreme energies. Gaining a 10-fold increase in statistics compared to the Fluorescence Detector data, we find evidence that the rate of change of the average $X_\mathrm{max}$ with the logarithm of energy features three breaks at $6.5\pm0.6~(\mathrm{stat})\pm1~(\mathrm{sys})$ EeV, $11\pm 2~(\mathrm{stat})\pm1~(\mathrm{sys})$ EeV, and $31\pm5~(\mathrm{stat})\pm3~(\mathrm{sys})$ EeV, in the vicinity to the three prominent features (ankle, instep, suppression) of the cosmic-ray flux. The energy evolution of the mean and standard deviation of the measured $X_\mathrm{max}$ distributions indicates that the mass composition becomes increasingly heavier and purer, thus being incompatible with a large fraction of light nuclei between 50 EeV and 100 EeV., Comment: submitted to Phys. Rev. Lett., 10 pages, 3 figures, 1 table

Amid growing concerns over AI's societal risks--ranging from civilizational collapse to misinformation and systemic bias--this study explores the perceptions of AI experts and the general US registered voters on the likelihood and impact of 18 specific AI risks, alongside their policy preferences for managing these risks. While both groups favor international oversight over national or corporate governance, our survey reveals a discrepancy: voters perceive AI risks as both more likely and more impactful than experts, and also advocate for slower AI development. Specifically, our findings indicate that policy interventions may best assuage collective concerns if they attempt to more carefully balance mitigation efforts across all classes of societal-scale risks, effectively nullifying the near-vs-long-term debate over AI risks. More broadly, our results will serve not only to enable more substantive policy discussions for preventing and mitigating AI risks, but also to underscore the challenge of consensus building for effective policy implementation., Comment: 9 pages, 18 page supplementary materials

Legal practice has witnessed a sharp rise in products incorporating artificial intelligence (AI). Such tools are designed to assist with a wide range of core legal tasks, from search and summarization of caselaw to document drafting. But the large language models used in these tools are prone to "hallucinate," or make up false information, making their use risky in high-stakes domains. Recently, certain legal research providers have touted methods such as retrieval-augmented generation (RAG) as "eliminating" (Casetext, 2023) or "avoid[ing]" hallucinations (Thomson Reuters, 2023), or guaranteeing "hallucination-free" legal citations (LexisNexis, 2023). Because of the closed nature of these systems, systematically assessing these claims is challenging. In this article, we design and report on the first preregistered empirical evaluation of AI-driven legal research tools. We demonstrate that the providers' claims are overstated. While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time. We also document substantial differences between systems in responsiveness and accuracy. Our article makes four key contributions. It is the first to assess and report the performance of RAG-based proprietary legal AI tools. Second, it introduces a comprehensive, preregistered dataset for identifying and understanding vulnerabilities in these systems. Third, it proposes a clear typology for differentiating between hallucinations and accurate legal responses. Last, it provides evidence to inform the responsibilities of legal professionals in supervising and verifying AI outputs, which remains a central open question for the responsible integration of AI into law., Comment: Our dataset, tool outputs, and labels will be made available upon publication. This version of the manuscript (May 30, 2024) is updated to reflect an evaluation of Westlaw's AI-Assisted Research

Gliomas are the most common malignant primary brain tumors in adults and one of the deadliest types of cancer. There are many challenges in treatment and monitoring due to the genetic diversity and high intrinsic heterogeneity in appearance, shape, histology, and treatment response. Treatments include surgery, radiation, and systemic therapies, with magnetic resonance imaging (MRI) playing a key role in treatment planning and post-treatment longitudinal assessment. The 2024 Brain Tumor Segmentation (BraTS) challenge on post-treatment glioma MRI will provide a community standard and benchmark for state-of-the-art automated segmentation models based on the largest expert-annotated post-treatment glioma MRI dataset. Challenge competitors will develop automated segmentation models to predict four distinct tumor sub-regions consisting of enhancing tissue (ET), surrounding non-enhancing T2/fluid-attenuated inversion recovery (FLAIR) hyperintensity (SNFH), non-enhancing tumor core (NETC), and resection cavity (RC). Models will be evaluated on separate validation and test datasets using standardized performance metrics utilized across the BraTS 2024 cluster of challenges, including lesion-wise Dice Similarity Coefficient and Hausdorff Distance. Models developed during this challenge will advance the field of automated MRI segmentation and contribute to their integration into clinical practice, ultimately enhancing patient care., Comment: 10 pages, 4 figures, 1 table

Previous work on Universal Transformers (UTs) has demonstrated the importance of parameter sharing across layers. By allowing recurrence in depth, UTs have advantages over standard Transformers in learning compositional generalizations, but layer-sharing comes with a practical limitation of parameter-compute ratio: it drastically reduces the parameter count compared to the non-shared model with the same dimensionality. Naively scaling up the layer size to compensate for the loss of parameters makes its computational resource requirements prohibitive. In practice, no previous work has succeeded in proposing a shared-layer Transformer design that is competitive in parameter count-dominated tasks such as language modeling. Here we propose MoEUT (pronounced "moot"), an effective mixture-of-experts (MoE)-based shared-layer Transformer architecture, which combines several recent advances in MoEs for both feedforward and attention layers of standard Transformers together with novel layer-normalization and grouping schemes that are specific and crucial to UTs. The resulting UT model, for the first time, slightly outperforms standard Transformers on language modeling tasks such as BLiMP and PIQA, while using significantly less compute and memory., Comment: Accepted to NeurIPS 2024

Searching dependency graphs and manipulating them can be a time consuming and challenging task to get right. We document Semgrex, a system for searching dependency graphs, and introduce Ssurgeon, a system for manipulating the output of Semgrex. The compact language used by these systems allows for easy command line or API processing of dependencies. Additionally, integration with publicly released toolkits in Java and Python allows for searching text relations and attributes over natural text., Comment: Georgetown University Round Table (GURT) 2023

Large language models, such as GPT-4 and Med-PaLM, have shown impressive performance on clinical tasks; however, they require access to compute, are closed-source, and cannot be deployed on device. Mid-size models such as BioGPT-large, BioMedLM, LLaMA 2, and Mistral 7B avoid these drawbacks, but their capacity for clinical tasks has been understudied. To help assess their potential for clinical use and help researchers decide which model they should use, we compare their performance on two clinical question-answering (QA) tasks: MedQA and consumer query answering. We find that Mistral 7B is the best performing model, winning on all benchmarks and outperforming models trained specifically for the biomedical domain. While Mistral 7B's MedQA score of 63.0% approaches the original Med-PaLM, and it often can produce plausible responses to consumer health queries, room for improvement still exists. This study provides the first head-to-head assessment of open source mid-sized models on clinical tasks., Comment: 25 pages, 8 figures

The vast majority of the popular English named entity recognition (NER) datasets contain American or British English data, despite the existence of many global varieties of English. As such, it is unclear whether they generalize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze NER model performance on low-resource English variants from around the world. We test widely used NER toolkits and transformer models, including models using the pre-trained contextual models RoBERTa and ELECTRA, on three datasets: a commonly used British English newswire dataset, CoNLL 2003, a more American focused dataset OntoNotes, and our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops-over 10 F1 in some cases-when tested on the Worldwide English dataset. Upon examination of region-specific errors, we observe the greatest performance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a combined model trained on the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2 F1 on both test sets., Comment: EMNLP Findings 2023

We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent advances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a language to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are both proposed and tested by the system, finding evidence for some and not others. We provide evidence that the insights from these simulations of social interactions are not available to the LLM purely through direct elicitation. When given its proposed structural causal model for each scenario, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those estimates. In the auction experiment, the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from the LLM are inaccurate. However, the LLM's predictions are dramatically improved if the model can condition on the fitted structural causal model. In short, the LLM knows more than it can (immediately) tell.

Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. We release a generic ReFT training library publicly at https://github.com/stanfordnlp/pyreft., Comment: preprint

The flux of ultra-high energy cosmic rays reaching Earth above the ankle energy (5 EeV) can be described as a mixture of nuclei injected by extragalactic sources with very hard spectra and a low rigidity cutoff. Extragalactic magnetic fields existing between the Earth and the closest sources can affect the observed CR spectrum by reducing the flux of low-rigidity particles reaching Earth. We perform a combined fit of the spectrum and distributions of depth of shower maximum measured with the Pierre Auger Observatory including the effect of this magnetic horizon in the propagation of UHECRs in the intergalactic space. We find that, within a specific range of the various experimental and phenomenological systematics, the magnetic horizon effect can be relevant for turbulent magnetic field strengths in the local neighbourhood of order $B_{\rm rms}\simeq (50-100)\,{\rm nG}\,(20\rm{Mpc}/{d_{\rm s})( 100\,\rm{kpc}/L_{\rm coh}})^{1/2}$, with $d_{\rm s}$ the typical intersource separation and $L_{\rm coh}$ the magnetic field coherence length. When this is the case, the inferred slope of the source spectrum becomes softer and can be closer to the expectations of diffusive shock acceleration, i.e., $\propto E^{-2}$. An additional cosmic-ray population with higher source density and softer spectra, presumably also extragalactic and dominating the cosmic-ray flux at EeV energies, is also required to reproduce the overall spectrum and composition results for all energies down to 0.6~EeV., Comment: Matches published version

Instruction tuning is an important step in making language models useful for direct user interaction. However, many legal tasks remain out of reach for most open LLMs and there do not yet exist any large scale instruction datasets for the domain. This critically limits research in this application area. In this work, we curate LawInstruct, a large legal instruction dataset, covering 17 jurisdictions, 24 languages and a total of 12M examples. We present evidence that domain-specific pretraining and instruction tuning improve performance on LegalBench, including improving Flan-T5 XL by 8 points or 16\% over the baseline. However, the effect does not generalize across all tasks, training regimes, model sizes, and other factors. LawInstruct is a resource for accelerating the development of models with stronger information processing and decision making capabilities in the legal domain.

Magnon spin currents in the ferrimagnetic garnet Tb3Fe5O12 with 4f electrons were examined through the spin-Seebeck effect and neutron scattering measurements. The compound shows a magnetic compensation, where the spin-Seebeck signal reverses above and below Tcomp = 249.5(4) K. Unpolarized neutron scattering unveils two major magnon branches with finite energy gaps, which are well-explained in the framework of spin-wave theory. Their temperature dependencies and the direction of the precession motion of magnetic moments, i.e. magnon polarization, defined using polarized neutrons, explain the reversal at Tcomp and decay of the spin-Seebeck signals at low temperatures. We illustrate an example that momentum- and energy-resolved microscopic information is a prerequisite to understanding the magnon spin current., Comment: 7 pages, 5 figures

Scientific publishing lays the foundation of science by disseminating research findings, fostering collaboration, encouraging reproducibility, and ensuring that scientific knowledge is accessible, verifiable, and built upon over time. Recently, there has been immense speculation about how many people are using large language models (LLMs) like ChatGPT in their academic writing, and to what extent this tool might have an effect on global scientific practices. However, we lack a precise measure of the proportion of academic writing substantially modified or produced by LLMs. To address this gap, we conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals, using a population-level statistical framework to measure the prevalence of LLM-modified content over time. Our statistical estimation operates on the corpus level and is more robust than inference on individual instances. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers (up to 17.5%). In comparison, Mathematics papers and the Nature portfolio showed the least LLM modification (up to 6.3%). Moreover, at an aggregate level, our analysis reveals that higher levels of LLM-modification are associated with papers whose first authors post preprints more frequently, papers in more crowded research areas, and papers of shorter lengths. Our findings suggests that LLMs are being broadly used in scientific writings.

Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks. However, these models have hundreds of billions of parameters, are computationally expensive to run, require users to send their input data over the internet, and are trained on unknown data sources. Can smaller, more targeted models compete? To address this question, we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with much larger models, such as achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics. This demonstrates that smaller models can potentially serve as transparent, privacy-preserving, economical and environmentally friendly foundations for particular NLP applications, such as in biomedicine. The model is available on the Hugging Face Hub: https://huggingface.co/stanford-crfm/BioMedLM., Comment: 23 pages

Following natural language instructions by executing actions in digital environments (e.g. web-browsers and REST APIs) is a challenging task for language model (LM) agents. Unfortunately, LM agents often fail to generalize to new environments without human demonstrations. This work presents BAGEL, a method for bootstrapping LM agents without human supervision. BAGEL converts a seed set of randomly explored trajectories or synthetic instructions, into demonstrations, via round-trips between two noisy LM components: an LM labeler which converts a trajectory into a synthetic instruction, and a zero-shot LM agent which maps the synthetic instruction into a refined trajectory. By performing these round-trips iteratively, BAGEL quickly converts the initial distribution of trajectories towards those that are well-described by natural language. We use BAGEL demonstrations to adapt a zero shot LM agent at test time via in-context learning over retrieved demonstrations, and find improvements of over 2-13% absolute on ToolQA and MiniWob++, with up to 13x reduction in execution failures., Comment: ICML 2024 Camera ready version

Interventions on model-internal states are fundamental operations in many areas of AI, including model editing, steering, robustness, and interpretability. To facilitate such research, we introduce $\textbf{pyvene}$, an open-source Python library that supports customizable interventions on a range of different PyTorch modules. $\textbf{pyvene}$ supports complex intervention schemes with an intuitive configuration format, and its interventions can be static or include trainable parameters. We show how $\textbf{pyvene}$ provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others. We illustrate the power of the library via interpretability analyses using causal abstraction and knowledge localization. We publish our library through Python Package Index (PyPI) and provide code, documentation, and tutorials at https://github.com/stanfordnlp/pyvene., Comment: 8 pages, 3 figures