Search

The evaluation of generative or discriminative large language model (LLM)-based systems is often a complex multi-dimensional problem. Typically, a set of system configuration alternatives are evaluated on one or more benchmark datasets, each with one or more evaluation metrics, which may differ between datasets. We often want to evaluate -- with a statistical measure of significance -- whether systems perform differently either on a given dataset according to a single metric, on aggregate across metrics on a dataset, or across datasets. Such evaluations can be done to support decision-making, such as deciding whether a particular system component change (e.g., choice of LLM or hyperparameter values) significantly improves performance over the current system configuration, or, more generally, whether a fixed set of system configurations (e.g., a leaderboard list) have significantly different performances according to metrics of interest. We present a framework implementation that automatically performs the correct statistical tests, properly aggregates the statistical results across metrics and datasets (a nontrivial task), and can visualize the results. The framework is demonstrated on the multi-lingual code generation benchmark CrossCodeEval, for several state-of-the-art LLMs.

Anisotropic materials with low symmetries hold significant promise for next-generation electronic and quantum devices. 2M-WS2, a candidate for topological superconductivity, has garnered considerable interest. However, a comprehensive understanding of how its anisotropic features contribute to unconventional superconductivity, along with a simple, reliable method to identify its crystal orientation, remains elusive. Here, we combine theoretical and experimental approaches to investigate angle- and polarization-dependent anisotropic Raman modes of 2M-WS2. Through first-principles calculations, we predict and analyze phonon dispersion and lattice vibrations of all Raman modes in 2M-WS2. We establish a direct correlation between their anisotropic Raman spectra and high-resolution transmission electron microscopy images. Finally, we demonstrate that anisotropic Raman spectroscopy can accurately determine the crystal orientation and twist angle between two stacked 2M-WS2 layers. Our findings provide insights into the electron-phonon coupling and anisotropic properties of 2M-WS2, paving the way for the use of anisotropic materials in advanced electronic and quantum devices., Comment: 19 pages, 5 figures

In 1972, Branko Gr\"unbaum conjectured that any arrangement of $n>2$ pairwise crossing pseudocircles in the plane can have at most $2n-2$ digons (regions enclosed by exactly two pseudoarcs), with the bound being tight. While this conjecture has been confirmed for cylindrical arrangements of pseudocircles and more recently for geometric circles, we extend these results to any simple arrangement of pairwise intersecting pseudocircles. Using techniques from the above-mentioned special cases, we provide a complete proof of Gr\"unbaum's conjecture that has stood open for over five decades.

Leveraging reciprocal-space proximity effect between superconducting bulk and topological surface states (TSSs) offers a promising way to topological superconductivity. However, elucidating the mutual influence of bulk and TSSs on topological superconductivity remains a challenge. Here, we report pioneering transport evidence of a thickness-dependent transition from conventional to unconventional superconductivity in 2M-phase WS2 (2M-WS2). As the sample thickness reduces, we see clear changes in key superconducting metrics, including critical temperature, critical current, and carrier density. Notably, while thick 2M-WS2 samples show conventional superconductivity, with an in-plane (IP) upper critical field constrained by the Pauli limit, samples under 20 nm exhibit a pronounced IP critical field enhancement, inversely correlated with 2D carrier density. This marks a distinct crossover to unconventional superconductivity with strong spin-orbit-parity coupling. Our findings underscore the crucial role of sample thickness in accessing topological states in 2D topological superconductors, offering pivotal insights into future studies of topological superconductivity., Comment: 15 pages, 4 figures

Digital media platforms (e.g., social media, science blogs) offer opportunities to communicate scientific content to general audiences at scale. However, these audiences vary in their scientific expertise, literacy levels, and personal backgrounds, making effective science communication challenging. To address this challenge, we designed TranSlider, an AI-powered tool that generates personalized translations of scientific text based on individual user profiles (e.g., hobbies, location, and education). Our tool features an interactive slider that allows users to steer the degree of personalization from 0 (weakly relatable) to 100 (strongly relatable), leveraging LLMs to generate the translations with given degrees. Through an exploratory study with 15 participants, we investigated both the utility of these AI-personalized translations and how interactive reading features influenced users' understanding and reading experiences. We found that participants who preferred higher degrees of personalization appreciated the relatable and contextual translations, while those who preferred lower degrees valued concise translations with subtle contextualization. Furthermore, participants reported the compounding effect of multiple translations on their understanding of scientific content. Given these findings, we discuss several implications of AI-personalized translation tools in facilitating communication in collaborative contexts., Comment: 23 pages, 5 figures, 1 table

Two-dimensional (2D) van der Waals (vdW) magnets with layer-dependent magnetic states and/or diverse magnetic interactions and anisotropies have attracted extensive research interest. Despite the advances, a notable challenge persists in effectively manipulating the tunneling anisotropic magnetoresistance (TAMR) of 2D vdW magnet-based magnetic tunnel junctions (MTJs). Here, we report the novel and anomalous tunneling magnetoresistance (TMR) oscillations and pioneering demonstration of bias and gate voltage controllable TAMR in 2D vdw MTJs, utilizing few-layer CrPS4. This material, inherently an antiferromagnet, transitions to a canted magnetic order upon application of external magnetic fields. Through TMR measurements, we unveil the novel, layer-dependent oscillations in the tunneling resistance for few-layer CrPS4 devices under both out-of-plane and in-plane magnetic fields, with a pronounced controllability via gate voltage. Intriguingly, we demonstrate that both the polarity and magnitude of TAMR in CrPS4 can be effectively tuned through either a bias or gate voltage. We further elucidate the mechanism behind this electrically tunable TAMR through first-principles calculations. The implications of our findings are far-reaching, providing new insights into 2D magnetism and opening avenues for the development of innovative spintronic devices based on 2D vdW magnets., Comment: 22 pages, 4 figures

Effective control of magnetic phases in two-dimensional magnets would constitute crucial progress in spintronics, holding great potential for future computing technologies. Here, we report a new approach of leveraging tunneling current as a tool for controlling spin states in CrI3. We reveal that a tunneling current can deterministically switch between spin-parallel and spin-antiparallel states in few-layer CrI3, depending on the polarity and amplitude of the current. We propose a mechanism involving nonequilibrium spin accumulation in the graphene electrodes in contact with the CrI3 layers. We further demonstrate tunneling current-tunable stochastic switching between multiple spin states of the CrI3 tunnel devices, which goes beyond conventional bi-stable stochastic magnetic tunnel junctions and has not been documented in two-dimensional magnets. Our findings not only address the existing knowledge gap concerning the influence of tunneling currents in controlling the magnetism in two-dimensional magnets, but also unlock possibilities for energy-efficient probabilistic and neuromorphic computing., Comment: 23 pages, 4 figures

It has been reported that LLMs can recognize their own writing. As this has potential implications for AI safety, yet is relatively understudied, we investigate the phenomenon, seeking to establish whether it robustly occurs at the behavioral level, how the observed behavior is achieved, and whether it can be controlled. First, we find that the Llama3-8b-Instruct chat model - but not the base Llama3-8b model - can reliably distinguish its own outputs from those of humans, and present evidence that the chat model is likely using its experience with its own outputs, acquired during post-training, to succeed at the writing recognition task. Second, we identify a vector in the residual stream of the model that is differentially activated when the model makes a correct self-written-text recognition judgment, show that the vector activates in response to information relevant to self-authorship, present evidence that the vector is related to the concept of "self" in the model, and demonstrate that the vector is causally related to the model's ability to perceive and assert self-authorship. Finally, we show that the vector can be used to control both the model's behavior and its perception, steering the model to claim or disclaim authorship by applying the vector to the model's output as it generates it, and steering the model to believe or disbelieve it wrote arbitrary texts by applying the vector to them as the model reads them., Comment: 10 pages, 13 figs, 2 tables, accepted as conference paper to ICLR 2025

Activation engineering is becoming increasingly popular as a means of online control of large language models (LLMs). In this work, we extend the idea of inference-time steering with vectors that represent a behavioral direction of interest to tuning those vectors directly into the model, obviating the need for online control. First, we identify activation vectors related to honesty in an open-source LLM (Llama-2-13b-chat). Next, we demonstrate that model output can be made more or less honest by adding positive or negative multiples of these vectors to residual stream activations during generation. Then, we show that a similar effect can be achieved by fine-tuning the vectors directly into the model, by use of a dual loss function based on the cosine similarity of residual stream activations to the vectors combined with a standard token-based loss ("representation tuning"). Finally, we compare the generations in response to honesty-probing prompts from the resulting models to those from models fine-tuned with a token-based loss alone, and to those from the untuned model subjected to online steering. Overall, fine-tuning the vectors into the models using the cosine similarity plus token loss showed a stronger effect than online steering, and generalized better than using the standard loss, suggesting the potential utility of this approach as a safety measure. Code and data are available at https://github.com/cma1114/representation_tuning. Tuned models are available at https://huggingface.co/collections/cackerman/representation-tuning-66da1e5ab41cd1b824687d9f., Comment: 10 pages, 7 figures, 6 tables

Neuroimaging studies, such as the Human Connectome Project (HCP), often collect multi-faceted and multi-block data to study the complex human brain. However, these data are often analyzed in a pairwise fashion, which can hinder our understanding of how different brain-related measures interact with each other. In this study, we comprehensively analyze the multi-block HCP data using the Data Integration via Analysis of Subspaces (DIVAS) method. We integrate structural and functional brain connectivity, substance use, cognition, and genetics in an exhaustive five-block analysis. This gives rise to the important finding that genetics is the single data modality most predictive of brain connectivity, outside of brain connectivity itself. Nearly 14\% of the variation in functional connectivity (FC) and roughly 12\% of the variation in structural connectivity (SC) is attributed to shared spaces with genetics. Moreover, investigations of shared space loadings provide interpretable associations between particular brain regions and drivers of variability, such as alcohol consumption in the substance-use data block. Novel Jackstraw hypothesis tests are developed for the DIVAS framework to establish statistically significant loadings. For example, in the (FC, SC, and Substance Use) shared space, these novel hypothesis tests highlight largely negative functional and structural connections suggesting the brain's role in physiological responses to increased substance use. Furthermore, our findings have been validated using a subset of genetically relevant siblings or twins not studied in the main analysis., Comment: 38 pages, 6 figures

We present spectroscopic data for four metastable state clear-out transitions between 399 nm and 412 nm for all even long-lived isotopes of Yb$^+$ as well as their hyperfine structure in $^{171}$Yb$^+$. The strong $^2 \rm{D}_{3/2} \rightarrow {}^1[1/2]_{1/2}$ transition at 412 nm represents an attractive alternative for the standard 935 nm repumper used in cooling and trapping experiments, while the transition to the $^3[3/2]_{3/2}$ state at 411 nm clears out the $^2 $F$_{7/2}$ state, for which typically 638 nm or 760 nm are used. These two alternative transitions simplify the experimental setup by removing the need for infrared optics to cool and manipulate Yb$^+$ and may be of particular interest when considering integrated photonics solutions. We also present data for the transitions $^2 $D$_{3/2} \rightarrow {}^3[1/2]_{3/2}$ at 399 nm, and $^2 $D$_{3/2} \rightarrow {}^1[5/2]_{5/2}$ at 410 nm including decay branching ratios of the excited states., Comment: 7 pages, 7 figures

DaCapo is a specialized deep learning library tailored to expedite the training and application of existing machine learning approaches on large, near-isotropic image data. In this correspondence, we introduce DaCapo's unique features optimized for this specific domain, highlighting its modular structure, efficient experiment management tools, and scalable deployment capabilities. We discuss its potential to improve access to large-scale, isotropic image segmentation and invite the community to explore and contribute to this open-source initiative.

We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets., Comment: Published in the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) findings

In this work, we analyze two large-scale surveys to examine how individuals think about sharing smartphone access with romantic partners as a function of trust in relationships. We find that the majority of couples have access to each others' devices, but may have explicit or implicit boundaries on how this access is to be used. Investigating these boundaries and related social norms, we find that there is little consensus about the level of smartphone access (i.e., transparency), or lack thereof (i.e., privacy) that is desirable in romantic contexts. However, there is broad agreement that the level of access should be mutual and consensual. Most individuals understand trust to be the basis of their decisions about transparency and privacy. Furthermore, we find individuals have crossed these boundaries, violating their partners' privacy and betraying their trust. We examine how, when, why, and by whom these betrayals occur. We consider the ramifications of these boundary violations in the case of intimate partner violence. Finally, we provide recommendations for design changes to enable technological enforcement of boundaries currently enforced by trust, bringing access control in line with users' sharing preferences.

This paper presents a routine to align an optical tweezer on a single trapped ion and use the ion as a probe to characterize the tweezer. We find a smallest tweezer waist of $2.3(2)\,\mu$m, which is in agreement with the theoretical minimal attainable waist of $2.5(2)\,\mu$m in our setup. We characterize the spatial dependence of the tweezer Rabi frequency which is suppressed by a factor of 19(3) in the immediate surrounding of the ion. We investigate the effects of optical forces and coherent population trapping on the ion. Finally, we show that the challenges posed by these forces can be overcome, and that the number of tweezers can be easily scaled up to reach several ions by using a spatial light modulator.

A long-standing open conjecture of Branko Gr\"unbaum from 1972 states that any simple arrangement of $n$ pairwise intersecting pseudocircles in the plane can have at most $2n-2$ digons. Agarwal et al. proved this conjecture for arrangements of pairwise intersecting pseudocircles in which there is a common point surrounded by all pseudocircles. Recently, Felsner, Roch and Scheucher showed that Gr\"unbaum's conjecture is true for arrangements of pairwise intersecting pseudocircles in which there are three pseudocircles every pair of which create a digon. In this paper we prove this over 50-year-old conjecture of Gr\"unbaum for any simple arrangement of pairwise intersecting circles in the plane.

We introduce a novel LLM based solution design approach that utilizes combinatorial optimization and sampling. Specifically, a set of factors that influence the quality of the solution are identified. They typically include factors that represent prompt types, LLM inputs alternatives, and parameters governing the generation and design alternatives. Identifying the factors that govern the LLM solution quality enables the infusion of subject matter expert knowledge. Next, a set of interactions between the factors are defined and combinatorial optimization is used to create a small subset $P$ that ensures all desired interactions occur in $P$. Each element $p \in P$ is then developed into an appropriate benchmark. Applying the alternative solutions on each combination, $p \in P$ and evaluating the results facilitate the design of a high quality LLM solution pipeline. The approach is especially applicable when the design and evaluation of each benchmark in $P$ is time-consuming and involves manual steps and human evaluation. Given its efficiency the approach can also be used as a baseline to compare and validate an autoML approach that searches over the factors governing the solution.

This research paper discusses the advances made in the past decade in biomedicine and Large Language Models. To understand how the advances have been made hand-in-hand with one another, the paper also discusses the integration of Natural Language Processing techniques and tools into biomedicine. Finally, the goal of this paper is to expand on a survey conducted last year (2023) by introducing a new list of questions and prompts for the top two language models. Through this survey, this paper seeks to quantify the improvement made in the reasoning abilities in LLMs and to what extent those improvements are felt by the average user. Additionally, this paper seeks to extend research on retrieval of biological literature by prompting the LLM to answer open-ended questions in great depth., Comment: 19 pages, 5 figures

Language Models have demonstrated remarkable capabilities on some tasks while failing dramatically on others. The situation has generated considerable interest in understanding and comparing the capabilities of various Language Models (LMs) but those efforts have been largely ad hoc with results that are often little more than anecdotal. This is in stark contrast with testing and evaluation processes used in healthcare, radar signal processing, and other defense areas. In this paper, we describe Test and Evaluation of Language Models (TEL'M) as a principled approach for assessing the value of current and future LMs focused on high-value commercial, government and national security applications. We believe that this methodology could be applied to other Artificial Intelligence (AI) technologies as part of the larger goal of "industrializing" AI.

Social media systems are as varied as they are pervasive. They have been almost universally adopted for a broad range of purposes including work, entertainment, activism, and decision making. As a result, they have also diversified, with many distinct designs differing in content type, organization, delivery mechanism, access control, and many other dimensions. In this work, we aim to characterize and then distill a concise design space of social media systems that can help us understand similarities and differences, recognize potential consequences of design choices, and identify spaces for innovation. Our model, which we call Form-From, characterizes social media based on (1) the form of the content, either threaded or flat, and (2) from where or from whom one might receive content, ranging from spaces to networks to the commons. We derive Form-From inductively from a larger set of 62 dimensions organized into 10 categories. To demonstrate the utility of our model, we trace the history of social media systems as they traverse the Form-From space over time, and we identify common design patterns within cells of the model.

We study semantic models of probabilistic programming languages over graphs, and establish a connection to graphons from graph theory and combinatorics. We show that every well-behaved equational theory for our graph probabilistic programming language corresponds to a graphon, and conversely, every graphon arises in this way. We provide three constructions for showing that every graphon arises from an equational theory. The first is an abstract construction, using Markov categories and monoidal indeterminates. The second and third are more concrete. The second is in terms of traditional measure theoretic probability, which covers 'black-and-white' graphons. The third is in terms of probability monads on the nominal sets of Gabbay and Pitts. Specifically, we use a variation of nominal sets induced by the theory of graphs, which covers Erd\H{o}s-R\'enyi graphons. In this way, we build new models of graph probabilistic programming from graphons., Comment: Accepted for POPL 2024