Search

This paper investigates the problem-solving capabilities of Large Language Models (LLMs) by evaluating their performance on stumpers, unique single-step intuition problems that pose challenges for human solvers but are easily verifiable. We compare the performance of four state-of-the-art LLMs (Davinci-2, Davinci-3, GPT-3.5-Turbo, GPT-4) to human participants. Our findings reveal that the new-generation LLMs excel in solving stumpers and surpass human performance. However, humans exhibit superior skills in verifying solutions to the same problems. This research enhances our understanding of LLMs' cognitive abilities and provides insights for enhancing their problem-solving potential across various domains.

In this study, we asked whether Montessori schools, which tend to have high student engagement, are associated with lower average rates of chronic absenteeism and/or smaller racial disparities therein relative to non-Montessori schools. We use multilevel modeling to answer this question, following an approach proposed in Author et al., (in press) for analyzing racial disparities in count outcomes. We identified a sample of Title 1 Montessori and non-Montessori schools using propensity score matching and data from the Civil Rights Data Collection. We did not observe significant differences in average overall rates of chronic absenteeism across school types, nor in average racial disparities in the rates of chronic absenteeism between Black and White or Hispanic and White students, though Montessori schools had slightly lower average rates for White students. We discuss shortcomings in the way chronic absenteeism data is collected, and how these shortcomings do not allow researchers to answer questions about why students are chronically absent, thereby limiting intervention work. [This paper was published in "Frontiers in Education.]

Common methods of measuring discipline disproportionality can produce contradictory results and obscure base-rate information. In this paper, we show how using multilevel modeling to analyze discipline disparities resolves ambiguities inherent in traditional measures of disparities: relative rate ratios and risk differences. One previous study suggests there is less racial discipline disproportionality in Montessori schools, so we used our new approach, along with relative rate ratios and risk differences, to compare discipline disproportionality in a sample of Title 1 Montessori and non-Montessori schools identified using propensity score matching. Using the multilevel model clarified results from other measures: discipline disproportionality was similar across school settings, even though overall rates were significantly lower in the Montessori schools. [This paper was published in "Journal of Research on Educational Effectiveness.]

We investigate the real-time dynamics of the sub-Ohmic spin-boson model across a broad range of coupling strengths, using the numerically exact inchworm quantum Monte Carlo algorithm. From short- and intermediate-time dynamics starting from an initially decoupled state, we extract signatures of the zero-temperature quantum phase transition between localized and delocalized states. We show that the dynamical phase diagram thus obtained differs from the equilibrium phase diagram in both the values of critical couplings and the associated critical exponents. We also identify and quantitatively analyze two competing mechanisms for the crossover between coherent oscillations and incoherent decay. Deep in the sub-Ohmic regime, the crossover is driven by the damping of the oscillation amplitude, while closer to the Ohmic regime the oscillation frequency itself drops sharply to zero at large coupling.

Sponges, the basalmost members of the animal kingdom, exhibit a range of complex architectures in which microfluidic channels connect multitudes of spherical chambers lined with choanocytes, flagellated filter-feeding cells. Choanocyte chambers can possess scores or even hundreds of such cells, which drive complex flows entering through porous walls and exiting into the sponge channels. One of the mysteries of the choanocyte chamber is its spherical shape, as it seems inappropriate for inducing directional transport since many choanocyte flagella beat in opposition to such a flow. Here we combine direct imaging of choanocyte chambers in living sponges with computational studies of many-flagella models to understand the connection between chamber architecture and directional flow. We find that those flagella that beat against the flow play a key role in raising the pressure inside the choanocyte chamber, with the result that the mechanical pumping efficiency, calculated from the pressure rise and flow rate, reaches a maximum at a small outlet opening angle. Comparison between experimental observations and the results of numerical simulations reveal that the chamber diameter, flagellar wave number and the outlet opening angle of the freshwater sponge $E. muelleri$, as well as several other species, are related in a manner that maximizes the mechanical pumping efficiency. These results indicate the subtle balances at play during morphogenesis of choanocyte chambers, and give insights into the physiology and body design of sponges., Comment: 13 pages, 13 figures

It has recently been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. In this work, we argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking. We provide a broad overview of possible attack surfaces and attack goals. Based on a series of concrete examples, we discuss, categorize and systematize attacks that coerce varied unintended behaviors, such as misdirection, model control, denial-of-service, or data extraction. We analyze these attacks in controlled experiments, and find that many of them stem from the practice of pre-training LLMs with coding capabilities, as well as the continued existence of strange "glitch" tokens in common LLM vocabularies that should be removed for security reasons., Comment: 32 pages. Implementation available at https://github.com/JonasGeiping/carving

The humble Petri dish is perhaps the simplest setting in which to examine the locomotion of swimming organisms, particularly those whose body size is tens of microns to millimetres. The fluid layer in such a container has a bottom no-slip surface and a stress-free upper boundary. It is of fundamental interest to understand the flow fields produced by the elementary and composite singularities of Stokes flow in this geometry. Building on the few particular cases that have previously been considered in the literature, we study here the image systems for the primary singularities of Stokes flow subject to such boundary conditions - the stokeslet, rotlet, source, rotlet dipole, source dipole and stresslet - paying particular attention to the far-field behavior. In several key situations, the depth-averaged fluid flow is accurately captured by the solution of an associated Brinkman equation whose screening length is proportional to the depth of the fluid layer. The case of hydrodynamic bound states formed by spinning microswimmers near a no-slip surface, discovered first using the alga $Volvox$, is reconsidered in the geometry of a Petri dish, where the power-law attractive interaction between microswimmers acquires unusual exponentially screened oscillations., Comment: 27 pages, 7 figures

In this work, we study the issue of reward hacking on the response length, a challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on LLMs. A well-formatted, verbose but less helpful response from the LLMs can often deceive LLMs or even human evaluators to achieve high scores. The same issue also holds for some reward models in RL. To address the challenges in both training and evaluation, we establish a more reliable evaluation protocol for comparing different training configurations, which inspects the trade-off between LLM evaluation score and response length obtained by varying training hyperparameters. Based on this evaluation, we conduct large-scale studies, where the results shed insights into the efficacy of hyperparameters and tricks used in RL on mitigating length bias. We further propose to improve the reward model by jointly training two linear heads on shared feature representations to predict the rewards, one trained to correlate with length, and the other trained to decorrelate with length and therefore focus more on the actual content. We then discard the length head in RL to prevent reward hacking on length. Experiments demonstrate that our approach almost eliminates the reward correlation with length, and improves the obtained policy by a significant margin.

The emergence of Large Language Models (LLMs), has opened exciting possibilities for constructing computational simulations designed to replicate human behavior accurately. Current research suggests that LLM-based agents become increasingly human-like in their performance, sparking interest in using these AI agents as substitutes for human participants in behavioral studies. However, LLMs are complex statistical learners without straightforward deductive rules, making them prone to unexpected behaviors. Hence, it is crucial to study and pinpoint the key behavioral distinctions between humans and LLM-based agents. In this study, we highlight the limitations of LLMs in simulating human interactions, particularly focusing on LLMs' ability to simulate political debates on topics that are important aspects of people's day-to-day lives and decision-making processes. Our findings indicate a tendency for LLM agents to conform to the model's inherent social biases despite being directed to debate from certain political perspectives. This tendency results in behavioral patterns that seem to deviate from well-established social dynamics among humans. We reinforce these observations using an automatic self-fine-tuning method, which enables us to manipulate the biases within the LLM and demonstrate that agents subsequently align with the altered biases. These results underscore the need for further research to develop methods that help agents overcome these biases, a critical step toward creating more realistic simulations.

Vision-Language Models (VLMs) excel in generating textual responses from visual inputs, but their versatility raises security concerns. This study takes the first step in exposing VLMs' susceptibility to data poisoning attacks that can manipulate responses to innocuous, everyday prompts. We introduce Shadowcast, a stealthy data poisoning attack where poison samples are visually indistinguishable from benign images with matching texts. Shadowcast demonstrates effectiveness in two attack types. The first is a traditional Label Attack, tricking VLMs into misidentifying class labels, such as confusing Donald Trump for Joe Biden. The second is a novel Persuasion Attack, leveraging VLMs' text generation capabilities to craft persuasive and seemingly rational narratives for misinformation, such as portraying junk food as healthy. We show that Shadowcast effectively achieves the attacker's intentions using as few as 50 poison samples. Crucially, the poisoned samples demonstrate transferability across different VLM architectures, posing a significant concern in black-box settings. Moreover, Shadowcast remains potent under realistic conditions involving various text prompts, training data augmentation, and image compression techniques. This work reveals how poisoned VLMs can disseminate convincing yet deceptive misinformation to everyday, benign users, emphasizing the importance of data integrity for responsible VLM deployments. Our code is available at: https://github.com/umd-huang-lab/VLM-Poisoning., Comment: Accepted by Thirty-Eighth Annual Conference on Neural Information Processing Systems (Neurips 2024)

Berry phases have long been known to significantly alter the properties of periodic systems, giving rise to anomalous terms in the semiclassical equations of motion describing wave-packet dynamics. In non-Hermitian systems, generalizations of the Berry connection have been proposed and shown to have novel effects on dynamics and transport. In this work, we expand upon these results by deriving the full set of semiclassical equations of motion for wave-packet dynamics in a non-Hermitian system subject to complex external electric fields, which are realizable as gain gradients. We show that the non-Hermiticities of both the band Hamiltonian and the external potential give rise to anomalous weight rate and velocity terms which depend on the geometric properties of the eigenfunctions, including the quantum metric tensor. These analytical results are compared with numerical lattice simulations which reveal these anomalous terms even in one-dimension. Our work expands the range of phenomena expected to be detectable in experimental setups, which should be realizable in currently available metamaterials and classical wave systems, including mechanical, acoustic, and optical., Comment: 19 pages, 6 figures

Real world data is an increasingly utilized resource for post-market monitoring of vaccines and provides insight into real world effectiveness. However, outside of the setting of a clinical trial, heterogeneous mechanisms may drive observed breakthrough infection rates among vaccinated individuals; for instance, waning vaccine-induced immunity as time passes and the emergence of a new strain against which the vaccine has reduced protection. Analyses of infection incidence rates are typically predicated on a presumed mechanism in their choice of an "analytic time zero" after which infection rates are modeled. In this work, we propose an explicit test for driving mechanism situated in a standard Cox proportional hazards framework. We explore the test's performance in simulation studies and in an illustrative application to real world data. We additionally introduce subgroup differences in infection incidence and evaluate the impact of time zero misspecification on bias and coverage of model estimates. In this study we observe strong power and controlled type I error of the test to detect the correct infection-driving mechanism under various settings. Similar to previous studies, we find mitigated bias and greater coverage of estimates when the analytic time zero is correctly specified or accounted for.

We consider the fluctuations in the number of particles in a box of size L^d in Z^d, d>=1, in the (infinite volume) translation invariant stationary states of the facilitated exclusion process, also called the conserved lattice gas model. When started in a Bernoulli (product) measure at density rho, these systems approach, as t goes to infinity, a "frozen" state for rho<=rho_c, with rho_c=1/2 for d=1 and rho_c<1/2 for d>=2. At rho=rho_c the limiting state is hyperuniform, that is, the variance of the number of particles in the box grows slower than L^d. We give a general description of how the variances at different scales of L behave as rho increases to rho_c. On the largest scale, L>>L_2, the fluctuations are normal (in fact the same as in the original product measure), while in a region L_1<=2.), Comment: 22 pages, 5 figures. To be submitted to the Focus Issue on Hyperuniformity of the Journal of Physics: Condensed Matter

Classical gamma-ray bursts (GRBs) have two distinct emission episodes: prompt emission from ultra-relativistic ejecta and afterglow from shocked circumstellar material. While both components are extremely luminous in known GRBs, a variety of scenarios predict the existence of luminous afterglow emission with little or no associated high-energy prompt emission. We present AT 2019pim, the first secure example of this phenomenon to be identified. Serendipitously discovered during follow-up observations of a gravitational-wave trigger and located in a contemporaneous TESS sector, it is hallmarked by a fast-rising (t ~ 2 hr), luminous (M_UV,peak ~ -24.4 mag) optical transient with accompanying luminous X-ray and radio emission. No gamma-ray emission consistent with the time and location of the transient was detected by Fermi-GBM or by Konus, placing strong limits on an accompanying GRB. We investigate several independent observational aspects of the afterglow in the context of constraints on relativistic motion and find all of them are consistent with an initial Lorentz factor of Gamma_0 ~ 30-50, significantly lower than in any well-observed GRB and consistent with the theoretically-predicted "dirty fireball" scenario in which the high-energy prompt emission is stifled by pair production. However, we cannot rule out a structured jet model in which only the line-of-sight material was ejected at low-Gamma, off-axis from a classical high-Gamma jet core. This event represents a milestone in orphan afterglow searches, demonstrating that luminous afterglows with weak or no detectable gamma-ray radiation exist in nature and can be discovered by high-cadence optical surveys., Comment: Submitted to MNRAS

Detecting text generated by modern large language models is thought to be hard, as both LLMs and humans can exhibit a wide range of complex behaviors. However, we find that a score based on contrasting two closely related language models is highly accurate at separating human-generated and machine-generated text. Based on this mechanism, we propose a novel LLM detector that only requires simple calculations using a pair of pre-trained LLMs. The method, called Binoculars, achieves state-of-the-art accuracy without any training data. It is capable of spotting machine text from a range of modern LLMs without any model-specific modifications. We comprehensively evaluate Binoculars on a number of text sources and in varied situations. Over a wide range of document types, Binoculars detects over 90% of generated samples from ChatGPT (and other LLMs) at a false positive rate of 0.01%, despite not being trained on any ChatGPT data., Comment: 20 pages, code available at https://github.com/ahans30/Binoculars

We present Scalable Interpolant Transformers (SiT), a family of generative models built on the backbone of Diffusion Transformers (DiT). The interpolant framework, which allows for connecting two distributions in a more flexible way than standard diffusion models, makes possible a modular study of various design choices impacting generative models built on dynamical transport: learning in discrete or continuous time, the objective function, the interpolant that connects the distributions, and deterministic or stochastic sampling. By carefully introducing the above ingredients, SiT surpasses DiT uniformly across model sizes on the conditional ImageNet 256x256 and 512x512 benchmark using the exact same model structure, number of parameters, and GFLOPs. By exploring various diffusion coefficients, which can be tuned separately from learning, SiT achieves an FID-50K score of 2.06 and 2.62, respectively., Comment: ECCV 2024; Code available: https://github.com/willisma/SiT

In the burgeoning age of generative AI, watermarks act as identifiers of provenance and artificial content. We present WAVES (Watermark Analysis Via Enhanced Stress-testing), a benchmark for assessing image watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised of a diverse range of stress tests. The attacks in WAVES range from traditional image distortions to advanced, novel variations of diffusive, and adversarial attacks. Our evaluation examines two pivotal dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. Our novel, comprehensive evaluation reveals previously undetected vulnerabilities of several modern watermarking algorithms. We envision WAVES as a toolkit for the future development of robust watermarks. The project is available at https://wavesbench.github.io/, Comment: Accepted by ICML 2024

We study gradient regularity for mixed local-nonlocal problems modelled upon \[ -\Delta_p u +(-\Delta_p)^su=\mu\qquad\text{for} \quad 2-\tfrac{1}{n}

In this White Paper, we present recommendations for the scientific community and funding agencies to foster the infrastructure for a collaborative multi-messenger and time-domain astronomy (MMA/TDA) ecosystem. MMA/TDA is poised for breakthrough discoveries in the coming decade. In much the same way that expanding beyond the optical bandpass revealed entirely new and unexpected discoveries, cosmic messengers beyond light (i.e., gravitational waves, neutrinos, and cosmic rays) open entirely new windows to answer some of the most fundamental questions in (astro)physics: heavy element synthesis, equation of state of dense matter, particle acceleration, etc. This field was prioritized as a frontier scientific pursuit in the 2020 Decadal Survey on Astronomy and Astrophysics via its "New Windows on the Dynamic Universe" theme. MMA/TDA science presents technical challenges distinct from those experienced in other disciplines. Successful observations require coordination across myriad boundaries -- different cosmic messengers, ground vs. space, international borders, etc. -- all for sources that may not be well localized, and whose brightness may be changing rapidly with time. Add that all of this work is undertaken by real human beings, with distinct backgrounds, experiences, cultures, and expectations, that often conflict. To address these challenges and help MMA/TDA realize its full scientific potential in the coming decade (and beyond), the second in a series of community workshops sponsored by the U.S. National Science Foundation (NSF) and NASA titled "Windows on the Universe: Establishing the Infrastructure for a Collaborative Multi-Messenger Ecosystem" was held on October 16-18, 2023 in Tucson, AZ. Here we present the primary recommendations from this workshop focused on three key topics -- hardware, software, and people and policy. [abridged], Comment: Workshop white paper

Lattice gauge theories (LGT) play a central role in modern physics, providing insights into high-energy physics, condensed matter physics, and quantum computation. Due to the nontrivial structure of the Hilbert space of LGT systems, entanglement in such systems is tricky to define. However, when one limits themselves to superselection-resolved entanglement, that is, entanglement corresponding to specific gauge symmetry sectors (commonly denoted as superselection sectors), this problem disappears, and the entanglement becomes well-defined. The study of superselection-resolved entanglement is interesting in LGT for an additional reason: when the gauge symmetry is strictly obeyed, superselection-resolved entanglement becomes the only distillable contribution to the entanglement. In our work, we study the behavior of superselection-resolved entanglement in LGT systems. We employ a tensor network construction for gauge-invariant systems as defined by Zohar and Burrello (2016) and find that, in a vast range of cases, the leading term in superselection-resolved entanglement depends on the number of corners in the partition, that is, corner-law entanglement. To our knowledge, this is the first case of such a corner-law being observed in any lattice system., Comment: 12 pages, 6 figures, comments are welcome

Recently, Pyramid Adversarial training (Herrmann et al., 2022) has been shown to be very effective for improving clean accuracy and distribution-shift robustness of vision transformers. However, due to the iterative nature of adversarial training, the technique is up to 7 times more expensive than standard training. To make the method more efficient, we propose Universal Pyramid Adversarial training, where we learn a single pyramid adversarial pattern shared across the whole dataset instead of the sample-wise patterns. With our proposed technique, we decrease the computational cost of Pyramid Adversarial training by up to 70% while retaining the majority of its benefit on clean performance and distribution-shift robustness. In addition, to the best of our knowledge, we are also the first to find that universal adversarial training can be leveraged to improve clean model performance.

We prove that for any measurable mapping $T$ into the space of matrices with positive determinant, there is a diffeomorphism whose derivative equals $T$ outside a set of measure less than $\varepsilon$. We use this fact to prove that for any measurable mapping $T$ into the space of matrices with non-zero determinant (with no sign restriction), there is an almost everywhere approximately differentiable homeomorphism whose derivative equals $T$ almost everywhere.

The emerging field of Diverse Intelligence seeks to identify, formalize, and understand commonalities in behavioral competencies across a wide range of implementations. Especially interesting are simple systems that provide unexpected examples of memory, decision-making, or problem-solving in substrates that at first glance do not appear to be complex enough to implement such capabilities. We seek to develop tools to help understand the minimal requirements for such capabilities, and to learn to recognize and predict basal forms of intelligence in unconventional substrates. Here, we apply novel analyses to the behavior of classical sorting algorithms, short pieces of code which have been studied for many decades. To study these sorting algorithms as a model of biological morphogenesis and its competencies, we break two formerly-ubiquitous assumptions: top-down control (instead, showing how each element within a array of numbers can exert minimal agency and implement sorting policies from the bottom up), and fully reliable hardware (instead, allowing some of the elements to be "damaged" and fail to execute the algorithm). We quantitatively characterize sorting activity as the traversal of a problem space, showing that arrays of autonomous elements sort themselves more reliably and robustly than traditional implementations in the presence of errors. Moreover, we find the ability to temporarily reduce progress in order to navigate around a defect, and unexpected clustering behavior among the elements in chimeric arrays whose elements follow one of two different algorithms. The discovery of emergent problem-solving capacities in simple, familiar algorithms contributes a new perspective to the field of Diverse Intelligence, showing how basal forms of intelligence can emerge in simple systems without being explicitly encoded in their underlying mechanics.

In Nature there are significant relationships known between microorganisms from two kingdoms of life, as in the supply of vitamin B$_{12}$ by bacteria to algae. Such interactions motivate general investigations into the spatio-temporal dynamics of metabolite exchanges. Here we study by experiment and theory a model system: a coculture of the bacterium $B. subtilis$, an obligate aerobe that is chemotactic to oxygen, and a nonmotile mutant of the alga $C. reinhardtii$, which photosynthetically produces oxygen when illuminated. Strikingly, when a shaft of light illuminates a thin, initially uniform suspension of the two, the chemotactic influx of bacteria to the photosynthetically active region leads to expulsion of the algae from that area. This effect arises from algal transport due to spatially-varying collective behavior of bacteria, and is mathematically related to the ``turbulent diamagnetism" associated with magnetic flux expulsion in stars., Comment: 9 pages, 5 figures

This paper explores the benefits of incorporating natural ventilation (NV) simulation into a generative process of designing residential buildings to improve energy efficiency and indoor thermal comfort. Our proposed workflow uses the Wave Function Collapse algorithm to generate a diverse set of plausible floor plans. It also includes post-COVID occupant presence models while incorporating adaptive comfort models. We conduct four sets of experiments using the workflow, and the simulated results suggest that multi-mode cooling strategies combining conventional air conditioning with NV can often significantly reduce energy use while introducing only slight reductions in thermal comfort., Comment: 12 pages, 14 figures

The goal of this series is to chronicle opinions and issues in the field of machine learning as they stand today and as they change over time. The plan is to host this survey periodically until the AI singularity paperclip-frenzy-driven doomsday, keeping an updated list of topical questions and interviewing new community members for each edition. In this issue, we probed people's opinions on interpretable AI, the value of benchmarking in modern NLP, the state of progress towards understanding deep learning, and the future of academia.

Semantic segmentation is a core task in computer vision. Existing methods are generally divided into two categories: automatic and interactive. Interactive approaches, exemplified by the Segment Anything Model (SAM), have shown promise as pre-trained models. However, current adaptation strategies for these models tend to lean towards either automatic or interactive approaches. Interactive methods depend on prompts user input to operate, while automatic ones bypass the interactive promptability entirely. Addressing these limitations, we introduce a novel paradigm and its first model: the Automatic and Interactive Segment Anything Model (AI-SAM). In this paradigm, we conduct a comprehensive analysis of prompt quality and introduce the pioneering Automatic and Interactive Prompter (AI-Prompter) that automatically generates initial point prompts while accepting additional user inputs. Our experimental results demonstrate AI-SAM's effectiveness in the automatic setting, achieving state-of-the-art performance. Significantly, it offers the flexibility to incorporate additional user prompts, thereby further enhancing its performance. The project page is available at https://github.com/ymp5078/AI-SAM., Comment: 11 pages, 9 figures