Search

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,700 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai., Comment: 27 pages, 6 figures

Performance of large language models (LLMs) on medical tasks has traditionally been evaluated using multiple choice question benchmarks. However, such benchmarks are highly constrained, saturated with repeated impressive performance by LLMs, and have an unclear relationship to performance in real clinical scenarios. Clinical reasoning, the process by which physicians employ critical thinking to gather and synthesize clinical data to diagnose and manage medical problems, remains an attractive benchmark for model performance. Prior LLMs have shown promise in outperforming clinicians in routine and complex diagnostic scenarios. We sought to evaluate OpenAI's o1-preview model, a model developed to increase run-time via chain of thought processes prior to generating a response. We characterize the performance of o1-preview with five experiments including differential diagnosis generation, display of diagnostic reasoning, triage differential diagnosis, probabilistic reasoning, and management reasoning, adjudicated by physician experts with validated psychometrics. Our primary outcome was comparison of the o1-preview output to identical prior experiments that have historical human controls and benchmarks of previous LLMs. Significant improvements were observed with differential diagnosis generation and quality of diagnostic and management reasoning. No improvements were observed with probabilistic reasoning or triage differential diagnosis. This study highlights o1-preview's ability to perform strongly on tasks that require complex critical thinking such as diagnosis and management while its performance on probabilistic reasoning tasks was similar to past models. New robust benchmarks and scalable evaluation of LLM capabilities compared to human physicians are needed along with trials evaluating AI in real clinical settings.

For any pair of integers $m$ and $n$ such that $3

For certain values of the wave speed parameter, evolution equations for the temperature of a region of fuel admit traveling wave solutions describing fire fronts. We consider such a system in the form of a nonlinear reaction-diffusion equation with a first-order forcing term capturing the combined effects of ambient and fire-induced wind. The fire-induced wind is introduced by way of a piecewise continuous function that ``switches'' in space. We demonstrate that, in the case of a spatially dependent wind, traveling wave solutions corresponding to fire fronts exist for a continuum of wave speeds rather than for a single unique speed. Using geometric methods, we determine the range of allowable speeds, refine this range to only those fronts which will persist in nature, and develop a selection mechanism to identify the specific wind configuration corresponding to the most stable solution. For this spectrally preferred front, we find that the wind switch occurs ahead of the fireline in a manner consistent with the physics of air entrainment. Even when the wind is not coupled to the temperature and is instead imposed as an external forcing, the conditions on the existence and stability of front solutions force the wind term in to a configuration reflective of physical reality.

In wildfires, atmospheric turbulence plays a major role in the transfer of turbulent kinetic energy. Understanding how turbulence feeds back into a dynamical system is important, down to the varying small scales of fuel structures (i.e. pine needles, grass). Large eddy simulations (LES) are a common way of numerically representing turbulence. The Smagorinsky model (1963) serves as one of the most studied sub-grid scale representations in LES. In this investigation, the Smagorinsky model was implemented in HIGRAD/FIRETEC, LANL's coupled fire-atmosphere model. The Smagorinsky turbulent kinetic energy (TKE) was compared to FIRETEC's 1.5-order TKE eddy-viscosity subgrid-scale model, known as the Linn turbulence model. This was done in simulations of flow over flat terrain with a homogeneous, cuboidal canopy in the center of the domain. Examinations of the modeled vertical TKE profile and turbulent statistics at the leading edge, and throughout the canopy, show that the Smagorinsky model provides comparable results to that of the original closure model posed in FIRETEC., Comment: 24 pages, 10 figures

Vorticity is a key characteristic of flow patterns that determine wildland fire behavior, frontal evolution, and wind-canopy interaction. Investigating the role of vorticity in the flow fields around vegetation can help us better understand fire-atmosphere feedback and the influences of vegetation on this feedback. In modeling vorticity, ``perhaps the greatest knowledge gap exists in understanding which terms in the vorticity equation dominate [...] (and) when one or the other might dominate" (Potter, 2012). In this study, we investigate the role of vorticity in boundary layer dynamics and canopy/forest edge effects using HIGRAD/FIRETEC, a three-dimensional, two-phase transport model that conserves mass, momentum, energy, and chemical species. A vorticity transport equation was derived and discretized. Simulations were performed over a cuboidal homogeneous canopy surrounded by surface vegetation. This derivation led to the discovery of a drag tilting and stretching term, which shows that gradients in the aerodynamic drag of the vegetation, tied to heterogeneities in surface area-to-volume ratio, play an important role in the generation of vorticity. Results from the vorticity budget analysis show that this term contributes significantly in the areas where these gradients are present, namely the edges of the canopy., Comment: 24 pages, 10 figures

The Rayleigh-Taylor (RT) instability is ubiquitously observed, yet has traditionally been studied using ideal fluid models. Collisionality can vary strongly across the fluid interface, and previous work demonstrates the necessity of kinetic models to completely capture dynamics in certain collisional regimes. Where previous kinetic simulations used spatially- and temporally-constant collision frequency, this work presents 5-dimensional (two spatial, three velocity dimensions) continuum-kinetic simulations of the RT instability using a more realistic spatially-varying collision frequency. Three cases of collisional variation are explored for two Atwood numbers: low to intermediate, intermediate to high, and low to high. The low to intermediate case exhibits no RT instability growth, while the intermediate to high case is similar to a fluid limit kinetic case with interface widening biased towards the lower collisionality region. A novel contribution of this work is the low to high collisionality case that shows significantly altered instability growth through upward movement of the interface and damped spike growth due to increased free-streaming particle diffusion in the lower region. Contributions to the energy-flux from the non-Maxwellian portions of the distribution function are not accessible to fluid models and are greatest in magnitude in the spike and regions of low collisionality. Increasing the Atwood number results in greater RT instability growth and reduced upward interface movement. Deviation of the distribution function from Maxwellian is inversely proportional to collision frequency and concentrated around the fluid interface. The linear phase of RT instability growth is well-described by theoretical linear growth rates accounting for viscosity and diffusion.

Active galactic nuclei (AGN) play an integral role in galaxy formation and evolution by influencing galaxies and their environments through radio jet feedback. Historically, interpreting observations of radio galaxies and quantifying radio jet feedback has been challenging due to degeneracies between their physical parameters. In particular, it is well-established that different combinations of jet kinetic power and environment density can yield indistinguishable radio continuum properties, including apparent size and Stokes I luminosity. We present an approach to breaking this degeneracy by probing the line-of-sight environment with Faraday rotation. We study this effect in simulations of three-dimensional relativistic magnetohydrodynamic AGN jets in idealised environments with turbulent magnetic fields. We generate synthetic Stokes I emission and Faraday rotation measure (RM) maps, which enable us to distinguish between our simulated sources. We find enhanced RMs near the jet head and lobe edges and an RM reversal across the jet axis. We show that increasing the environment density and the average cluster magnetic field strength broadens the distribution of Faraday rotation measure values. We study the depolarisation properties of our sources, finding that the hotspot regions depolarise at lower frequencies than the lobes. We quantify the effect of depolarisation on the RM distribution, finding that the frequency at which the source is too depolarised to measure the RM distribution accurately is a probe of environmental properties. This technique offers a range of new opportunities for upcoming surveys, including probing radio galaxy environments and determining more accurate estimates of the AGN feedback budget., Comment: 17 pages, 11 figures. Submitted for publication in MNRAS

Multimodal foundation models have shown compelling but conflicting performance in medical image interpretation. However, the mechanisms by which these models integrate and prioritize different data modalities, including images and text, remain poorly understood. Here, using a diverse collection of 1014 multimodal medical cases, we evaluate the unimodal and multimodal image interpretation abilities of proprietary (GPT-4, Gemini Pro 1.0) and open-source (Llama-3.2-90B, LLaVA-Med-v1.5) multimodal foundational models with and without the use of text descriptions. Across all models, image predictions were largely driven by exploiting text, with accuracy increasing monotonically with the amount of informative text. By contrast, human performance on medical image interpretation did not improve with informative text. Exploitation of text is a double-edged sword; we show that even mild suggestions of an incorrect diagnosis in text diminishes image-based classification, reducing performance dramatically in cases the model could previously answer with images alone. Finally, we conducted a physician evaluation of model performance on long-form medical cases, finding that the provision of images either reduced or had no effect on model performance when text is already highly informative. Our results suggest that multimodal AI models may be useful in medical diagnostic reasoning but that their accuracy is largely driven, for better and worse, by their exploitation of text.

We present results from a pair of high resolution, long timescale ($\sim10^5 GM/c^3$), global, three dimensional magnetohydrodynamical accretion disk simulations with differing initial magnetic plasma $\beta$ in order to study the effects of initial toroidal field strength on production of large-scale poloidal field. We initialize our disks in approximate equilibrium with purely toroidal magnetic fields of strength $\beta_0=5$ and $\beta_0=200$. We also perform a limited resolution study. We find that simulations of differing field strength diverge early in their evolution and remain distinct over the time studied, indicating that initial magnetic conditions leave a persistent imprint in our simulations. Neither simulation enters the Magnetically Arrested Disk (MAD) regime. Both simulations are able to produce poloidal fields from initially-toroidal fields, with the $\beta_0=5$ simulation evolving clear signs of a large-scale poloidal field. We make a cautionary note that computational artifacts in the form of large-scale vortices may be introduced in the combination of initially-weak field and disk-internal mesh refinement boundaries, as evidenced by the production of an $m=1$ mode overdensity in the weak field simulation. Our results demonstrate that the initial toroidal field strength plays a vital role in simulated disk evolution for the models studied., Comment: 17 pages, 11 figures, accepted to ApJ

Rayleigh-Taylor (RT) instabilities are prevalent in many physical regimes ranging from astrophysical to laboratory plasmas and have primarily been studied using fluid models, the majority of which have been ideal fluid models. This work is the first of its kind to present a 5-dimensional (2 spatial dimensions, 3 velocity space dimensions) simulation using the continuum-kinetic model to study the effect of the collisional mean-free-path and transport on the instability growth. The continuum-kinetic model provides noise-free access to the full particle distribution function permitting a detailed investigation of the role of kinetic physics in hydrodynamic phenomena such as the RT instability. For long mean-free-path, there is no RT instability growth, but as collisionality increases, particles relax towards the Maxwellian velocity distribution, and the kinetic simulations reproduce the fluid simulation results. An important and novel contribution of this work is in the intermediate collisional cases that are not accessible with traditional fluid models and require kinetic modeling. Simulations of intermediate collisional cases show that the RT instability evolution is significantly altered compared to the highly collisional fluid-like cases. Specifically, the growth rate of the intermediate collisionality RT instability is lower than the high collisionality case while also producing a significantly more diffused interface. The higher moments of the distribution function play a more significant role relative to inertial terms for intermediate collisionality during the evolution of the RT instability interface. Particle energy-flux is calculated from moments of the distribution and shows that transport is significantly altered in the intermediate collisional case and deviates much more so from the high collisionality limit of the fluid regime.