Search

The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These systems pose novel and under-explored risks. In this report, we provide a structured taxonomy of these risks by identifying three key failure modes (miscoordination, conflict, and collusion) based on agents' incentives, as well as seven key risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security) that can underpin them. We highlight several important instances of each risk, as well as promising directions to help mitigate them. By anchoring our analysis in a range of real-world examples and experimental evidence, we illustrate the distinct challenges posed by multi-agent systems and their implications for the safety, governance, and ethics of advanced AI., Comment: Cooperative AI Foundation, Technical Report #1

In this paper, we leverage existing statistical methods to better understand feature learning from data. We tackle this by modifying the model-free variable selection method, Feature Ordering by Conditional Independence (FOCI), which is introduced in \cite{azadkia2021simple}. While FOCI is based on a non-parametric coefficient of conditional dependence, we introduce its parametric, differentiable approximation. With this approximate coefficient of correlation, we present a new algorithm called difFOCI, which is applicable to a wider range of machine learning problems thanks to its differentiable nature and learnable parameters. We present difFOCI in three contexts: (1) as a variable selection method with baseline comparisons to FOCI, (2) as a trainable model parametrized with a neural network, and (3) as a generic, widely applicable neural network regularizer, one that improves feature learning with better management of spurious correlations. We evaluate difFOCI on increasingly complex problems ranging from basic variable selection in toy examples to saliency map comparisons in convolutional networks. We then show how difFOCI can be incorporated in the context of fairness to facilitate classifications without relying on sensitive data.

Recent studies have raised concerns about the effectiveness of Classifier-Free Guidance (CFG), indicating that in low-dimensional settings, it can lead to overshooting the target distribution and reducing sample diversity. In this work, we demonstrate that in infinite and sufficiently high-dimensional contexts CFG effectively reproduces the target distribution, revealing a blessing-of-dimensionality result. Additionally, we explore finite-dimensional effects, precisely characterizing overshoot and variance reduction. Based on our analysis, we introduce non-linear generalizations of CFG. Through numerical simulations on Gaussian mixtures and experiments on class-conditional and text-to-image diffusion models, we validate our analysis and show that our non-linear CFG offers improved flexibility and generation quality without additional computation cost.

Cosmic Voids are a promising probe of cosmology for spectroscopic galaxy surveys due to their unique response to cosmological parameters. Their combination with other probes promises to break parameter degeneracies. Due to simplifying assumptions, analytical models for void statistics are only representative of a subset of the full void population. We present a set of neural-based emulators for void summary statistics of watershed voids, which retain more information about the full void population than simplified analytical models. We build emulators for the void size function and void density profiles traced by the halo number density using the Quijote suite of simulations for a broad range of the $\Lambda\mathrm{CDM}$ parameter space. The emulators replace the computation of these statistics from computationally expensive cosmological simulations. We demonstrate the cosmological constraining power of voids using our emulators, which offer orders-of-magnitude acceleration in parameter estimation, capture more cosmological information compared to analytic models, and produce more realistic posteriors compared to Fisher forecasts. We find that the parameters $\Omega_m$ and $\sigma_8$ in this Quijote setup can be recovered to $14.4\%$ and $8.4\%$ accuracy respectively using void density profiles; including the additional information in the void size function improves the accuracy on $\sigma_8$ to $6.8\%$. We demonstrate the robustness of our approach to two important variables in the underlying simulations, the resolution, and the inclusion of baryons. We find that our pipeline is robust to variations in resolution, and we show that the posteriors derived from the emulated void statistics are unaffected by the inclusion of baryons with the Magneticum hydrodynamic simulations. This opens up the possibility of a baryon-independent probe of the large-scale structure., Comment: 13 pages, 10 figures

This paper claims that machine learning (ML) largely overlooks an important facet of general intelligence: robustness to a qualitatively unknown future in an open world. Such robustness relates to Knightian uncertainty (KU) in economics, i.e. uncertainty that cannot be quantified, which is excluded from consideration in ML's key formalisms. This paper aims to identify this blind spot, argue its importance, and catalyze research into addressing it, which we believe is necessary to create truly robust open-world AI. To help illuminate the blind spot, we contrast one area of ML, reinforcement learning (RL), with the process of biological evolution. Despite staggering ongoing progress, RL still struggles in open-world situations, often failing under unforeseen situations. For example, the idea of zero-shot transferring a self-driving car policy trained only in the US to the UK currently seems exceedingly ambitious. In dramatic contrast, biological evolution routinely produces agents that thrive within an open world, sometimes even to situations that are remarkably out-of-distribution (e.g. invasive species; or humans, who do undertake such zero-shot international driving). Interestingly, evolution achieves such robustness without explicit theory, formalisms, or mathematical gradients. We explore the assumptions underlying RL's typical formalisms, showing how they limit RL's engagement with the unknown unknowns characteristic of an ever-changing complex world. Further, we identify mechanisms through which evolutionary processes foster robustness to novel and unpredictable challenges, and discuss potential pathways to algorithmically embody them. The conclusion is that the intriguing remaining fragility of ML may result from blind spots in its formalisms, and that significant gains may result from direct confrontation with the challenge of KU.

Do horror writers have worse childhoods than other writers? Though biographical details are known about many writers, quantitatively exploring such a qualitative hypothesis requires significant human effort, e.g. to sift through many biographies and interviews of writers and to iteratively search for quantitative features that reflect what is qualitatively of interest. This paper explores the potential to quickly prototype these kinds of hypotheses through (1) applying LLMs to estimate properties of concrete entities like specific people, companies, books, kinds of animals, and countries; (2) performing off-the-shelf analysis methods to reveal possible relationships among such properties (e.g. linear regression); and towards further automation, (3) applying LLMs to suggest the quantitative properties themselves that could help ground a particular qualitative hypothesis (e.g. number of adverse childhood events, in the context of the running example). The hope is to allow sifting through hypotheses more quickly through collaboration between human and machine. Our experiments highlight that indeed, LLMs can serve as useful estimators of tabular data about specific entities across a range of domains, and that such estimations improve with model scale. Further, initial experiments demonstrate the potential of LLMs to map a qualitative hypothesis of interest to relevant concrete variables that the LLM can then estimate. The conclusion is that LLMs offer intriguing potential to help illuminate scientifically interesting patterns latent within the internet-scale data they are trained upon.

How much cosmological information can we reliably extract from existing and upcoming large-scale structure observations? Many summary statistics fall short in describing the non-Gaussian nature of the late-time Universe in comparison to existing and upcoming measurements. In this article we demonstrate that we can identify optimal summary statistics and that we can link them with existing summary statistics. Using simulation based inference (SBI) with automatic data-compression, we learn summary statistics for galaxy catalogs in the context of cosmological parameter estimation. By construction these summary statistics do not require the ability to write down an explicit likelihood. We demonstrate that they can be used for efficient parameter inference. These summary statistics offer a new avenue for analyzing different simulation models for baryonic physics with respect to their relevance for the resulting cosmological features. The learned summary statistics are low-dimensional, feature the underlying simulation parameters, and are similar across different network architectures. To link our models, we identify the relevant scales associated to our summary statistics (e.g. in the range of modes between $k= 5 - 30 h/\mathrm{Mpc}$) and we are able to match the summary statistics to underlying simulation parameters across various simulation models., Comment: 44 pages, 14 figures

Natural-language assistants are designed to provide users with helpful responses while avoiding harmful outputs, largely achieved through alignment to human preferences. Yet there is limited understanding of whether alignment techniques may inadvertently perpetuate or even amplify harmful biases inherited from their pre-aligned base models. This issue is compounded by the choice of bias evaluation benchmarks in popular preference-finetuned models, which predominantly focus on dominant social categories, such as binary gender, thereby limiting insights into biases affecting underrepresented groups. Towards addressing this gap, we center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias in LLMs. Our key contributions include: 1) a comprehensive survey of bias evaluation modalities across leading preference-finetuned LLMs, highlighting critical gaps in gender-diverse representation, 2) systematic evaluation of gender-diverse biases across 12 models spanning Direct Preference Optimization (DPO) stages, uncovering harms popular bias benchmarks fail to detect, and 3) a flexible framework for measuring harmful biases in implicit reward signals applicable to other social contexts. Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning (SFT), and can amplify two forms of real-world gender-diverse harms from their base models: stigmatization and gender non-affirmative language. We conclude with recommendations tailored to DPO and broader alignment practices, advocating for the adoption of community-informed bias evaluation frameworks to more effectively identify and address underrepresented harms in LLMs., Comment: Accepted to 2024 Neurips Queer in AI Workshop

Social media platforms have evolved rapidly in modernity without strong regulation. One clear obstacle faced by current users is that of toxicity. Toxicity on social media manifests through a number of forms, including harassment, negativity, misinformation or other means of divisiveness. In this paper, we characterize literature surrounding toxicity, formalize a definition of toxicity, propose a novel cycle of internet extremism, list current approaches to toxicity detection, outline future directions to minimize toxicity in future social media endeavors, and identify current gaps in research space. We present a novel perspective of the negative impacts of social media platforms and fill a gap in literature to help improve the future of social media platforms., Comment: 12 pages, 2 figures, 2 tables, Cycle of Internet Extremism

We present nifty-ls, a software package for fast and accurate evaluation of the Lomb-Scargle periodogram. nifty-ls leverages the fact that Lomb-Scargle can be computed using a non-uniform FFT (NUFFT), which we evaluate with the Flatiron Institute NUFFT package (finufft). This approach achieves a many-fold speedup over the Press & Rybicki (1989) method as implemented in Astropy and is simultaneously many orders of magnitude more accurate. nifty-ls also supports fast evaluation on GPUs via CUDA and integrates with the Astropy Lomb-Scargle interface. nifty-ls is publicly available as open-source software., Comment: 4 pages, 1 figure, submitted to RNAAS. Software available at https://github.com/flatironinstitute/nifty-ls/

Recognizing errors in assembly and maintenance procedures is valuable for industrial applications, since it can increase worker efficiency and prevent unplanned down-time. Although assembly state recognition is gaining attention, none of the current works investigate assembly error localization. Therefore, we propose StateDiffNet, which localizes assembly errors based on detecting the differences between a (correct) intended assembly state and a test image from a similar viewpoint. StateDiffNet is trained on synthetically generated image pairs, providing full control over the type of meaningful change that should be detected. The proposed approach is the first to correctly localize assembly errors taken from real ego-centric video data for both states and error types that are never presented during training. Furthermore, the deployment of change detection to this industrial application provides valuable insights and considerations into the mechanisms of state-of-the-art change detection algorithms. The code and data generation pipeline are publicly available at: https://timschoonbeek.github.io/error_seg., Comment: 23 pages (14 main paper, 2 references, 7 supplementary), 15 figures (8 main paper, 7 supplementary). Accepted at ECCV Vision-based InduStrial InspectiON (VISION) workshop

In the context of medical decision making, counterfactual prediction enables clinicians to predict treatment outcomes of interest under alternative courses of therapeutic actions given observed patient history. In this work, we present G-Transformer for counterfactual outcome prediction under dynamic and time-varying treatment strategies. Our approach leverages a Transformer architecture to capture complex, long-range dependencies in time-varying covariates while enabling g-computation, a causal inference method for estimating the effects of dynamic treatment regimes. Specifically, we use a Transformer-based encoder architecture to estimate the conditional distribution of relevant covariates given covariate and treatment history at each time point, then produces Monte Carlo estimates of counterfactual outcomes by simulating forward patient trajectories under treatment strategies of interest. We evaluate G-Transformer extensively using two simulated longitudinal datasets from mechanistic models, and a real-world sepsis ICU dataset from MIMIC-IV. G-Transformer outperforms both classical and state-of-the-art counterfactual prediction models in these settings. To the best of our knowledge, this is the first Transformer-based architecture that supports g-computation for counterfactual outcome prediction under dynamic and time-varying treatment strategies.

Empirical evidence for a gap between the computational powers of classical and quantum computers has been provided by experiments that sample the output distributions of two-dimensional quantum circuits. Many attempts to close this gap have utilized classical simulations based on tensor network techniques, and their limitations shed light on the improvements to quantum hardware required to frustrate classical simulability. In particular, quantum computers having in excess of $\sim 50$ qubits are primarily vulnerable to classical simulation due to restrictions on their gate fidelity and their connectivity, the latter determining how many gates are required (and therefore how much infidelity is suffered) in generating highly-entangled states. Here, we describe recent hardware upgrades to Quantinuum's H2 quantum computer enabling it to operate on up to $56$ qubits with arbitrary connectivity and $99.843(5)\%$ two-qubit gate fidelity. Utilizing the flexible connectivity of H2, we present data from random circuit sampling in highly connected geometries, doing so at unprecedented fidelities and a scale that appears to be beyond the capabilities of state-of-the-art classical algorithms. The considerable difficulty of classically simulating H2 is likely limited only by qubit number, demonstrating the promise and scalability of the QCCD architecture as continued progress is made towards building larger machines., Comment: Includes minor updates to the text and an updated author list to include researchers who made technical contributions in upgrading the machine to 56 qubits but were left off the original version by mistake

We measure the clustering of Lyman Alpha Emitting galaxies (LAEs) selected from the One-hundred-square-degree DECam Imaging in Narrowbands (ODIN) survey, with spectroscopic follow-up from Dark Energy Spectroscopic Instrument (DESI). We use DESI spectroscopy to optimize our selection and to constrain the interloper fraction and redshift distribution of our narrow-band selected sources. We select samples of 4000 LAEs at z=2.45 and 3.1 in 9 sq.deg. centered on the COSMOS field with median LyA fluxes of 10^{-16}erg/s/cm2. Covariances and cosmological inferences are obtained from a series of mock catalogs built upon high-resolution N-body simulations that match the footprint, number density, redshift distribution and observed clustering of the sample. We find that both samples have a correlation length of r_0=(3.0\pm 0.2)Mpc/h. Within our fiducial cosmology these correspond to 3D number densities of 10^{-3} h^3/Mpc^3 and, from our mock catalogs, biases of 1.7 and 2.0 at z=2.45 and 3.1, respectively. We discuss the implications of these measurements for the use of LAEs as large-scale structure tracers for high-redshift cosmology., Comment: 26 pages, 13 figures; minor revisions to match version accepted by JCAP

We present the development of a new astrochemical research tool HILTRAC, the Highly Instrumented Low Temperature ReAction Chamber. The instrument is based on a pulsed form of the CRESU (Cin\'etique de R\'eaction en \'Ecoulement Supersonique Uniforme, meaning reaction kinetics in a uniform supersonic flow) apparatus, with the aim of collecting kinetics and spectroscopic information on gas phase chemical reactions important in interstellar space or planetary atmospheres. We discuss the apparatus design and its flexibility, the implementation of pulsed laser photolysis followed by laser induced fluorescence (PLP-LIF), and the first implementation of direct infrared frequency comb spectroscopy (DFCS) coupled to the uniform supersonic flow. Achievable flow temperatures range from 32(3) - 111(9) K, characterising a total of five Laval nozzles for use with N2 and Ar buffer gases by pressure impact measurements. These results were further validated using LIF and DFCS measurements of the CH radical and OCS, respectively. Spectroscopic constants and linelists for OCS are reported for the 1001 band near $2890 - 2940 cm^{-1}$ for both $OC^{32}S$ and $OC^{34}S$, measured using DFCS. Additional peaks in the spectrum are tentatively assigned to the OCS-Ar complex. The first reaction rate coefficients for the CH + OCS reaction measured between 32(3) K and 58(5) K are reported. The reaction rate coefficient at 32(3) K was measured to be $3.9(4) \times 10^{10} cm^3 molecule^{-1} s^{-1}$ and the reaction was found to exhibit no observable temperature dependence over this low temperature range., Comment: The following article has been submitted to The Journal of Chemical Physics. After it is published, it will be found at https://publishing.aip.org/resources/librarians/products/journals/ Copyright 2024 Daniel I. Lucas, Th\'eo Guillaume, Dwayne E. Heard, and Julia H. Lehman. This article is distributed under a Creative Commons Attribution (CC BY) License

Using a scale-free $N$-body simulation generated with the ABACUS $N$-body code, we test the robustness of halo mass accretion histories via their convergence to self-similarity. We compare two halo finders, ROCKSTAR and COMPASO. We find superior self-similarity in halo mass accretion histories determined using ROCKSTAR, with convergence to 5% or better between $\sim10^2$ to $10^5$ particles. For COMPASO we find weaker convergence over a similar region, with at least 10% between $\sim10^2$ to $10^4$ particles. Furthermore, we find the convergence to self-similarity improves as the simulation evolves, with the largest and deepest regions of convergence appearing after the scale factor quadrupled from the time at which non-linear structures begin to form. With sufficient time evolution, halo mass accretion histories are converged to self-similarity within 5% with as few as $\sim70$ particles for COMPASO and within 2% for as few as $\sim30$ particles for ROCKSTAR., Comment: 19 pages, 12 figures

This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user, across one or more domains, in line with the user's expectations. The paper starts by considering the technology itself, providing an overview of AI assistants, their technical foundations and potential range of applications. It then explores questions around AI value alignment, well-being, safety and malicious uses. Extending the circle of inquiry further, we next consider the relationship between advanced AI assistants and individual users in more detail, exploring topics such as manipulation and persuasion, anthropomorphism, appropriate relationships, trust and privacy. With this analysis in place, we consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants. Finally, we conclude by providing a range of recommendations for researchers, developers, policymakers and public stakeholders.

A measurement of a primordial non-Gaussianity (PNG) signal through late- or early-Universe probes has the potential to transform our understanding of the physics of the primordial Universe. While large-scale structure observables in principle contain vital information, interpreting these measurements is challenging due to poorly understood astrophysical effects. Luckily, $N$-body simulations, such as the public \textsc{AbacusPNG} set presented in this study, consisting of 9 boxes, each of size $L_{\rm box} = 2~{\rm Gpc}/h$ and particle mass of $1.01 \times 10^{10} \ M_\odot/h$, provide a viable path forward. As validation, we find good agreement between the simulations and our expectations from one-loop perturbation theory and the `separate universe' method for the matter bispectrum, matter power spectrum and the halo bias parameter associated with PNG, $b_\phi$. As a science application, we investigate the link between halo assembly bias and $b_\phi$ for halo properties known to play a vital role in accurately predicting galaxy clustering: concentration, shear (environment), and accretion rate. We find a strong response for all three parameters, suggesting that the connection between $b_\phi$ and the assembly history of halos needs to be taken into account by future PNG analyses. We further perform the first study of the $b_\phi$ parameter from fits to early DESI data of the luminous red galaxy (LRG) and quasi-stellar object (QSO) samples and comment on the effect on $f_{\rm NL}$ constraints for the allowed galaxy-halo models (note that $\sigma [f_{\rm NL}] \propto \frac{\sigma [b_\phi]}{b_\phi}$). We find that the error on $f_{\rm NL}$ is 21, 6, 22 for the LRGs at $z = 0.5$ and $z = 0.8$ and QSOs at $z = 1.4$, respectively, suggesting that a thorough understanding of galaxy assembly bias is warranted so as to perform robust high-precision analysis of local-type PNG with future surveys., Comment: 21 pages, 7 figures, 2 tables

Gamma-Ray Bursts (GRBs), due to their high luminosities are detected up to redshift 10, and thus have the potential to be vital cosmological probes of early processes in the universe. Fulfilling this potential requires a large sample of GRBs with known redshifts, but due to observational limitations, only 11\% have known redshifts ($z$). There have been numerous attempts to estimate redshifts via correlation studies, most of which have led to inaccurate predictions. To overcome this, we estimated GRB redshift via an ensemble supervised machine learning model that uses X-ray afterglows of long-duration GRBs observed by the Neil Gehrels Swift Observatory. The estimated redshifts are strongly correlated (a Pearson coefficient of 0.93) and have a root mean square error, namely the square root of the average squared error $\langle\Delta z^2\rangle$, of 0.46 with the observed redshifts showing the reliability of this method. The addition of GRB afterglow parameters improves the predictions considerably by 63\% compared to previous results in peer-reviewed literature. Finally, we use our machine learning model to infer the redshifts of 154 GRBs, which increase the known redshifts of long GRBs with plateaus by 94\%, a significant milestone for enhancing GRB population studies that require large samples with redshift., Comment: 12 Figures, 24 pages. Accepted for publication at The Astrophysical Journal Supplement Series

Understanding the physical mechanism behind the formation of a co-rotating thin plane of satellite galaxies, like the one observed around the Milky Way (MW), has been challenging. The perturbations induced by a massive satellite galaxy, like the Large Magellanic Cloud (LMC) provide valuable insight into this problem. The LMC induces an apparent co-rotating motion in the outer halo by displacing the inner regions of the halo with respect to the outer halo. Using the Latte suite of FIRE-2 cosmological simulations of MW-mass galaxies, we confirm that the apparent motion of the outer halo induced by the infall of a massive satellite changes the observed distribution of orbital poles of outer-halo tracers, including satellites. We quantify the changes in the distribution of orbital poles using the two-point angular correlation function and find that all satellites induce changes. However, the most massive satellites with pericentric passages between 30-100kpc induce the largest changes. The best LMC-like satellite analog shows the largest change in orbital pole distribution. The dispersion of orbital poles decreases by 20{\deg} during the first two pericentric passages. Even when excluding the satellites brought in with the LMC-like satellite, there is clustering of orbital poles. These results suggest that in the MW, the recent pericentric passage of the LMC should have changed the observed distribution of orbital poles of all other satellites. Therefore, studies of kinematically-coherent planes of satellites that seek to place the MW in a cosmological context should account for the existence of a massive satellite like the LMC., Comment: 20 pages, 10 figures. ApJ submitted, Comments are welcome

Sepsis is a life-threatening condition triggered by an extreme infection response. Our objective is to forecast sepsis patient outcomes using their medical history and treatments, while learning interpretable state representations to assess patients' risks in developing various adverse outcomes. While neural networks excel in outcome prediction, their limited interpretability remains a key issue. In this work, we use knowledge distillation via constrained variational inference to distill the knowledge of a powerful "teacher" neural network model with high predictive power to train a "student" latent variable model to learn interpretable hidden state representations to achieve high predictive performance for sepsis outcome prediction. Using real-world data from the MIMIC-IV database, we trained an LSTM as the "teacher" model to predict mortality for sepsis patients, given information about their recent history of vital signs, lab values and treatments. For our student model, we use an autoregressive hidden Markov model (AR-HMM) to learn interpretable hidden states from patients' clinical time series, and use the posterior distribution of the learned state representations to predict various downstream outcomes, including hospital mortality, pulmonary edema, need for diuretics, dialysis, and mechanical ventilation. Our results show that our approach successfully incorporates the constraint to achieve high predictive power similar to the teacher model, while maintaining the generative performance., Comment: Extended Abstract presented at Machine Learning for Health (ML4H) symposium 2023, December 10th, 2023, New Orleans, United States, 12 pages

Automated classification of cancer pathology reports can extract information from unstructured reports and categorize each report into structured diagnosis and severity categories. Thus, such system can reduce the burden for populating tumor registries, help registration for clinical trial as well as developing large dataset for deep learning model development using true pathologic ground truth. However, the content of breast pathology reports can be difficult for categorize due to the high linguistic variability in content and wide variety of potential diagnoses >50. Existing NLP models are primarily focused on developing classifier for primary breast cancer types (e.g. IDC, DCIS, ILC) and tumor characteristics, and ignore the rare diagnosis of cancer subtypes. We then developed a hierarchical hybrid transformer-based pipeline (59 labels) - Hierarchical Classification System for Breast Cancer Specimen Report (HCSBC), which utilizes the potential of the transformer context-preserving NLP technique and compared our model to several state of the art ML and DL models. We trained the model on the EUH data and evaluated our model's performance on two external datasets - MGH and Mayo Clinic. We publicly release the code and a live application under Huggingface spaces repository

A recent line of empirical studies has demonstrated that SGD might exhibit a heavy-tailed behavior in practical settings, and the heaviness of the tails might correlate with the overall performance. In this paper, we investigate the emergence of such heavy tails. Previous works on this problem only considered, up to our knowledge, online (also called single-pass) SGD, in which the emergence of heavy tails in theoretical findings is contingent upon access to an infinite amount of data. Hence, the underlying mechanism generating the reported heavy-tailed behavior in practical settings, where the amount of training data is finite, is still not well-understood. Our contribution aims to fill this gap. In particular, we show that the stationary distribution of offline (also called multi-pass) SGD exhibits 'approximate' power-law tails and the approximation error is controlled by how fast the empirical distribution of the training data converges to the true underlying data distribution in the Wasserstein metric. Our main takeaway is that, as the number of data points increases, offline SGD will behave increasingly 'power-law-like'. To achieve this result, we first prove nonasymptotic Wasserstein convergence bounds for offline SGD to online SGD as the number of data points increases, which can be interesting on their own. Finally, we illustrate our theory on various experiments conducted on synthetic data and neural networks., Comment: In Neural Information Processing Systems (NeurIPS), Spotlight Presentation, 2023