Search

The main results of this paper are the establishment of sharp constants for several families of critical Sobolev embeddings. These inequalities were pioneered by David R. Adams, while the sharp constant in the first order case is due to Andrea Cianchi. We also prove a trace improvement of an inequality obtained independently by K. Hansson and H. Brezis and S. Wainger., Comment: 25 pages

The challenge of mapping AI architectures to GPU hardware is creating a critical bottleneck in AI progress. Despite substantial efforts, hand-written custom kernels fail to meet their theoretical performance thresholds, even on well-established operations like linear attention. The diverse hardware capabilities of GPUs might suggest that we need a wide variety of techniques to achieve high performance. However, our work explores whether a small number of key abstractions can drastically simplify the process. We present ThunderKittens (TK), a framework for writing performant AI kernels while remaining easy to use and maintain. Our abstractions map to the three levels of the GPU hierarchy: (1) at the warp-level, we provide 16x16 matrix tiles as basic data structures and PyTorch-like parallel compute operations over tiles, (2) at the thread-block level, we provide a template for overlapping asynchronous operations across parallel warps, and (3) at the grid-level, we provide support to help hide the block launch and tear-down, and memory costs. We show the value of TK by providing kernels that match or outperform prior kernels for a range of AI operations. We match CuBLAS and FlashAttention-3 on GEMM and attention inference performance and outperform the strongest baselines by $10-40\%$ on attention backwards, $8\times$ on state space models, and $14\times$ on linear attention.

Recent works show we can linearize large language models (LLMs) -- swapping the quadratic attentions of popular Transformer-based LLMs with subquadratic analogs, such as linear attention -- avoiding the expensive pretraining costs. However, linearizing LLMs often significantly degrades model quality, still requires training over billions of tokens, and remains limited to smaller 1.3B to 7B LLMs. We thus propose Low-rank Linear Conversion via Attention Transfer (LoLCATs), a simple two-step method that improves LLM linearizing quality with orders of magnitudes less memory and compute. We base these steps on two findings. First, we can replace an LLM's softmax attentions with closely-approximating linear attentions, simply by training the linear attentions to match their softmax counterparts with an output MSE loss ("attention transfer"). Then, this enables adjusting for approximation errors and recovering LLM quality simply with low-rank adaptation (LoRA). LoLCATs significantly improves linearizing quality, training efficiency, and scalability. We significantly reduce the linearizing quality gap and produce state-of-the-art subquadratic LLMs from Llama 3 8B and Mistral 7B v0.1, leading to 20+ points of improvement on 5-shot MMLU. Furthermore, LoLCATs does so with only 0.2% of past methods' model parameters and 0.4% of their training tokens. Finally, we apply LoLCATs to create the first linearized 70B and 405B LLMs (50x larger than prior work). When compared with prior approaches under the same compute budgets, LoLCATs significantly improves linearizing quality, closing the gap between linearized and original Llama 3.1 70B and 405B LLMs by 77.8% and 78.1% on 5-shot MMLU., Comment: 47 pages, 20 figures, 18 tables, preprint

This paper illustrates a further application of topological data analysis to the study of self-organising models for chemical and biological systems. In particular, we investigate whether topological summaries can capture the parameter dependence of pattern topology in reaction diffusion systems, by examining the homology of sublevel sets of solutions to Turing reaction diffusion systems for a range of parameters. We demonstrate that a topological clustering algorithm can reveal how pattern topology depends on parameters, using the chlorite--iodide--malonic acid system, and the prototypical Schnakenberg system for illustration. In addition, we discuss the prospective application of such clustering, for instance in refining priors for detailed parameter estimation for self-organising systems., Comment: 24 pages, 9 figures

A central question which originates in the celebrated work in the 1980's of DiPerna and Majda asks what is the optimal decay $f > 0$ such that uniform rates $|\omega|(Q) \leq f(|Q|)$ of the vorticity maximal functions guarantee strong convergence without concentrations of approximate solutions to energy-conserving weak solutions of the $2$D Euler equations with vortex sheet initial data. A famous result of Majda (1993) shows $f(r) = [\log (1/r)]^{-1/2}$, $r<1/2$, as the optimal decay for \emph{distinguished} sign vortex sheets. In the general setting of \emph{mixed} sign vortex sheets, DiPerna and Majda (1987) established $f(r) = [\log (1/r)]^{-\alpha}$ with $\alpha > 1$ as a sufficient condition for the lack of concentrations, while the expected gap $\alpha \in (1/2, 1]$ remains as an open question. In this paper we resolve the DiPerna-Majda $2$D gap problem: In striking contrast to the well-known case of distinguished sign vortex sheets, we identify $f(r) = [\log (1/r)]^{-1}$ as the optimal regularity for mixed sign vortex sheets that rules out concentrations. For the proof, we propose a novel method to construct explicitly solutions with mixed sign to the $2$D Euler equations in such a way that wild behaviour creates within the relevant geometry of \emph{sparse} cubes (i.e., these cubes are not necessarily pairwise disjoint, but their possible overlappings can be controlled in a sharp fashion). Such a strategy is inspired by the recent work of the first author and Milman \cite{DM} where strong connections between energy conservation and sparseness are established., Comment: 24 pages

The Regeneration Cavity (RC) is a critical component of the Any Light Particle Search II (ALPS II) experiment. It increases the signal from possible axions and axion-like particles in the experiment by nearly four orders of magnitude. The total round-trip optical losses of the power circulating in the cavity must be minimized in order to maximize the resonant enhancement of the cavity, which is an important figure of merit for ALPS II. Lower optical losses also increase the cavity storage time and with the 123 meter long ALPS II RC we have demonstrated the longest storage time of a two-mirror optical cavity. We measured a storage time of $7.17 \pm 0.01$ ms, equivalent to a linewidth of 44.4 Hz and a finesse of 27,500 at a wavelength of 1064 nm., Comment: 16 pages, 8 figures, 1 table

Prompts are how humans communicate with LLMs. Informative prompts are essential for guiding LLMs to produce the desired output. However, prompt engineering is often tedious and time-consuming, requiring significant expertise, limiting its widespread use. We propose Conversational Prompt Engineering (CPE), a user-friendly tool that helps users create personalized prompts for their specific tasks. CPE uses a chat model to briefly interact with users, helping them articulate their output preferences and integrating these into the prompt. The process includes two main stages: first, the model uses user-provided unlabeled data to generate data-driven questions and utilize user responses to shape the initial instruction. Then, the model shares the outputs generated by the instruction and uses user feedback to further refine the instruction and the outputs. The final result is a few-shot prompt, where the outputs approved by the user serve as few-shot examples. A user study on summarization tasks demonstrates the value of CPE in creating personalized, high-performing prompts. The results suggest that the zero-shot prompt obtained is comparable to its - much longer - few-shot counterpart, indicating significant savings in scenarios involving repetitive tasks with large text volumes.

Counterexample-driven genetic programming (CDGP) uses specifications provided as formal constraints to generate the training cases used to evaluate evolving programs. It has also been extended to combine formal constraints and user-provided training data to solve symbolic regression problems. Here we show how the ideas underlying CDGP can also be applied using only user-provided training data, without formal specifications. We demonstrate the application of this method, called ``informal CDGP,'' to software synthesis problems. Our results show that informal CDGP finds solutions faster (i.e. with fewer program executions) than standard GP. Additionally, we propose two new variants to informal CDGP, and find that one produces significantly more successful runs on about half of the tested problems. Finally, we study whether the addition of counterexample training cases to the training set is useful by comparing informal CDGP to using a static subsample of the training set, and find that the addition of counterexamples significantly improves performance.

Fine-tuning Large Language Models (LLMs) is an effective method to enhance their performance on downstream tasks. However, choosing the appropriate setting of tuning hyperparameters (HPs) is a labor-intensive and computationally expensive process. Here, we provide recommended HP configurations for practical use-cases that represent a better starting point for practitioners, when considering two SOTA LLMs and two commonly used tuning methods. We describe Coverage-based Search (CBS), a process for ranking HP configurations based on an offline extensive grid search, such that the top ranked configurations collectively provide a practical robust recommendation for a wide range of datasets and domains. We focus our experiments on Llama-3-8B and Mistral-7B, as well as full fine-tuning and LoRa, conducting a total of > 10,000 tuning experiments. Our results suggest that, in general, Llama-3-8B and LoRA should be preferred, when possible. Moreover, we show that for both models and tuning methods, exploring only a few HP configurations, as recommended by our analysis, can provide excellent results in practice, making this work a valuable resource for practitioners.

In this paper we explore several applications of the recently introduced spaces of functions of bounded $\beta$-dimensional mean oscillation for $\beta \in (0,n]$ to regularity theory of critical exponent elliptic equations. We first show that functions with gradient in weak-$L^n$ are in $BMO^\beta$ for any $\beta \in (0,n]$, improving the classical result $\nabla u\in L^n$ implies $u\in BMO$. We apply this result to the Poisson equation $-\Delta u = \operatorname*{div} F$ with zero boundary conditions in a bounded $C^1$ domain to show that $u\in BMO^{\beta}$ when $F$ is in weak-$L^n$. Next, we consider the $n$-Laplace equation \begin{align*} -\operatorname*{div}( |\nabla U|^{n-2} \nabla U) &= F \text{ in } \Omega, \newline U &=0 \text{ on }\partial \Omega. \end{align*} with $F\in L^1(\Omega)$ and show that the classical result $u\in BMO$ can be improved to $u\in BMO^\beta$. Finally, we consider the $n$-Laplace equation in the case when $F \in L^1$, $\operatorname*{div} F=0$ and prove that for smooth domains $\Omega$ we have the estimate \begin{align*} \|\nabla U \|_{L^n} \mathbb \leq C \, \|F\|^{1/(n-1)}_{L^1}, \end{align*} where the constant $C$ is independent of $F$., Comment: 18 pages, 1 figure

Recurrent large language models that compete with Transformers in language modeling perplexity are emerging at a rapid rate (e.g., Mamba, RWKV). Excitingly, these architectures use a constant amount of memory during inference. However, due to the limited memory, recurrent LMs cannot recall and use all the information in long contexts leading to brittle in-context learning (ICL) quality. A key challenge for efficient LMs is selecting what information to store versus discard. In this work, we observe the order in which information is shown to the LM impacts the selection difficulty. To formalize this, we show that the hardness of information recall reduces to the hardness of a problem called set disjointness (SD), a quintessential problem in communication complexity that requires a streaming algorithm (e.g., recurrent model) to decide whether inputted sets are disjoint. We empirically and theoretically show that the recurrent memory required to solve SD changes with set order, i.e., whether the smaller set appears first in-context. Our analysis suggests, to mitigate the reliance on data order, we can put information in the right order in-context or process prompts non-causally. Towards that end, we propose: (1) JRT-Prompt, where context gets repeated multiple times in the prompt, effectively showing the model all data orders. This gives $11.0 \pm 1.3$ points of improvement, averaged across $16$ recurrent LMs and the $6$ ICL tasks, with $11.9\times$ higher throughput than FlashAttention-2 for generation prefill (length $32$k, batch size $16$, NVidia H100). We then propose (2) JRT-RNN, which uses non-causal prefix-linear-attention to process prompts and provides $99\%$ of Transformer quality at $360$M params., $30$B tokens and $96\%$ at $1.3$B params., $50$B tokens on average across the tasks, with $19.2\times$ higher throughput for prefill than FA2.

We establish an approach to trace inequalities for potential-type operators based on an appropriate modification of an interpolation theorem due to Calder\'on. We develop a general theoretical tool for establishing boundedness of notoriously difficult operators (such as potentials) on certain specific types of rearrangement-invariant function spaces from analogous properties of operators that are easier to handle (such as fractional maximal operators). The key ingredient for the development of the theory is the initial pair of endpoint estimates for the easier operator whose pivotal example is based on a two-weight inequality of Sawyer. Among various applications we obtain a generalization of the celebrated trace inequality involving the Riesz potential and the Hausdorff content by Korobkov and Kristensen., Comment: 30 pages

The problem-solving performance of many evolutionary algorithms, including genetic programming systems used for program synthesis, depends on the values of hyperparameters including mutation rates. The mutation method used to produce some of the best results to date on software synthesis benchmark problems, Uniform Mutation by Addition and Deletion (UMAD), adds new genes into a genome at a predetermined rate and then deletes genes at a rate that balances the addition rate, producing no size change on average. While UMAD with a predetermined addition rate outperforms many other mutation and crossover schemes, we do not expect a single rate to be optimal across all problems or all generations within one run of an evolutionary system. However, many current adaptive mutation schemes such as self-adaptive mutation rates suffer from pathologies like the vanishing mutation rate problem, in which the mutation rate quickly decays to zero. We propose an adaptive bandit-based scheme that addresses this problem and essentially removes the need to specify a mutation rate. Although the proposed scheme itself introduces hyperparameters, we either set these to good values or ensemble them in a reasonable range. Results on software synthesis and symbolic regression problems validate the effectiveness of our approach., Comment: 12 pages, 4 figures. Accepted at GECCO'24

Ensuring AI models align with human values is essential for their safety and functionality. Reinforcement learning from human feedback (RLHF) uses human preferences to achieve this alignment. However, preferences sourced from diverse populations can result in point estimates of human values that may be sub-optimal or unfair to specific groups. We propose Pareto Optimal Preference Learning (POPL), which frames discrepant group preferences as objectives with potential trade-offs, aiming for policies that are Pareto-optimal on the preference dataset. POPL utilizes Lexicase selection, an iterative process to select diverse and Pareto-optimal solutions. Our empirical evaluations demonstrate that POPL surpasses baseline methods in learning sets of reward functions, effectively catering to distinct groups without access to group numbers or membership labels. Furthermore, we illustrate that POPL can serve as a foundation for techniques optimizing specific notions of group fairness, ensuring inclusive and equitable AI model alignment.

In this paper, we define spaces of measures $DS_\beta(\mathbb{R}^d)$ with dimensional stability $\beta \in (0,d)$. These spaces bridge between $M_b(\mathbb{R}^d)$, the space of finite Radon measures, and $DS_d(\mathbb{R}^d)= \mathrm{H}^1(\mathbb{R}^d)$, the real Hardy space. We show the spaces $DS_\beta(\mathbb{R}^d)$ support Sobolev inequalities for $\beta \in (0,d]$, while for any $\beta \in [0,d]$ we show that the lower Hausdorff dimension of an element of $DS_\beta(\mathbb{R}^d)$ is at least $\beta$., Comment: 30 pages

In this paper we prove extension results for functions in Besov spaces. Our results are new in the homogeneous setting, while our technique applies equally in the inhomogeneous setting to obtain new proofs of classical results. While our results include $p>1$, of principle interest is the case $p=1$, where we show that \begin{equation*} \int_{\mathbb{R}_{+}^{n+1}}t^{a}|\nabla^{m+1}u(x,t)|\;dtdx\lesssim\left\vert f\right\vert _{B^{m-a,1}(\mathbb{R}^{n})} \end{equation*} for all $f \in \dot{B}^{m-a,1}(\mathbb{R}^{n})$ (the homogeneous Besov space) where $u$ is a suitably scaled heat extension of $f$., Comment: 37 pages

Why is language vague? Vagueness may be explained and rationalized if it can be shown that vague language is more useful to speaker and hearer than precise language. In a well-known paper, Lipman proposes a game-theoretic account of vagueness in terms of mixed strategy that leads to a puzzle: vagueness cannot be strictly better than precision at equilibrium. More recently, \'Egr\'e, Spector, Mortier and Verheyen have put forward a Bayesian account of vagueness establishing that using vague words can be strictly more informative than using precise words. This paper proposes to compare both results and to explain why they are not in contradiction. Lipman's definition of vagueness relies exclusively on a property of signaling strategies, without making any assumptions about the lexicon, whereas \'Egr\'e et al.'s involves a layer of semantic content. We argue that the semantic account of vagueness is needed, and more adequate and explanatory of vagueness.

Financial firms often rely on fundamental factor models to explain correlations among asset returns and manage risk. Yet after major events, e.g., COVID-19, analysts may reassess whether existing risk models continue to fit well: specifically, after accounting for a set of known factor exposures, are the residuals of the asset returns independent? With this motivation, we introduce the mosaic permutation test, a nonparametric goodness-of-fit test for preexisting factor models. Our method can leverage modern machine learning techniques to detect model violations while provably controlling the false positive rate, i.e., the probability of rejecting a well-fitting model, without making asymptotic approximations or parametric assumptions. This property helps prevent analysts from unnecessarily rebuilding accurate models, which can waste resources and increase risk. To illustrate our methodology, we apply the mosaic permutation test to the BlackRock Fundamental Equity Risk (BFRE) model. Although the BFRE model generally explains the most significant correlations among assets, we find evidence of unexplained correlations among certain real estate stocks, and we show that adding new factors improves model fit. We implement our methods in the python package mosaicperm., Comment: 42 pages, 13 figures

The Traveling Thief Problem is an NP-hard combination of the well known traveling salesman and knapsack packing problems. In this paper, we use symbolic regression to learn useful features of near-optimal packing plans, which we then use to design efficient metaheuristic genetic algorithms for the traveling thief algorithm. By using symbolic regression again to initialize the metaheuristic GA with near-optimal individuals, we are able to design a fast, interpretable, and effective packing initialization scheme. Comparisons against previous initialization schemes validates our algorithm design., Comment: 23 pages

We describe a technique for measuring the complex reflectivity of an optical cavity with a resonant local oscillator laser and an auxiliary probe laser, each coupled via opposite ends of the cavity. A heterodyne sensing scheme is then used to observe the phase and amplitude of the interference beat-note between the promptly reflected field and the cavity transmitted field injected through the far mirror. Since the local oscillator laser must pass through the cavity before interfering with the probe laser these measurements are not only independent of the spatial coupling of either laser to the cavity, but also obtained at the in-situ position of the cavity Eigenmode. This technique was demonstrated on a 19 m cavity to measure the individual transmissivities of each of the mirrors as well as the round trip optical losses to an accuracy of several parts per million., Comment: 16 pages, 6 figures, 1 table

Lexicase selection has been shown to provide advantages over other selection algorithms in several areas of evolutionary computation and machine learning. In its standard form, lexicase selection filters a population or other collection based on randomly ordered training cases that are considered one at a time. This iterated filtering process can be time-consuming, particularly in settings with large numbers of training cases. In this paper, we propose a new method that is nearly equivalent to lexicase selection in terms of the individuals that it selects, but which does so significantly more quickly. The new method, called DALex (for Diversely Aggregated Lexicase), selects the best individual with respect to a weighted sum of training case errors, where the weights are randomly sampled. This allows us to formulate the core computation required for selection as matrix multiplication instead of recursive loops of comparisons, which in turn allows us to take advantage of optimized and parallel algorithms designed for matrix multiplication for speedup. Furthermore, we show that we can interpolate between the behavior of lexicase selection and its "relaxed" variants, such as epsilon or batch lexicase selection, by adjusting a single hyperparameter, named "particularity pressure," which represents the importance granted to each individual training case. Results on program synthesis, deep learning, symbolic regression, and learning classifier systems demonstrate that DALex achieves significant speedups over lexicase selection and its relaxed variants while maintaining almost identical problem-solving performance. Under a fixed computational budget, these savings free up resources that can be directed towards increasing population size or the number of generations, enabling the potential for solving more difficult problems., Comment: 15 pages, 4 figures. Accepted at EuroGP'24

In recent years, neuroscience has made significant progress in building large-scale artificial neural network (ANN) models of brain activity and behavior. However, there is no consensus on the most efficient ways to collect data and design experiments to develop the next generation of models. This article explores the controversial opinions that have emerged on this topic in the domain of vision and language. Specifically, we address two critical points. First, we weigh the pros and cons of using qualitative insights from empirical results versus raw experimental data to train models. Second, we consider model-free (intuition-based) versus model-based approaches for data collection, specifically experimental design and stimulus selection, for optimal model development. Finally, we consider the challenges of developing a synergistic approach to experimental design and model building, including encouraging data and model sharing and the implications of iterative additions to existing models. The goal of the paper is to discuss decision points and propose directions for both experimenters and model developers in the quest to understand the brain.

Purpose: This study aims to develop a high-resolution whole-brain multi-parametric quantitative MRI approach for simultaneous mapping of myelin-water fraction (MWF), T1, T2, and proton-density (PD), all within a clinically feasible scan time. Methods: We developed 3D ViSTa-MRF, which combined Visualization of Short Transverse relaxation time component (ViSTa) technique with MR Fingerprinting (MRF), to achieve high-fidelity whole-brain MWF and T1/T2/PD mapping on a clinical 3T scanner. To achieve fast acquisition and memory-efficient reconstruction, the ViSTa-MRF sequence leverages an optimized 3D tiny-golden-angle-shuffling spiral-projection acquisition and joint spatial-temporal subspace reconstruction with optimized preconditioning algorithm. With the proposed ViSTa-MRF approach, high-fidelity direct MWF mapping was achieved without a need for multi-compartment fitting that could introduce bias and/or noise from additional assumptions or priors. Results: The in-vivo results demonstrate the effectiveness of the proposed acquisition and reconstruction framework to provide fast multi-parametric mapping with high SNR and good quality. The in-vivo results of 1mm- and 0.66mm-iso datasets indicate that the MWF values measured by the proposed method are consistent with standard ViSTa results that are 30x slower with lower SNR. Furthermore, we applied the proposed method to enable 5-minute whole-brain 1mm-iso assessment of MWF and T1/T2/PD mappings for infant brain development and for post-mortem brain samples. Conclusions: In this work, we have developed a 3D ViSTa-MRF technique that enables the acquisition of whole-brain MWF, quantitative T1, T2, and PD maps at 1mm and 0.66mm isotropic resolution in 5 and 15 minutes, respectively. This advancement allows for quantitative investigations of myelination changes in the brain., Comment: 38 pages, 12 figures and 1 table

One potential drawback of using aggregated performance measurement in machine learning is that models may learn to accept higher errors on some training cases as compromises for lower errors on others, with the lower errors actually being instances of overfitting. This can lead to both stagnation at local optima and poor generalization. Lexicase selection is an uncompromising method developed in evolutionary computation, which selects models on the basis of sequences of individual training case errors instead of using aggregated metrics such as loss and accuracy. In this paper, we investigate how lexicase selection, in its general form, can be integrated into the context of deep learning to enhance generalization. We propose Gradient Lexicase Selection, an optimization framework that combines gradient descent and lexicase selection in an evolutionary fashion. Our experimental results demonstrate that the proposed method improves the generalization performance of various widely-used deep neural network architectures across three image classification benchmarks. Additionally, qualitative analysis suggests that our method assists networks in learning more diverse representations. Our source code is available on GitHub: https://github.com/ld-ing/gradient-lexicase., Comment: ICLR 2022

Navigating deceptive domains has often been a challenge in machine learning due to search algorithms getting stuck at sub-optimal local optima. Many algorithms have been proposed to navigate these domains by explicitly maintaining diversity or equivalently promoting exploration, such as Novelty Search or other so-called Quality Diversity algorithms. In this paper, we present an approach with promise to solve deceptive domains without explicit diversity maintenance by optimizing a potentially large set of defined objectives. These objectives can be extracted directly from the environment by sub-aggregating the raw performance of individuals in a variety of ways. We use lexicase selection to optimize for these objectives as it has been shown to implicitly maintain population diversity. We compare this technique with a varying number of objectives to a commonly used quality diversity algorithm, MAP-Elites, on a set of discrete optimization as well as reinforcement learning domains with varying degrees of deception. We find that decomposing objectives into many objectives and optimizing them outperforms MAP-Elites on the deceptive domains that we explore. Furthermore, we find that this technique results in competitive performance on the diversity-focused metrics of QD-Score and Coverage, without explicitly optimizing for these things. Our ablation study shows that this technique is robust to different subaggregation techniques. However, when it comes to non-deceptive, or ``illumination" domains, quality diversity techniques generally outperform our objective-based framework with respect to exploration (but not exploitation), hinting at potential directions for future work., Comment: Published at the Workshop on Agent Learning in Open-Endedness (ALOE) at NeurIPS 2023

Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures such as Transformers scale quadratically along both these axes. We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension? We introduce Monarch Mixer (M2), a new architecture that uses the same sub-quadratic primitive along both sequence length and model dimension: Monarch matrices, a simple class of expressive structured matrices that captures many linear transforms, achieves high hardware efficiency on GPUs, and scales sub-quadratically. As a proof of concept, we explore the performance of M2 in three domains: non-causal BERT-style language modeling, ViT-style image classification, and causal GPT-style language modeling. For non-causal BERT-style modeling, M2 matches BERT-base and BERT-large in downstream GLUE quality with up to 27% fewer parameters, and achieves up to 9.1$\times$ higher throughput at sequence length 4K. On ImageNet, M2 outperforms ViT-b by 1% in accuracy, with only half the parameters. Causal GPT-style models introduce a technical challenge: enforcing causality via masking introduces a quadratic bottleneck. To alleviate this bottleneck, we develop a novel theoretical view of Monarch matrices based on multivariate polynomial evaluation and interpolation, which lets us parameterize M2 to be causal while remaining sub-quadratic. Using this parameterization, M2 matches GPT-style Transformers at 360M parameters in pretraining perplexity on The PILE--showing for the first time that it may be possible to match Transformer quality without attention or MLPs., Comment: NeurIPS 2023 (Oral)

Reinforcement Learning from Human Feedback (RLHF) has shown potential in qualitative tasks where easily defined performance measures are lacking. However, there are drawbacks when RLHF is commonly used to optimize for average human preferences, especially in generative tasks that demand diverse model responses. Meanwhile, Quality Diversity (QD) algorithms excel at identifying diverse and high-quality solutions but often rely on manually crafted diversity metrics. This paper introduces Quality Diversity through Human Feedback (QDHF), a novel approach that progressively infers diversity metrics from human judgments of similarity among solutions, thereby enhancing the applicability and effectiveness of QD algorithms in complex and open-ended domains. Empirical studies show that QDHF significantly outperforms state-of-the-art methods in automatic diversity discovery and matches the efficacy of QD with manually crafted diversity metrics on standard benchmarks in robotics and reinforcement learning. Notably, in open-ended generative tasks, QDHF substantially enhances the diversity of text-to-image generation from a diffusion model and is more favorably received in user studies. We conclude by analyzing QDHF's scalability, robustness, and quality of derived diversity metrics, emphasizing its strength in open-ended optimization tasks. Code and tutorials are available at https://liding.info/qdhf., Comment: ICML 2024

Many causal estimands are only partially identifiable since they depend on the unobservable joint distribution between potential outcomes. Stratification on pretreatment covariates can yield sharper partial identification bounds; however, unless the covariates are discrete with relatively small support, this approach typically requires consistent estimation of the conditional distributions of the potential outcomes given the covariates. Thus, existing approaches may fail under model misspecification or if consistency assumptions are violated. In this study, we propose a unified and model-agnostic inferential approach for a wide class of partially identified estimands, based on duality theory for optimal transport problems. In randomized experiments, our approach can wrap around any estimates of the conditional distributions and provide uniformly valid inference, even if the initial estimates are arbitrarily inaccurate. Also, our approach is doubly robust in observational studies. Notably, this property allows analysts to use the multiplier bootstrap to select covariates and models without sacrificing validity even if the true model is not included. Furthermore, if the conditional distributions are estimated at semiparametric rates, our approach matches the performance of an oracle with perfect knowledge of the outcome model. Finally, we propose an efficient computational framework, enabling implementation on many practical problems in causal inference., Comment: 59 pages, 4 figures

The problem of selecting a model given a set of candidates remains a challenging one that pervades many scientific fields. We employ techniques from the theory of Lie groups to analyse the symmetries in differential equation models of population growth, with the aim of informing the model selection problem. To illustrate the use of Lie symmetries in model selection, we apply them to simulated data and to coral reef data from the Great Barrier Reef, demonstrating that the trivial symmetries can distinguish between candidate models. A method for finding locally optimal parameters for multi-parameter symmetries is presented, and the paper concludes with related results, some open problems, and avenues of further research., Comment: 15 pages, 5 figures

Nature often builds physical structures tailored for specific information processing tasks with computations encoded using diverse phenomena. These can sometimes outperform typical general-purpose computers. However, describing the construction and function of these unconventional computers is often challenging. Here, we address this by introducing constructor algorithms in the context of a robotic wire machine that can be programmed to build networks of connected wires in response to a problem and then act upon these to efficiently carry out a desired computation. We show how this approach can be used to solve the NP-complete Subset Sum Problem (SSP) and provide information about the number of solutions through changes in the voltages and currents measured across these networks. This work provides a foundation for building unconventional computers that encode information purely in the lengths and connections of electrically conductive wires. It also demonstrates the power of computing paradigms beyond digital logic and opens avenues to more fully harness the inherent computational capabilities of diverse physical, chemical and biological substrates., Comment: 14 pages, 4 figures

Recent advances with large language models (LLM) illustrate their diverse capabilities. We propose a novel algorithm, staged speculative decoding, to accelerate LLM inference in small-batch, on-device scenarios. We address the low arithmetic intensity of small-batch inference by improving upon previous work in speculative decoding. First, we restructure the speculative batch as a tree, which reduces generation costs and increases the expected tokens per batch. Second, we add a second stage of speculative decoding. Taken together, we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L model while perfectly preserving output quality., Comment: Published at ES-FOMO at ICML 2023