Search

Classical kernel machines have historically faced significant challenges in scaling to large datasets and model sizes--a key ingredient that has driven the success of neural networks. In this paper, we present a new methodology for building kernel machines that can scale efficiently with both data size and model size. Our algorithm introduces delayed projections to Preconditioned Stochastic Gradient Descent (PSGD) allowing the training of much larger models than was previously feasible, pushing the practical limits of kernel-based learning. We validate our algorithm, EigenPro4, across multiple datasets, demonstrating drastic training speed up over the existing methods while maintaining comparable or better classification accuracy., Comment: arXiv admin note: text overlap with arXiv:2302.02605

A fundamental question is understanding the rate at which random quantum circuits converge to the Haar measure. One quantity which is important in establishing this rate is the spectral gap of a random quantum ensemble. In this work we establish a new bound on the spectral gap of the t-th moment of a one-dimensional brickwork architecture on N qudits. This bound is independent of both t and N, provided t does not exceed the qudit dimension q. We also show that the bound is nearly optimal. The improved spectral gaps gives large improvements to the constant factors in known results on the approximate t-design depths of the 1D brickwork, of generic circuit architectures, and of specially-constructed architectures which scramble in depth O(log N). We moreover show that the spectral gap gives the dominant epsilon-dependence of the t-design depth at small epsilon. Our spectral gap bound is obtained by bounding the N-site 1D brickwork architecture by the spectra of 3-site operators. We then exploit a block-triangular hierarchy and a global symmetry in these operators in order to efficiently bound them. The technical methods used are a qualitatively different approach for bounding spectral gaps and and have little in common with previous techniques., Comment: 53 pages, 6 figures

Recent advances in machine learning have led to increased interest in reproducing kernel Banach spaces (RKBS) as a more general framework that extends beyond reproducing kernel Hilbert spaces (RKHS). These works have resulted in the formulation of representer theorems under several regularized learning schemes. However, little is known about an optimization method that encompasses these results in this setting. This paper addresses a learning problem on Banach spaces endowed with a reproducing kernel, focusing on efficient optimization within RKBS. To tackle this challenge, we propose an algorithm based on mirror descent (MDA). Our approach involves an iterative method that employs gradient steps in the dual space of the Banach space using the reproducing kernel. We analyze the convergence properties of our algorithm under various assumptions and establish two types of results: first, we identify conditions under which a linear convergence rate is achievable, akin to optimization in the Euclidean setting, and provide a proof of the linear rate; second, we demonstrate a standard convergence rate in a constrained setting. Moreover, to instantiate this algorithm in practice, we introduce a novel family of RKBSs with $p$-norm ($p \neq 2$), characterized by both an explicit dual map and a kernel., Comment: 42 pages, 3 figures

Transformers exhibit In-Context Learning (ICL), where these models solve new tasks by using examples in the prompt without additional training. In our work, we identify and analyze two key components of ICL: (1) context-scaling, where model performance improves as the number of in-context examples increases and (2) task-scaling, where model performance improves as the number of pre-training tasks increases. While transformers are capable of both context-scaling and task-scaling, we empirically show that standard Multi-Layer Perceptrons (MLPs) with vectorized input are only capable of task-scaling. To understand how transformers are capable of context-scaling, we first propose a significantly simplified transformer architecture without key, query, value weights. We show that it performs ICL comparably to the original GPT-2 model in various statistical learning tasks including linear regression, teacher-student settings. Furthermore, a single block of our simplified transformer can be viewed as data dependent feature map followed by an MLP. This feature map on its own is a powerful predictor that is capable of context-scaling but is not capable of task-scaling. We show empirically that concatenating the output of this feature map with vectorized data as an input to MLPs enables both context-scaling and task-scaling. This finding provides a simple setting to study context and task-scaling for ICL.

We study the Laplacian of the undirected De Bruijn graph over an alphabet $A$ of order $k$. While the eigenvalues of this Laplacian were found in 1998 by Delorme and Tillich [1], an explicit description of its eigenvectors has remained elusive. In this work, we find these eigenvectors in closed form and show that they yield a natural and canonical basis for the cut- and cycle-spaces of De Bruijn graphs. Remarkably, we find that the cycle basis we construct is a basis for the cycle space of both the undirected and the directed De Bruijn graph. This is done by developing an analogue of the Fourier transform on the De Bruijn graph, which acts to diagonalize the Laplacian. Moreover, we show that the cycle-space of De Bruijn graphs, when considering all possible orders of $k$ simultaneously, contains a rich algebraic structure, that of a graded Hopf algebra.

Nonlinear metasurfaces offer a new paradigm to realize optical nonlinear devices with new and unparalleled behavior compared to nonlinear crystals, due to the interplay between photonic resonances and materials properties. The complicated interdependency between efficiency and emission directionality of the nonlinear optical signal on the existence, localization, and lifetimes of photonic resonances, as well as on the nonlinear susceptibility, makes it extremely difficult to design optimal metasurfaces using conventional materials and geometries. Inverse design using topology optimization is a powerful design tool for photonic structures, but traditional approaches developed for linear photonics are not suitable for such high dimensional nonlinear problems. Here, we use a topology optimization approach to inverse-design a fabrication-robust nonlinear metasurface that includes quantum-engineered resonant nonlinearities in semiconductor heterostructures for efficient and directional second harmonic generation. Furthermore, we also demonstrate that under practical constraints, among all the parameters, the nonlinear modal overlap emerges as the dominant parameter that enhances conversion efficiency, a finding that contrasts with intuition-driven studies that often emphasize Purcell enhancement. Our results open new opportunities for optimizing nonlinear processes in nanophotonic structures for novel light sources, quantum information applications, and communication.

Neural networks trained to solve modular arithmetic tasks exhibit grokking, a phenomenon where the test accuracy starts improving long after the model achieves 100% training accuracy in the training process. It is often taken as an example of "emergence", where model ability manifests sharply through a phase transition. In this work, we show that the phenomenon of grokking is not specific to neural networks nor to gradient descent-based optimization. Specifically, we show that this phenomenon occurs when learning modular arithmetic with Recursive Feature Machines (RFM), an iterative algorithm that uses the Average Gradient Outer Product (AGOP) to enable task-specific feature learning with general machine learning models. When used in conjunction with kernel machines, iterating RFM results in a fast transition from random, near zero, test accuracy to perfect test accuracy. This transition cannot be predicted from the training loss, which is identically zero, nor from the test loss, which remains constant in initial iterations. Instead, as we show, the transition is completely determined by feature learning: RFM gradually learns block-circulant features to solve modular arithmetic. Paralleling the results for RFM, we show that neural networks that solve modular arithmetic also learn block-circulant features. Furthermore, we present theoretical evidence that RFM uses such block-circulant features to implement the Fourier Multiplication Algorithm, which prior work posited as the generalizing solution neural networks learn on these tasks. Our results demonstrate that emergence can result purely from learning task-relevant features and is not specific to neural architectures nor gradient descent-based optimization methods. Furthermore, our work provides more evidence for AGOP as a key mechanism for feature learning in neural networks.

Time-domain astrophysics continues to grow rapidly, with the inception of new surveys drastically increasing data volumes. Democratised, distributed approaches to training sets for machine learning classifiers are crucial to make the most of this torrent of discovery -- with citizen science approaches proving effective at meeting these requirements. In this paper, we describe the creation of and the initial results from the $\textit{Kilonova Seekers}$ citizen science project, built to find transient phenomena from the GOTO telescopes in near real-time. $\textit{Kilonova Seekers}$ launched in July 2023 and received over 600,000 classifications from approximately 2,000 volunteers over the course of the LIGO-Virgo-KAGRA O4a observing run. During this time, the project has yielded 20 discoveries, generated a `gold-standard' training set of 17,682 detections for augmenting deep-learned classifiers, and measured the performance and biases of Zooniverse volunteers on real-bogus classification. This project will continue throughout the lifetime of GOTO, pushing candidates at ever-greater cadence, and directly facilitate the next-generation classification algorithms currently in development., Comment: 20 pages, 15 figures. Accepted in MNRAS

We present the largest optical photometry compilation of Gamma-Ray Bursts (GRBs) with redshifts ($z$). We include 64813 observations of 535 events (including upper limits) from 28 February 1997 up to 18 August 2023. We also present a user-friendly web tool \textit{grbLC} which allows users the visualization of photometry, coordinates, redshift, host galaxy extinction, and spectral indices for each event in our database. Furthermore, we have added a Gamma Ray Coordinate Network (GCN) scraper that can be used to collect data by gathering magnitudes from the GCNs. The web tool also includes a package for uniformly investigating colour evolution. We compute the optical spectral indices for 138 GRBs for which we have at least 4 filters at the same epoch in our sample and craft a procedure to distinguish between GRBs with and without colour evolution. By providing a uniform format and repository for the optical catalogue, this web-based archive is the first step towards unifying several community efforts to gather the photometric information for all GRBs with known redshifts. This catalogue will enable population studies by providing light curves (LCs) with better coverage since we have gathered data from different ground-based locations. Consequently, these LCs can be used to train future LC reconstructions for an extended inference of the redshift. The data gathering also allows us to fill some of the orbital gaps from Swift in crucial points of the LCs, e.g., at the end of the plateau emission or where a jet break is identified., Comment: 20 pages, 16 figures, 2 tables. Submitted to MNRAS, this version matches the third revision. The Online Materials and data will be available after the publication

Nonlinear intersubband polaritonic metasurfaces support one of the strongest known ultrafast nonlinear responses in the mid-infrared frequency range across all condensed matter systems. Beyond harmonic generation and frequency mixing, these nonlinearities can be leveraged for ultrafast optical switching and power limiting, based on tailored transitions from strong to weak polaritonic coupling. Here, we demonstrate synergistic optimization of materials and photonic nanostructures to achieve large reflection contrast in ultrafast polaritonic metasurface limiters. The devices are based on optimized semiconductor heterostructure materials that minimize the intersubband transition linewidth and reduce absorption in optically saturated nanoresonators, achieving a record-high reflection contrast of 54% experimentally. We also discuss opportunities to further boost the metrics of performance of this class of ultrafast limiters, showing that reflection contrast as high as 94% may be realistically achieved using all-dielectric intersubband polaritonic metasurfaces.

Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been measured in a variety of settings, its emergence is typically explained via data-agnostic approaches, such as the unconstrained features model. In this work, we introduce a data-dependent setting where DNC forms due to feature learning through the average gradient outer product (AGOP). The AGOP is defined with respect to a learned predictor and is equal to the uncentered covariance matrix of its input-output gradients averaged over the training dataset. The Deep Recursive Feature Machine (Deep RFM) is a method that constructs a neural network by iteratively mapping the data with the AGOP and applying an untrained random feature map. We demonstrate empirically that DNC occurs in Deep RFM across standard settings as a consequence of the projection with the AGOP matrix computed at each layer. Further, we theoretically explain DNC in Deep RFM in an asymptotic setting and as a result of kernel learning. We then provide evidence that this mechanism holds for neural networks more generally. In particular, we show that the right singular vectors and values of the weights can be responsible for the majority of within-class variability collapse for DNNs trained in the feature learning regime. As observed in recent work, this singular structure is highly correlated with that of the AGOP.

Mitigating the retention of sensitive or private information in large language models is essential for enhancing privacy and safety. Existing unlearning methods, like Gradient Ascent and Negative Preference Optimization, directly tune models to remove unwanted information. However, these methods often become unstable because they fine-tune by maximizing cross-entropy loss, which is the opposite of traditional loss minimization in learning. This reversal creates instability, especially on larger datasets, as the model struggles to balance unlearning with maintaining language capacity, leading to over-unlearning. In this paper, we introduce UnDIAL (Unlearning via Self-Distillation on Adjusted Logits), a novel and robust unlearning method. Our approach leverages self-distillation to adjust logits and selectively reduce the influence of targeted tokens. This technique ensures smooth convergence and avoids catastrophic forgetting, even in challenging unlearning tasks with large datasets and sequential unlearning requests. Extensive experiments show that UnDIAL can achieve both robustness in unlearning and scalability while maintaining stable training dynamics and resilience to hyperparameter tuning.

A fundamental problem in machine learning is to understand how neural networks make accurate predictions, while seemingly bypassing the curse of dimensionality. A possible explanation is that common training algorithms for neural networks implicitly perform dimensionality reduction - a process called feature learning. Recent work posited that the effects of feature learning can be elicited from a classical statistical estimator called the average gradient outer product (AGOP). The authors proposed Recursive Feature Machines (RFMs) as an algorithm that explicitly performs feature learning by alternating between (1) reweighting the feature vectors by the AGOP and (2) learning the prediction function in the transformed space. In this work, we develop the first theoretical guarantees for how RFM performs dimensionality reduction by focusing on the class of overparametrized problems arising in sparse linear regression and low-rank matrix recovery. Specifically, we show that RFM restricted to linear models (lin-RFM) generalizes the well-studied Iteratively Reweighted Least Squares (IRLS) algorithm. Our results shed light on the connection between feature learning in neural networks and classical sparse recovery algorithms. In addition, we provide an implementation of lin-RFM that scales to matrices with millions of missing entries. Our implementation is faster than the standard IRLS algorithm as it is SVD-free. It also outperforms deep linear networks for sparse linear regression and low-rank matrix completion.

We present full photometric coverage and spectroscopic data for soft GRB 201015A with a redshift z = 0.426. Our data spans a time range of 85 days following the detection of GRB. These observations revealed an underlying supernova SN 201015A with a maximum at $8.54 \pm $1.48 days (rest frame) and an optical peak absolute magnitude $-19.45_{-0.47}^{+0.85}$ mag. The supernova stands out clearly, since the contribution of the afterglow at this time is not dominant, which made it possible to determine SN's parameters. A comparison of these parameters reveals that the SN 201015A is the earliest (the minimum $T_{max}$) known supernova associated with gamma-ray bursts. Spectroscopic observations during the supernova decay stage showed broad lines, indicating a large photospheric velocity, and identified this supernova as a type Ic-BL. Thus, the SN 201015A associated with the GRB 201015A becomes the 27th SN/GRB confirmed by both photometric and spectroscopic observations. Using the results of spectral analysis based on the available data of Fermi-GBM experiment, the parameters $E_\text{p,i} = 20.0 \pm 8.5$ keV and $E_\text{iso} = (1.1 \pm 0.2) \times 10^{50}$ erg were obtained. According to the position of the burst on the $E_\text{p,i}$-$E_\text{iso}$ correlation, GRB 201015A was classified as a Type II (long) gamma-ray burst, which was also confirmed by the $T_\text{90,i}$-$EH$ diagram., Comment: 15 pages, 10 figures

Kernel methods are a popular class of nonlinear predictive models in machine learning. Scalable algorithms for learning kernel models need to be iterative in nature, but convergence can be slow due to poor conditioning. Spectral preconditioning is an important tool to speed-up the convergence of such iterative algorithms for training kernel models. However computing and storing a spectral preconditioner can be expensive which can lead to large computational and storage overheads, precluding the application of kernel methods to problems with large datasets. A Nystrom approximation of the spectral preconditioner is often cheaper to compute and store, and has demonstrated success in practical applications. In this paper we analyze the trade-offs of using such an approximated preconditioner. Specifically, we show that a sample of logarithmic size (as a function of the size of the dataset) enables the Nystrom-based approximated preconditioner to accelerate gradient descent nearly as well as the exact preconditioner, while also reducing the computational and storage overheads.

In our era of enormous neural networks, empirical progress has been driven by the philosophy that more is better. Recent deep learning practice has found repeatedly that larger model size, more data, and more computation (resulting in lower training loss) improves performance. In this paper, we give theoretical backing to these empirical observations by showing that these three properties hold in random feature (RF) regression, a class of models equivalent to shallow networks with only the last layer trained. Concretely, we first show that the test risk of RF regression decreases monotonically with both the number of features and the number of samples, provided the ridge penalty is tuned optimally. In particular, this implies that infinite width RF architectures are preferable to those of any finite width. We then proceed to demonstrate that, for a large class of tasks characterized by powerlaw eigenstructure, training to near-zero training loss is obligatory: near-optimal performance can only be achieved when the training error is much smaller than the test error. Grounding our theory in real-world data, we find empirically that standard computer vision tasks with convolutional neural tangent kernels clearly fall into this class. Taken together, our results tell a simple, testable story of the benefits of overparameterization, overfitting, and more data in random feature models., Comment: Appeared in ICLR 2024

Optically resonant particles are key building blocks of many nanophotonic devices such as optical antennas and metasurfaces. Because the functionalities of such devices are largely determined by the optical properties of individual resonators, extending the attainable responses from a given particle is highly desirable. Practically, this is usually achieved by introducing an asymmetric dielectric environment. However, commonly used simple substrates have limited influences on the optical properties of the particles atop. Here, we show that the multipolar scattering of silicon microspheres can be effectively modified by placing the particles on a dielectric-covered mirror, which tunes the coupling between the Mie resonances of microspheres and the standing waves and waveguide modes in the dielectric spacer. This tunability allows selective excitation, enhancement, and suppression of the multipolar resonances and enables scattering at extended wavelengths, providing new opportunities in controlling light-matter interactions for various applications. We further demonstrate with experiments the detection of molecular fingerprints by single-particle mid-infrared spectroscopy, and, with simulations strong optical repulsive forces that could elevate the particles from a substrate., Comment: 16 pages, 4 figures

Unitary t-designs are distributions on the unitary group whose first t moments appear maximally random. Previous work has established several upper bounds on the depths at which certain specific random quantum circuit ensembles approximate t-designs. Here we show that these bounds can be extended to any fixed architecture of Haar-random two-site gates. This is accomplished by relating the spectral gaps of such architectures to those of 1D brickwork architectures. Our bound depends on the details of the architecture only via the typical number of layers needed for a block of the circuit to form a connected graph over the sites. When this quantity is independent of width, the circuit forms an approximate t-design in linear depth. We also give an implicit bound for nondeterministic architectures in terms of properties of the corresponding distribution over fixed architectures., Comment: 29 pages, 8 figures

Understanding the mechanism of how convolutional neural networks learn features from image data is a fundamental problem in machine learning and computer vision. In this work, we identify such a mechanism. We posit the Convolutional Neural Feature Ansatz, which states that covariances of filters in any convolutional layer are proportional to the average gradient outer product (AGOP) taken with respect to patches of the input to that layer. We present extensive empirical evidence for our ansatz, including identifying high correlation between covariances of filters and patch-based AGOPs for convolutional layers in standard neural architectures, such as AlexNet, VGG, and ResNets pre-trained on ImageNet. We also provide supporting theoretical evidence. We then demonstrate the generality of our result by using the patch-based AGOP to enable deep feature learning in convolutional kernel machines. We refer to the resulting algorithm as (Deep) ConvRFM and show that our algorithm recovers similar features to deep convolutional networks including the notable emergence of edge detectors. Moreover, we find that Deep ConvRFM overcomes previously identified limitations of convolutional kernels, such as their inability to adapt to local signals in images and, as a result, leads to sizable performance improvement over fixed convolutional kernels.

We present the results of our analysis of multiwavelength observations for the long gamma-ray burst GRB 200829A. The burst redshift $z \approx 1.29 \pm 0.04$ has been determined photometrically at the afterglow phase. In gamma rays the event is one of the brightest (in isotropic equivalent), $E_{iso} \gtrsim 10^{54}$ erg. The multicolor light curve of the GRB 200829A afterglow is characterized by chromatic behavior and the presence of a plateau gradually transitioning into a power-law decay that can also be interpreted as a quasi-synchronous inhomogeneity (flare). We assume that the presence of a chromatic inhomogeneity in the early afterglow is consistent with the model of a structured jet., Comment: 18 pages, 14 figures

In this paper, we first present an explanation regarding the common occurrence of spikes in the training loss when neural networks are trained with stochastic gradient descent (SGD). We provide evidence that the spikes in the training loss of SGD are "catapults", an optimization phenomenon originally observed in GD with large learning rates in [Lewkowycz et al. 2020]. We empirically show that these catapults occur in a low-dimensional subspace spanned by the top eigenvectors of the tangent kernel, for both GD and SGD. Second, we posit an explanation for how catapults lead to better generalization by demonstrating that catapults promote feature learning by increasing alignment with the Average Gradient Outer Product (AGOP) of the true predictor. Furthermore, we demonstrate that a smaller batch size in SGD induces a larger number of catapults, thereby improving AGOP alignment and test performance., Comment: ICML 2024

Modern machine learning paradigms, such as deep learning, occur in or close to the interpolation regime, wherein the number of model parameters is much larger than the number of data samples. In this work, we propose a regularity condition within the interpolation regime which endows the stochastic gradient method with the same worst-case iteration complexity as the deterministic gradient method, while using only a single sampled gradient (or a minibatch) in each iteration. In contrast, all existing guarantees require the stochastic gradient method to take small steps, thereby resulting in a much slower linear rate of convergence. Finally, we demonstrate that our condition holds when training sufficiently wide feedforward neural networks with a linear output layer.

When random label noise is added to a training dataset, the prediction error of a neural network on a label-noise-free test dataset initially improves during early training but eventually deteriorates, following a U-shaped dependence on training time. This behaviour is believed to be a result of neural networks learning the pattern of clean data first and fitting the noise later in the training, a phenomenon that we refer to as clean-priority learning. In this study, we aim to explore the learning dynamics underlying this phenomenon. We theoretically demonstrate that, in the early stage of training, the update direction of gradient descent is determined by the clean subset of training data, leaving the noisy subset has minimal to no impact, resulting in a prioritization of clean learning. Moreover, we show both theoretically and experimentally, as the clean-priority learning goes on, the dominance of the gradients of clean samples over those of noisy samples diminishes, and finally results in a termination of the clean-priority learning and fitting of the noisy samples.

Nearly all practical neural models for classification are trained using cross-entropy loss. Yet this ubiquitous choice is supported by little theoretical or empirical evidence. Recent work (Hui & Belkin, 2020) suggests that training using the (rescaled) square loss is often superior in terms of the classification accuracy. In this paper we propose the "squentropy" loss, which is the sum of two terms: the cross-entropy loss and the average square loss over the incorrect classes. We provide an extensive set of experiments on multi-class classification problems showing that the squentropy loss outperforms both the pure cross entropy and rescaled square losses in terms of the classification accuracy. We also demonstrate that it provides significantly better model calibration than either of these alternative losses and, furthermore, has less variance with respect to the random initialization. Additionally, in contrast to the square loss, squentropy loss can typically be trained using exactly the same optimization parameters, including the learning rate, as the standard cross-entropy loss, making it a true "plug-and-play" replacement. Finally, unlike the rescaled square loss, multiclass squentropy contains no parameters that need to be adjusted., Comment: 18 pages, 16 figures, 6 tables

Recent studies indicate that kernel machines can often perform similarly or better than deep neural networks (DNNs) on small datasets. The interest in kernel machines has been additionally bolstered by the discovery of their equivalence to wide neural networks in certain regimes. However, a key feature of DNNs is their ability to scale the model size and training data size independently, whereas in traditional kernel machines model size is tied to data size. Because of this coupling, scaling kernel machines to large data has been computationally challenging. In this paper, we provide a way forward for constructing large-scale general kernel models, which are a generalization of kernel machines that decouples the model and data, allowing training on large datasets. Specifically, we introduce EigenPro 3.0, an algorithm based on projected dual preconditioned SGD and show scaling to model and data sizes which have not been possible with existing kernel methods., Comment: Code is available at github.com/EigenPro/EigenPro3

The importance of tasks in information retrieval (IR) has been long argued for, addressed in different ways, often ignored, and frequently revisited. For decades, scholars made a case for the role that a user's task plays in how and why that user engages in search and what a search system should do to assist. But for the most part, the IR community has been too focused on query processing and assuming a search task to be a collection of user queries, often ignoring if or how such an assumption addresses the users accomplishing their tasks. With emerging areas of conversational agents and proactive IR, understanding and addressing users' tasks has become more important than ever before. In this paper, we provide various perspectives on where the state-of-the-art is with regard to tasks in IR, what are some of the bottlenecks in deriving and using task information, and how do we go forward from here. In addition to covering relevant literature, the paper provides a synthesis of historical and current perspectives on understanding, extracting, and addressing task-focused search. To ground ongoing and future research in this area, we present a new framing device for tasks using a tree-like structure and various moves on that structure that allow different interpretations and applications. Presented as a combination of synthesis of ideas and past works, proposals for future research, and our perspectives on technical, social, and ethical considerations, this paper is meant to help revitalize the interest and future work in task-based IR.

In recent years neural networks have achieved impressive results on many technological and scientific tasks. Yet, the mechanism through which these models automatically select features, or patterns in data, for prediction remains unclear. Identifying such a mechanism is key to advancing performance and interpretability of neural networks and promoting reliable adoption of these models in scientific applications. In this paper, we identify and characterize the mechanism through which deep fully connected neural networks learn features. We posit the Deep Neural Feature Ansatz, which states that neural feature learning occurs by implementing the average gradient outer product to up-weight features strongly related to model output. Our ansatz sheds light on various deep learning phenomena including emergence of spurious features and simplicity biases and how pruning networks can increase performance, the "lottery ticket hypothesis." Moreover, the mechanism identified in our work leads to a backpropagation-free method for feature learning with any machine learning model. To demonstrate the effectiveness of this feature learning mechanism, we use it to enable feature learning in classical, non-feature learning models known as kernel machines and show that the resulting models, which we refer to as Recursive Feature Machines, achieve state-of-the-art performance on tabular data.