Search

This article is concerned with the approximation and expressive powers of deep neural networks. This is an active research area currently producing many interesting papers. The results most commonly found in the literature prove that neural networks approximate functions with classical smoothness to the same accuracy as classical linear methods of approximation, e.g., approximation by polynomials or by piecewise polynomials on prescribed partitions. However, approximation by neural networks depending on n parameters is a form of nonlinear approximation and as such should be compared with other nonlinear methods such as variable knot splines or n-term approximation from dictionaries. The performance of neural networks in targeted applications such as machine learning indicate that they actually possess even greater approximation power than these traditional methods of nonlinear approximation. The main results of this article prove that this is indeed the case. This is done by exhibiting large classes of functions which can be efficiently captured by neural networks where classical nonlinear methods fall short of the task. The present article purposefully limits itself to studying the approximation of univariate functions by ReLU networks. Many generalizations to functions of several variables and other activation functions can be envisioned. However, even in this simplest of settings considered here, a theory that completely quantifies the approximation power of neural networks is still lacking.

This article provides a non-asymptotic analysis of the singular values (and Lyapunov exponents) of Gaussian matrix products in the regime where N, the number of terms in the product, is large and n, the size of the matrices, may be large or small and may depend on N. We obtain concentration estimates for sums of Lyapunov exponents, a quantitative rate for convergence of the empirical measure of the squared singular values to the uniform distribution on [0, 1], and results on the joint normality of Lyapunov exponents when N is sufficiently large as a function of n. Our technique consists of non-asymptotic versions of the ergodic theory approach at $$N=\infty $$ due originally to Furstenberg and Kesten (Ann Math Stat 31(2):457–469, 1960) in the 1960s, which were then further developed by Newman (Commun Math Phys 103(1):121–126, 1986) and Isopi and Newman (Commun Math Phys 143(3):591–598, 1992) as well as by a number of other authors in the 1980s. Our key technical idea is that small ball probabilities for volumes of random projections gives a way to quantify convergence in the multiplicative ergodic theorem for random matrices.

This textbook establishes a theoretical framework for understanding deep learning models of practical relevance. With an approach that borrows from theoretical physics, Roberts and Yaida provide clear and pedagogical explanations of how realistic deep neural networks actually work. To make results from the theoretical forefront accessible, the authors eschew the subject's traditional emphasis on intimidating formality without sacrificing accuracy. Straightforward and approachable, this volume balances detailed first-principle derivations of novel results with insight and intuition for theorists and practitioners alike. This self-contained textbook is ideal for students and researchers interested in artificial intelligence with minimal prerequisites of linear algebra, calculus, and informal probability theory, and it can easily fill a semester-long course on deep learning theory. For the first time, the exciting practical advances in modern artificial intelligence capabilities can be matched with a set of effective principles, providing a timeless blueprint for theoretical research in deep learning.

This paper concerns the asymptotic behavior of zeros and critical points for monochromatic random waves $$\phi _\lambda $$ of frequency $$\lambda $$ on a compact, smooth, Riemannian manifold (M, g) as $$\lambda \rightarrow \infty .$$ We prove global variance estimates for the measures of integration over the zeros and critical points of $$\phi _\lambda .$$ These global estimates hold for a wide class of manifolds—for example when (M, g) has no conjugate points—and rely on new local variance estimates on zeros and critical points of $$\phi _\lambda $$ in balls of radius $$\approx \lambda ^{-1}$$ around a fixed point. Our local results hold under conditions about the structure of geodesics that are generic in the space of all metrics on M.

We prove that smooth Wigner–Weyl spectral sums at an energy level E exhibit Airy scaling asymptotics across the classical energy surface Σ E . This was proved earlier by the authors for the isotropic harmonic oscillator and the proof is extended in this article to all quantum Hamiltonians −ℏ 2Δ + V where V is a confining potential with at most quadratic growth at infinity. The main tools are the Herman–Kluk initial value parametrix for the propagator and the Chester–Friedman–Ursell normal form for complex phases with a one-dimensional cubic degeneracy. This gives a rigorous account of Airy scaling asymptotics of spectral Wigner distributions of Berry, Ozorio de Almeida and other physicists.

This article concerns new off-diagonal estimates on the remainder and its derivatives in the pointwise Weyl law on a compact n-dimensional Riemannian manifold. As an application, we prove that near any non-self-focal point, the scaling limit of the spectral projector of the Laplacian onto frequency windows of constant size is a normalized Bessel function depending only on n.

We study the scaling asymptotics of the eigenspace projection kernels $\Pi_{\hbar, E}(x,y)$ of the isotropic Harmonic Oscillator $- \hbar ^2 \Delta + |x|^2$ of eigenvalue $E = \hbar(N + \frac{d}{2})$ in the semi-classical limit $\hbar \to 0$. The principal result is an explicit formula for the scaling asymptotics of $\Pi_{\hbar, E}(x,y)$ for $x,y$ in a $\hbar^{2/3}$ neighborhood of the caustic $\mathcal C_E$ as $\hbar \to 0.$ The scaling asymptotics are applied to the distribution of nodal sets of Gaussian random eigenfunctions around the caustic as $\hbar \to 0$. In previous work we proved that the density of zeros of Gaussian random eigenfunctions of $\hat{H}_{\hbar}$ have different orders in the Planck constant $\hbar$ in the allowed and forbidden regions: In the allowed region the density is of order $\hbar^{-1}$ while it is $\hbar^{-1/2}$ in the forbidden region. Our main result on nodal sets is that the density of zeros is of order $\hbar^{-\frac{2}{3}}$ in an $\hbar^{\frac{2}{3}}$-tube around the caustic. This tube radius is the `critical radius'. For annuli of larger inner and outer radii $\hbar^{\alpha}$ with $0< \alpha < \frac{2}{3}$ we obtain density results which interpolate between this critical radius result and our prior ones in the allowed and forbidden region. We also show that the Hausdorff $(d-2)$-dimensional measure of the intersection of the nodal set with the caustic is of order $\hbar^{- \frac{2}{3}}$., Comment: v3. Accepted to Communications in Mathematical Physics

Eigenspaces of the quantum isotropic Harmonic Oscillator $\hat{H}_{\hbar} : = - \frac{\hbar^2}{2} \Delta + \frac{||x||^2}{2}$ on $\mathbb{R}^d$ have extremally high multiplicites and the eigenspace projections $\Pi_{\hbar, E_N(\hbar)} $ have special asymptotic properties. This article gives a detailed study of their Wigner distributions $W_{\hbar, E_N(\hbar)}(x, \xi)$. Heuristically, if $E_N(\hbar) = E$, $W_{\hbar, E_N(\hbar)}(x, \xi)$ is the `quantization' of the energy surface $\Sigma_E$, and should be like the delta-function $\delta_{\Sigma_E}$ on $\Sigma_E$; rigorously, $W_{\hbar, E_N(\hbar)}(x, \xi)$ tends in a weak* sense to $\delta_{\Sigma_E}$. But its pointwise asymptotics and scaling asymptotics have more structure. The main results give Bessel asymptotics of $W_{\hbar, E_N(\hbar)}(x, \xi)$ in the interior $H(x, \xi) < E$ of $\Sigma_E$; interface Airy scaling asymptotics in tubes of radius $\hbar^{2/3}$ around $\Sigma_E$, with $(x, \xi)$ either in the interior or exterior of the energy ball; and exponential decay rates in the exterior of the energy surface., Comment: 28 pages, 3 figures, v2. Statement of Theorem 1.5 slightly modified. Several references added

To each generic complex polynomial $p(z)$ there is associated a labeled binary tree (here referred to as a "lemniscate tree") that encodes the topological type of the graph of $|p(z)|$. The branching structure of the lemniscate tree is determined by the configuration (i.e., arrangement in the plane) of the singular components of those level sets $|p(z)|=t$ passing through a critical point. In this paper, we address the question "How many branches appear in a typical lemniscate tree?" We answer this question first for a lemniscate tree sampled uniformly from the combinatorial class and second for the lemniscate tree arising from a random polynomial generated by i.i.d. zeros. From a more general perspective, these results take a first step toward a probabilistic treatment (within a specialized setting) of Arnold's program of enumerating algebraic Morse functions., 18 pages, 6 figures

Let (M, g) be a compact smooth Riemannian manifold. We obtain new off-diagonal estimates as {\lambda} tend to infinity for the remainder in the pointwise Weyl Law for the kernel of the spectral projector of the Laplacian onto functions with frequency at most {\lambda}. A corollary is that, when rescaled around a non self-focal point, the kernel of the spectral projector onto the frequency interval (\lambda, \lambda + 1] has a universal scaling limit as {\lambda} goes to infinity (depending only on the dimension of M). Our results also imply that if M has no conjugate points, then immersions of M into Euclidean space by an orthonormal basis of eigenfunctions with frequencies in (\lambda, \lambda + 1] are embeddings for all {\lambda} sufficiently large., Comment: Published version. Modified parametrix construction in Section 3. References added and typos corrected

A compact Riemannian manifold may be immersed into Euclidean space by using high frequency Laplace eigenfunctions. We study the geometry of the manifold viewed as a metric space endowed with the distance function from the ambient Euclidean space. As an application we give a new proof of a result of Burq-Lebeau and others on upper bounds for the sup-norms of random linear combinations of high frequency eigenfunctions., Comment: This article supersedes arXiv:1310.1361, which has now been withdrawn

We study products of random matrices in the regime where the number of terms and the size of the matrices simultaneously tend to infinity. Our main theorem is that the logarithm of the $\ell_2$ norm of such a product applied to any fixed vector is asymptotically Gaussian. The fluctuations we find can be thought of as a finite temperature correction to the limit in which first the size and then the number of matrices tend to infinity. Depending on the scaling limit considered, the mean and variance of the limiting Gaussian depend only on either the first two or the first four moments of the measure from which matrix entries are drawn. We also obtain explicit error bounds on the moments of the norm and the Kolmogorov-Smirnov distance to a Gaussian. Finally, we apply our result to obtain precise information about the stability of gradients in randomly initialized deep neural networks with ReLU activations. This provides a quantitative measure of the extent to which the exploding and vanishing gradient problem occurs in a fully connected neural network with ReLU activations and a given architecture., Comment: v1. 26p. Comments Welcome

We study properties of the nodal sets of high frequency eigenfunctions and quasimodes for radial perturbations of the Harmonic Oscillator. In particular, we consider nodal sets on spheres of large radius (in the classically forbidden region) for quasimodes with energies lying in intervals around a fixed energy $E$. For well chosen intervals we show that these nodal sets exhibit quantitatively different behavior compared to those of the unperturbed Harmonic Oscillator. These energy intervals are defined via a careful analysis of the eigenvalue spacings for the perturbed operator, based on analytic perturbation theory and linearization formulas for Laguerre polynomials., Comment: 22 pages. v1. comments welcome