Search

We show that the shortest $s$-$t$ path problem has the overlap-gap property in (i) sparse $\mathbf{G}(n,p)$ graphs and (ii) complete graphs with i.i.d. Exponential edge weights. Furthermore, we demonstrate that in sparse $\mathbf{G}(n,p)$ graphs, shortest path is solved by $O(\log n)$-degree polynomial estimators, and a uniform approximate shortest path can be sampled in polynomial time. This constitutes the first example in which the overlap-gap property is not predictive of algorithmic intractability for a (non-algebraic) average-case optimization problem., Comment: 29 pages

Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it. This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities. Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them. If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation. To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning., Comment: Updated organization of experimental results and methods introduction. Released the dataset and model weights artifact

The binary perceptron problem asks us to find a sign vector in the intersection of independently chosen random halfspaces with intercept $-\kappa$. We analyze the performance of the canonical discrepancy minimization algorithms of Lovett-Meka and Rothvoss/Eldan-Singh for the asymmetric binary perceptron problem. We obtain new algorithmic results in the $\kappa = 0$ case and in the large-$|\kappa|$ case. In the $\kappa\to-\infty$ case, we additionally characterize the storage capacity and complement our algorithmic results with an almost-matching overlap-gap lower bound., Comment: 58 pages

Gaussian mixture block models are distributions over graphs that strive to model modern networks: to generate a graph from such a model, we associate each vertex $i$ with a latent feature vector $u_i \in \mathbb{R}^d$ sampled from a mixture of Gaussians, and we add edge $(i,j)$ if and only if the feature vectors are sufficiently similar, in that $\langle u_i,u_j \rangle \ge \tau$ for a pre-specified threshold $\tau$. The different components of the Gaussian mixture represent the fact that there may be different types of nodes with different distributions over features -- for example, in a social network each component represents the different attributes of a distinct community. Natural algorithmic tasks associated with these networks are embedding (recovering the latent feature vectors) and clustering (grouping nodes by their mixture component). In this paper we initiate the study of clustering and embedding graphs sampled from high-dimensional Gaussian mixture block models, where the dimension of the latent feature vectors $d\to \infty$ as the size of the network $n \to \infty$. This high-dimensional setting is most appropriate in the context of modern networks, in which we think of the latent feature space as being high-dimensional. We analyze the performance of canonical spectral clustering and embedding algorithms for such graphs in the case of 2-component spherical Gaussian mixtures, and begin to sketch out the information-computation landscape for clustering and embedding in these models., Comment: 50 pages

It was recently shown that almost all solutions in the symmetric binary perceptron are isolated, even at low constraint densities, suggesting that finding typical solutions is hard. In contrast, some algorithms have been shown empirically to succeed in finding solutions at low density. This phenomenon has been justified numerically by the existence of subdominant and dense connected regions of solutions, which are accessible by simple learning algorithms. In this paper, we establish formally such a phenomenon for both the symmetric and asymmetric binary perceptrons. We show that at low constraint density (equivalently for overparametrized perceptrons), there exists indeed a subdominant connected cluster of solutions with almost maximal diameter, and that an efficient multiscale majority algorithm can find solutions in such a cluster with high probability, settling in particular an open problem posed by Perkins-Xu '21. In addition, even close to the critical threshold, we show that there exist clusters of linear diameter for the symmetric perceptron, as well as for the asymmetric perceptron under additional assumptions.

In this paper we consider coalescing random walks on a general connected graph $G=(V,E)$. We set up a unified framework to study the leading order of the decay rate of $P_t$, the expectation of the fraction of occupied sites at time $t$, particularly for the `Big Bang' regime where $t\ll t_{\text{coal}}:=\mathbb{E}[\inf\{s:\text{There is only one particle at time }s\}]$. Our results show that $P_t$ satisfies certain mean field behavior, if the graphs satisfy certain transience-like conditions. We apply this framework to two families of graphs: (1) graphs given by the configuration model with a degree distribution supported in $[3,\bar d]$ for some $\bar d\geq 3$, and (2) finite and infinite vertex-transitive graphs. In the first case, we show that for $1 \ll t \ll |V|$, $P_t$ decays in the order of $t^{-1}$, and $(tP_t)^{-1}$ is approximately the probability that two particles starting from the root of the corresponding unimodular Galton-Watson tree never collide after one of them leaves the root, which is also roughly $|V|/(2t_{\text{meet}})$, where $t_{\text{meet}}$ is the mean meeting time of two walkers. By taking the local weak limit, for the unimodular Galton-Watson tree we prove the convergence of $tP_t$ as $t\to\infty$. For the second family of graphs, if we take a sequence of finite graphs $G_n=(V_n, E_n)$, such that $t_{\text{meet}}=O(|V_n|)$ and the inverse of the spectral gap $t_{\text{rel}}$ is $o(|V_n|)$, then for $t_{\text{rel}}\ll t\ll t_{\text{coal}}$, $(tP_t)^{-1}$ is approximately the probability that two random walks never meet before time $t$, and also $|V|/(2t_{\text{meet}})$. In addition, we define a natural uniform transience condition, and show that it implies the above for all $1\ll t\ll t_{\text{coal}}$. Such estimates of $tP_t$ are also obtained for all infinite transient transitive unimodular graphs, in particular, all transient transitive amenable graphs., Comment: 72 pages

We consider the symmetric binary perceptron model, a simple model of neural networks that has gathered significant attention in the statistical physics, information theory and probability theory communities, with recent connections made to the performance of learning algorithms in Baldassi et al. '15. We establish that the partition function of this model, normalized by its expected value, converges to a lognormal distribution. As a consequence, this allows us to establish several conjectures for this model: (i) it proves the contiguity conjecture of Aubin et al. '19 between the planted and unplanted models in the satisfiable regime; (ii) it establishes the sharp threshold conjecture; (iii) it proves the frozen 1-RSB conjecture in the symmetric case, conjectured first by Krauth-M\'ezard '89 in the asymmetric case. In a recent work of Perkins-Xu '21, the last two conjectures were also established by proving that the partition function concentrates on an exponential scale, under an analytical assumption on a real-valued function. This left open the contiguity conjecture and the lognormal limit characterization, which are established here unconditionally, with the analytical assumption verified. In particular, our proof technique relies on a dense counter-part of the small graph conditioning method, which was developed for sparse models in the celebrated work of Robinson and Wormald.

The problem of learning graphons has attracted considerable attention across several scientific communities, with significant progress over the recent years in sparser regimes. Yet, the current techniques still require diverging degrees in order to succeed with efficient algorithms in the challenging cases where the local structure of the graph is homogeneous. This paper provides an efficient algorithm to learn graphons in the constant expected degree regime. The algorithm is shown to succeed in estimating the rank-$k$ projection of a graphon in the $L_2$ metric if the top $k$ eigenvalues of the graphon satisfy a generalized Kesten-Stigum condition., Comment: 32 pages

We give a discrete characterization of the trace of a class of Sobolev spaces on the Sierpinski gasket to the bottom line. This includes the L2 domain of the Laplacian as a special case. In addition, for Sobolev spaces of low orders, including the domain of the Dirichlet form, the trace spaces are Besov spaces on the line., Comment: 18 pages, 4 figures