Search

Graham, R\"odl, and Ruci\'nski originally posed the problem of determining the minimum number of monochromatic Schur triples that must appear in any 2-coloring of the first $n$ integers. This question was subsequently resolved independently by Datskovsky, Schoen, and Robertson and Zeilberger. Here we suggest studying a natural anti-Ramsey variant of this question and establish the first non-trivial bounds by proving that the maximum fraction of Schur triples that can be rainbow in a given $3$-coloring of the first $n$ integers is at least $0.4$ and at most $0.66656$. We conjecture the lower bound to be tight. This question is also motivated by a famous analogous problem in graph theory due to Erd\H{o}s and S\'os regarding the maximum number of rainbow triangles in any $3$-coloring of $K_n$, which was settled by Balogh et al., Comment: 10 pages, 1 figure

We present two novel six-colorings of the Euclidean plane that avoid monochromatic pairs of points at unit distance in five colors and monochromatic pairs at another specified distance $d$ in the sixth color. Such colorings have previously been known to exist for $0.41 < \sqrt{2} - 1 \le d \le 1 / \sqrt{5} < 0.45$. Our results significantly expand that range to $0.354 \le d \le 0.657$, the first improvement in 30 years. Notably, the constructions underlying this were derived by formalizing colorings suggested by a custom machine learning approach., Comment: An animated version of Figure 2 is available at https://christophspiegel.berlin/hn/fig2.pdf and can be viewed with Acrobat Reader

Neural Networks can be efficiently compressed through pruning, significantly reducing storage and computational demands while maintaining predictive performance. Simple yet effective methods like Iterative Magnitude Pruning (IMP, Han et al., 2015) remove less important parameters and require a costly retraining procedure to recover performance after pruning. However, with the rise of Large Language Models (LLMs), full retraining has become infeasible due to memory and compute constraints. In this study, we challenge the practice of retraining all parameters by demonstrating that updating only a small subset of highly expressive parameters is often sufficient to recover or even improve performance compared to full retraining. Surprisingly, retraining as little as 0.27%-0.35% of the parameters of GPT-architectures achieves comparable performance to One Shot IMP across various sparsity levels. Our approach, Parameter-Efficient Retraining after Pruning (PERP), drastically reduces compute and memory demands, enabling pruning and retraining of up to 30 billion parameter models on a single NVIDIA A100 GPU within minutes. Despite magnitude pruning being considered as unsuited for pruning LLMs, our findings show that PERP positions it as a strong contender against state-of-the-art retraining-free approaches such as Wanda (Sun et al., 2023) and SparseGPT (Frantar & Alistarh, 2023), opening up a promising alternative to avoiding retraining., Comment: 27 pages, 4 figures, 15 tables

We study a generalization of a famous result of Goodman and establish that asymptotically at least a $1/256$ fraction of all triangles needs to be monochromatic in any four-coloring of the edges of a complete graph. We also show that any large enough extremal construction must be based on a blow-up of one of the two $R(3,3,3)$ Ramsey-colorings of $K_{16}$. This result is obtained through an efficient flag algebra formulation by exploiting problem-specific combinatorial symmetries that also allows us to study some related problems., Comment: 45 pages

Neural networks can be significantly compressed by pruning, yielding sparse models with reduced storage and computational demands while preserving predictive performance. Model soups (Wortsman et al., 2022) enhance generalization and out-of-distribution (OOD) performance by averaging the parameters of multiple models into a single one, without increasing inference time. However, achieving both sparsity and parameter averaging is challenging as averaging arbitrary sparse models reduces the overall sparsity due to differing sparse connectivities. This work addresses these challenges by demonstrating that exploring a single retraining phase of Iterative Magnitude Pruning (IMP) with varied hyperparameter configurations such as batch ordering or weight decay yields models suitable for averaging, sharing identical sparse connectivity by design. Averaging these models significantly enhances generalization and OOD performance over their individual counterparts. Building on this, we introduce Sparse Model Soups (SMS), a novel method for merging sparse models by initiating each prune-retrain cycle with the averaged model from the previous phase. SMS preserves sparsity, exploits sparse network benefits, is modular and fully parallelizable, and substantially improves IMP's performance. We further demonstrate that SMS can be adapted to enhance state-of-the-art pruning-during-training approaches., Comment: ICLR24 Camera Ready, 9 pages, 5 pages references, 16 pages appendix

We study an analogue of the Ramsey multiplicity problem for additive structures, in particular establishing the minimum number of monochromatic 3-APs in 3-colorings of $\mathbb{F}_3^n$ as well as obtaining the first non-trivial lower bound for the minimum number of monochromatic 4-APs in 2-colorings of $\mathbb{F}_5^n$. The former parallels results by Cumings et al (2013) in extremal graph theory and the latter improves upon results of Saad and Wolf (2017) The lower bounds are notably obtained by extending the flag algebra calculus of Razborov (2007) to additive structures in vector spaces over finite fields., Comment: 20 pages, 6 figures

We study two related problems concerning the number of homogeneous subsets of given size in graphs that go back to questions of Erd\H{o}s. Most notably, we improve the upper bounds on the Ramsey multiplicity of $K_4$ and $K_5$ and settle the minimum number of independent sets of size $4$ in graphs with clique number at most $4$. Motivated by the elusiveness of the symmetric Ramsey multiplicity problem, we also introduce an off-diagonal variant and obtain tight results when counting monochromatic $K_4$ or $K_5$ in only one of the colors and triangles in the other. The extremal constructions for each problem turn out to be blow-ups of a graph of constant size and were found through search heuristics. They are complemented by lower bounds established using flag algebras, resulting in a fully computer-assisted approach. For some of our theorems we can also derive that the extremal construction is stable in a very strong sense. More broadly, these problems lead us to the study of the region of possible pairs of clique and independent set densities that can be realized as the limit of some sequence of graphs., Comment: 39 pages, 3 figures, published in Foundations of Computational Mathematics (2024), DOI: 10.1007/s10208-024-09675-6

Many existing Neural Network pruning approaches rely on either retraining or inducing a strong bias in order to converge to a sparse solution throughout training. A third paradigm, 'compression-aware' training, aims to obtain state-of-the-art dense models that are robust to a wide range of compression ratios using a single dense training run while also avoiding retraining. We propose a framework centered around a versatile family of norm constraints and the Stochastic Frank-Wolfe (SFW) algorithm that encourage convergence to well-performing solutions while inducing robustness towards convolutional filter pruning and low-rank matrix decomposition. Our method is able to outperform existing compression-aware approaches and, in the case of low-rank matrix decomposition, it also requires significantly less computational resources than approaches based on nuclear-norm regularization. Our findings indicate that dynamically adjusting the learning rate of SFW, as suggested by Pokutta et al. (2020), is crucial for convergence and robustness of SFW-trained models and we establish a theoretical foundation for that practice., Comment: 8 pages, 5 pages references, 14 pages appendix, 8 figures, and 11 tables

Many Neural Network Pruning approaches consist of several iterative training and pruning steps, seemingly losing a significant amount of their performance after pruning and then recovering it in the subsequent retraining phase. Recent works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance of the learning rate schedule during the retraining phase and propose specific heuristics for choosing such a schedule for IMP (Han et al., 2015). We place these findings in the context of the results of Li et al. (2020) regarding the training of models within a fixed training budget and demonstrate that, consequently, the retraining phase can be massively shortened using a simple linear learning rate schedule. Improving on existing retraining approaches, we additionally propose a method to adaptively select the initial value of the linear schedule. Going a step further, we propose similarly imposing a budget on the initial dense training phase and show that the resulting simple and efficient method is capable of outperforming significantly more complex or heavily parameterized state-of-the-art approaches that attempt to sparsify the network during training. These findings not only advance our understanding of the retraining phase, but more broadly question the belief that one should aim to avoid the need for retraining and reduce the negative effects of 'hard' pruning by incorporating the sparsification process into the standard training., Comment: ICLR2023 camera-ready version, 9 pages main text, 34 pages appendix, 2 tables, 3 figures in main text

This paper studies the empirical efficacy and benefits of using projection-free first-order methods in the form of Conditional Gradients, a.k.a. Frank-Wolfe methods, for training Neural Networks with constrained parameters. We draw comparisons both to current state-of-the-art stochastic Gradient Descent methods as well as across different variants of stochastic Conditional Gradients. In particular, we show the general feasibility of training Neural Networks whose parameters are constrained by a convex feasible region using Frank-Wolfe algorithms and compare different stochastic variants. We then show that, by choosing an appropriate region, one can achieve performance exceeding that of unconstrained stochastic Gradient Descent and matching state-of-the-art results relying on $L^2$-regularization. Lastly, we also demonstrate that, besides impacting performance, the particular choice of constraints can have a drastic impact on the learned representations., Comment: fixed coding error in figure 1 and extended abstract; 13 pages, Abstract 11 pages, 9 figures, 6 tables

The complexity in large-scale optimization can lie in both handling the objective function and handling the constraint set. In this respect, stochastic Frank-Wolfe algorithms occupy a unique position as they alleviate both computational burdens, by querying only approximate first-order information from the objective and by maintaining feasibility of the iterates without using projections. In this paper, we improve the quality of their first-order information by blending in adaptive gradients. We derive convergence rates and demonstrate the computational advantage of our method over the state-of-the-art stochastic Frank-Wolfe algorithms on both convex and nonconvex objectives. The experiments further show that our method can improve the performance of adaptive gradient algorithms for constrained optimization., Comment: 28 pages, 10 figures

We consider the problem of determining the maximum order of an induced vertex-disjoint union of cliques in a graph. More specifically, given some family of graphs $\mathcal{G}$ of equal order, we are interested in the parameter $a(\mathcal{G}) = \min_{G \in \mathcal{G}} \max \{ |U| : U \subseteq V, G[U] \text{ is a vertex-disjoint union of cliques} \}$. We determine the value of this parameter precisely when $\mathcal{G}$ is the family of comparability graphs of $n$-element posets with acyclic cover graph. In particular, we show that $a(\mathcal{G}) = (n+o(n))/\log_2 (n)$ in this class., Comment: 15 pages, 6 figures

A set of integers $S \subset \mathbb{N}$ is an $\alpha$-strong Sidon set if the pairwise sums of its elements are far apart by a certain measure depending on $\alpha$, more specifically if $| (x+w) - (y+z) | \geq \max \{ x^{\alpha},y^{\alpha},z^{\alpha},w^\alpha \}$ for every $x,y,z,w \in S$ satisfying $\max \{x,w\} \neq \max \{y,z\}$. We obtain a new lower bound for the growth of $\alpha$-strong infinite Sidon sets when $0 \leq \alpha < 1$. We also further extend that notion in a natural way by obtaining the first non-trivial bound for $\alpha$-strong infinite $B_h$ sets. In both cases, we study the implications of these bounds for the density of, respectively, the largest Sidon or $B_h$ set contained in a random infinite subset of $\mathbb{N}$. Our theorems improve on previous results by Kohayakawa, Lee, Moreira and R\"odl., Comment: 15 pages, fixed a calculation in the exponent on page 9

Let $k\geq 2$ be a positive integer. We study concentration results for the ordered representation functions $r^{\leq}_k(A,n) = \# \big\{ (a_1 \leq \dots \leq a_k) \in A^k : a_1+\dots+a_k = n \big\}$ and $r^{<}_k(A,n) = \# \big\{ (a_1 < \dots < a_k) \in A^k : a_1+\dots+a_k = n \big\}$ for any infinite set of non-negative integers $A$. Our main theorem is an Erd\H{o}s--Fuchs-type result for both functions: for any $c > 0$ and $\star \in \{\leq,<\}$ we show that $$\sum_{j = 0}^{n} \Big( r^{\star}_k(A,j) - c \Big) = o\big(n^{1/4} \log^{-1/2}n \big)$$ is not possible. We also show that the mean squared error $$E^\star_{k,c}(A,n)=\frac{1}{n} \sum_{j = 0}^{n} \Big( r^{\star}_k(A,j) - c \Big)^2$$ satisfies $\limsup_{n \to \infty} E^\star_{k,c}(A,n)>0$. These results extend two theorems for the non-ordered representation function proved by Erd\H{o}s and Fuchs in the case of $k=2$ (J. of the London Math. Society 1956)., Comment: 15 pages

We study the positional game where two players, Maker and Breaker, alternately select respectively $1$ and $b$ previously unclaimed edges of $K_n$. Maker wins if she succeeds in claiming all edges of some odd cycle in $K_n$ and Breaker wins otherwise. Improving on a result of Bednarska and Pikhurko, we show that Maker wins the odd cycle game if $b \leq ((4 - \sqrt{6})/5 + o(1)) n$. We furthermore introduce "connected rules" and study the odd cycle game under them, both in the Maker-Breaker as well as in the Client-Waiter variant., Comment: 22 pages, 3 figures

The Hales-Jewett Theorem states that any $r$-colouring of $[m]^n$ contains a monochromatic combinatorial line if $n$ is large enough. Shelah's proof of the theorem implies that for $m = 3$ there always exists a monochromatic combinatorial lines whose set of active coordinates is the union of at most $r$ intervals. Conlon and Kam\v{c}ev proved the existence of colourings for which it cannot be fewer than $r$ intervals if $r$ is odd. For $r = 2$ however, Leader and R\"aty showed that one can always find a monochromatic combinatorial line whose active coordinate set is a single interval. In this paper, we extend the result of Leader and R\"aty to the case of all even $r$, showing that one can always find a monochromatic combinatorial line in $[3]^n$ whose set of active coordinate is the union of at most $r-1$ intervals., Comment: 16 pages, 5 figures

A conjecture of Freiman gives an exact formula for the largest volume of a finite set $A$ of integers with given cardinality $k = |A|$ and doubling $T = |2A|$. The formula is known to hold when $T \le 3k-4$, for some small range over $3k-4$ and for families of structured sets called chains. In this paper we extend the formula to sets of every dimension and prove it for sets composed of three segments, giving structural results for the extremal case. A weaker extension to sets composed of a bounded number of segments is also discussed., Comment: 16 pages

Freiman's 2.4-Theorem states that any set $A \subset \mathbb{Z}_p$ satisfying $|2A| \leq 2.4|A| - 3 $ and $|A| < p/35$ can be covered by an arithmetic progression of length at most $|2A| - |A| + 1$. A more general result of Green and Ruzsa implies that this covering property holds for any set satisfying $|2A| \leq 3|A| - 4$ as long as the rather strong density requirement $|A| < p/10^{215}$ is satisfied. We present a version of this statement that allows for sets satisfying $|2A| \leq 2.48|A| - 7$ with the more modest density requirement of $|A| < p/10^{10}$., Comment: 13 pages, 1 figure

We prove that for pairwise co-prime numbers $k_1,\dots,k_d \geq 2$ there does not exist any infinite set of positive integers $A$ such that the representation function $r_A (n) = \{ (a_1, \dots, a_d) \in A^d : k_1 a_1 + \dots + k_d a_d = n \}$ becomes constant for $n$ large enough. This result is a particular case of our main theorem, which poses a further step towards answering a question of S\'ark\"ozy and S\'os and widely extends a previous result of Cilleruelo and Ru\'e for bivariate linear forms., Comment: Added clarifications regarding the particular notion of limit used in the first part of the paper. 11 pages

The concept of biased Maker-Breaker games, introduced by Chv\'atal and Erd{\H o}s, is a central topic in the field of positional games, with deep connections to the theory of random structures. For any given hypergraph ${\cal H}$ the main questions is to determine the smallest bias $q({\cal H})$ that allows Breaker to force that Maker ends up with an independent set of ${\cal H}$. Here we prove matching general winning criteria for Maker and Breaker when the game hypergraph satisfies a couple of natural `container-type' regularity conditions about the degree of subsets of its vertices. This will enable us to derive a hypergraph generalization of the $H$-building games, studied for graphs by Bednarska and {\L}uczak. Furthermore, we investigate the biased version of generalizations of the van der Waerden games introduced by Beck. We refer to these generalizations as Rado games and determine their threshold bias up to constant factors by applying our general criteria. We find it quite remarkable that a purely game theoretic deterministic approach provides the right order of magnitude for such a wide variety of hypergraphs, when the generalizations to hypergraphs in the analogous setup of sparse random discrete structures are usually quite challenging., Comment: 26 pages

We study the thresholds for the property of containing a solution to a linear homogeneous system in random sets. We expand a previous sparse Sz\'emeredi-type result of Schacht to the broadest class of matrices possible. We also provide a shorter proof of a sparse Rado result of Friedgut, R\"odl, Ruci\'nski and Schacht based on a hypergraph container approach due to Nenadov and Steger. Lastly we further extend these results to include some solutions with repeated entries using a notion of non-trivial solutions due to R\'uzsa as well as Ru\'e et al., Comment: 14 pages

We present a unified framework to study threshold functions for the existence of solutions to linear systems of equations in random sets which includes arithmetic progressions, sum-free sets, $B_{h}[g]$-sets and Hilbert cubes. In particular, we show that there exists a threshold function for the property "$\mathcal{A}$ contains a non-trivial solution of $M\cdot\textbf{x}=\textbf{0}$", where $\mathcal{A}$ is a random set and each of its elements is chosen independently with the same probability from the interval of integers $\{1,\dots,n\}$. Our study contains a formal definition of trivial solutions for any combinatorial structure, extending a previous definition by Ruzsa when dealing with a single equation. Furthermore, we study the behaviour of the distribution of the number of non-trivial solutions at the threshold scale. We show that it converges to a Poisson distribution whose parameter depends on the volumes of certain convex polytopes arising from the linear system under study as well as the symmetry inherent in the structures, which we formally define and characterize., Comment: New version with minor corrections and changes in notation. 24 Pages