1. Provable Tempered Overfitting of Minimal Nets and Typical Nets
- Author
-
Harel, Itamar, Hoza, William M., Vardi, Gal, Evron, Itay, Srebro, Nathan, and Soudry, Daniel
- Subjects
Computer Science - Machine Learning ,Statistics - Machine Learning - Abstract
We study the overfitting behavior of fully connected deep Neural Networks (NNs) with binary weights fitted to perfectly classify a noisy training set. We consider interpolation using both the smallest NN (having the minimal number of weights) and a random interpolating NN. For both learning rules, we prove overfitting is tempered. Our analysis rests on a new bound on the size of a threshold circuit consistent with a partial function. To the best of our knowledge, ours are the first theoretical results on benign or tempered overfitting that: (1) apply to deep NNs, and (2) do not require a very high or very low input dimension., Comment: 60 pages, 4 figures
- Published
- 2024