1. Geometry of fibers of the multiplication map of deep linear neural networks
- Author
-
Lehalleur, SImon Pepin and Rimányi, Richárd
- Subjects
Mathematics - Algebraic Geometry ,Mathematics - Representation Theory ,Statistics - Machine Learning ,16G20 05E14 62F15 62R01 - Abstract
We study the geometry of the algebraic set of tuples of composable matrices which multiply to a fixed matrix, using tools from the theory of quiver representations. In particular, we determine its codimension $C$ and the number $\theta$ of its top-dimensional irreducible components. Our solution is presented in three forms: a Poincar\'e series in equivariant cohomology, a quadratic integer program, and an explicit formula. In the course of the proof, we establish a surprising property: $C$ and $\theta$ are invariant under arbitrary permutations of the dimension vector. We also show that the real log-canonical threshold of the function taking a tuple to the square Frobenius norm of its product is $C/2$. These results are motivated by the study of deep linear neural networks in machine learning and Bayesian statistics (singular learning theory) and show that deep linear networks are in a certain sense ``mildly singular"., Comment: 28 pages, 2 figures. Comments welcome!
- Published
- 2024