Topological properties of basins of attraction of width bounded autoencoders.

In [A. Radhakrishnan, M. Belkin and C. Uhler, Overparameterized neural networks implement associative memory, Proc. Natl. Acad. Sci. USA 117(44) (2020) 27162–27170], the authors empirically show that autoencoders trained with standard SGD methods form basins of attraction around their training data. We consider network functions of width not exceeding the input dimension and prove that in this situation, such basins of attraction are bounded and their complement cannot have bounded components. Our conditions in these results are met in several experiments reported in [A. Radhakrishnan, M. Belkin and C. Uhler, Overparameterized neural networks implement associative memory, Proc. Natl. Acad. Sci. USA 117(44) (2020) 27162–27170] and we thus address a question posed therein. We also show that under some more restrictive conditions, the basins of attraction are path-connected. The necessity of the conditions in our results is demonstrated by means of examples. [ABSTRACT FROM AUTHOR]