Search

A remarkable characteristic of overparameterized deep neural networks (DNNs) is that their accuracy does not degrade when the network's width is increased. Recent evidence suggests that developing compressible representations is key for adjusting the complexity of large networks to the learning task at hand. However, these compressible representations are poorly understood. A promising strand of research inspired from biology is understanding representations at the unit level as it offers a more granular and intuitive interpretation of the neural mechanisms. In order to better understand what facilitates increases in width without decreases in accuracy, we ask: Are there mechanisms at the unit level by which networks control their effective complexity as their width is increased? If so, how do these depend on the architecture, dataset, and training parameters? We identify two distinct types of "frivolous" units that proliferate when the network's width is increased: prunable units which can be dropped out of the network without significant change to the output and redundant units whose activities can be expressed as a linear combination of others. These units imply complexity constraints as the function the network represents could be expressed by a network without them. We also identify how the development of these units can be influenced by architecture and a number of training factors. Together, these results help to explain why the accuracy of DNNs does not degrade when width is increased and highlight the importance of frivolous units toward understanding implicit regularization in DNNs.

This PhD project deals with two topics set within the broader context of visual and cognitive neuroscience. First of all, we investigate whether certain properties of the primate visual system might be found in rats. The second topic deals with how representations of visual input are not fixed but are influenced by recent sensory input or even cognitive processes. In a first set of studies in rats, we investigate whether and to what extent some general principles of the primate visual system (e.g. object representations) are also present in that of the rat. This includes comparing complex stimulus representations based on action potential activity measured at different levels of the visual system. In addition, we investigate how previous sensory input affects neural responses. A robust manifestation of the influence of recent input is neural adaptation or repetition suppression, which refers to a reduced neural activity when stimuli are repeated. Additionally neural responses could also be enhanced, for example when a stimulus is rare. These mechanisms have an important role in the predictive coding theory, which postulates that cortical responses signal violations of prior expectations. For the final part of the project, we moved to macaques for experiments that could not be performed in less cognitively developed animals. Here we investigate claims that perceptual expectations can modulate the strength of adaptation of action potential activity in visual cortex. status: published