Search

We explore the universality of neural encodings in convolutional neural networks trained on image classification tasks. We develop a procedure to directly compare the learned weights rather than their representations. It is based on a factorization of spatial and channel dimensions and measures the similarity of aligned weight covariances. We show that, for a range of layers of VGG-type networks, the learned eigenvectors appear to be universal across different natural image datasets. Our results suggest the existence of a universal neural encoding for natural images. They explain, at a more fundamental level, the success of transfer learning. Our work shows that, instead of aiming at maximizing the performance of neural networks, one can alternatively attempt to maximize the universality of the learned encoding, in order to build a principled foundation model., Comment: Appeared at the ICLR 2024 Workshop on Representational Alignment (Re-Align), 13 pages, 5 figures

How does the human visual cortex encode sensory information? To address this question, we explore the covariance structure of neural representations. We perform a cross-decomposition analysis of fMRI responses to natural images in multiple individuals from the Natural Scenes Dataset and find that neural representations systematically exhibit a power-law covariance spectrum over four orders of magnitude in ranks. This scale-free structure is found in multiple regions along the visual hierarchy, pointing to the existence of a generic encoding strategy in visual cortex. We also show that, up to a rotation, a large ensemble of principal axes of these population codes are shared across subjects, showing the existence of a universal high-dimensional representation. This suggests a high level of convergence in how the human brain learns to represent natural scenes despite individual differences in neuroanatomy and experience. We further demonstrate that a spectral approach is critical for characterizing population codes in their full extent, and in doing so, we reveal a vast space of uncharted dimensions that have been out of reach for conventional variance-weighted methods. A global view of neural representations thus requires embracing their high-dimensional nature and understanding them statistically rather than through visual or semantic interpretation of individual dimensions., Comment: 22 pages, 4 main figures, 7 supplementary figures

As weak lensing surveys go deeper, there is an increasing need for reliable characterization of non-Gaussian structures at small angular scales. Here we present the first cosmological constraints with weak lensing scattering transform, a statistical estimator that combines efficiency, robustness, and interpretability. With the Hyper Suprime-Cam survey (HSC) year 1 data, we obtain $\Omega_\text{m}=0.29_{-0.03}^{+0.04}$, $S_8\equiv \sigma_8(\Omega_\text{m}/0.3)^{0.5}=0.83\pm0.02$, and intrinsic alignment strength $A_\text{IA}=1.0\pm0.4$ through simulation-based forward modeling. Our constraints are consistent with those derived from Planck. The error bar of $\Omega_\text{m}$ is 2 times tighter than that obtained from the power spectrum when the same scale range is used. This constraining power is on par with that of convolutional neural networks, suggesting that further investment in spatial information extraction may not yield substantial benefits. We also point out an internal tension of $S_8$ estimates linked to a redshift bin around z ~ 1 in the HSC data. We found that discarding that bin leads to a consistent decrease of $S_8$ from 0.83 to 0.79, for all statistical estimators. We argue that photometric redshift estimation is now the main limitation in the estimation of $S_8$ using HSC. This limitation is likely to affect other ground-based weak lensing surveys reaching redshifts greater than one. Alternative redshift estimation techniques, like clustering redshifts, may help alleviate this limitation., Comment: 8 pages, 6 figures

Physicists routinely need probabilistic models for a number of tasks such as parameter inference or the generation of new realizations of a field. Establishing such models for highly non-Gaussian fields is a challenge, especially when the number of samples is limited. In this paper, we introduce scattering spectra models for stationary fields and we show that they provide accurate and robust statistical descriptions of a wide range of fields encountered in physics. These models are based on covariances of scattering coefficients, i.e. wavelet decomposition of a field coupled with a point-wise modulus. After introducing useful dimension reductions taking advantage of the regularity of a field under rotation and scaling, we validate these models on various multi-scale physical fields and demonstrate that they reproduce standard statistics, including spatial moments up to 4th order. These scattering spectra provide us with a low-dimensional structured representation that captures key properties encountered in a wide range of physical fields. These generic models can be used for data exploration, classification, parameter inference, symmetry detection, and component separation., Comment: 11 pages, 6 figures, plus appendices, updated to published version

A central question in deep learning is to understand the functions learned by deep networks. What is their approximation class? Do the learned weights and representations depend on initialization? Previous empirical work has evidenced that kernels defined by network activations are similar across initializations. For shallow networks, this has been theoretically studied with random feature models, but an extension to deep networks has remained elusive. Here, we provide a deep extension of such random feature models, which we call the rainbow model. We prove that rainbow networks define deterministic (hierarchical) kernels in the infinite-width limit. The resulting functions thus belong to a data-dependent RKHS which does not depend on the weight randomness. We also verify numerically our modeling assumptions on deep CNNs trained on image classification tasks, and show that the trained networks approximately satisfy the rainbow hypothesis. In particular, rainbow networks sampled from the corresponding random feature model achieve similar performance as the trained networks. Our results highlight the central role played by the covariances of network weights at each layer, which are observed to be low-rank as a result of feature learning., Comment: 59 pages, 10 figures. To appear at JMLR