Search

The emergence of similar representations between independently trained neural models has sparked significant interest in the representation learning community, leading to the development of various methods to obtain communication between latent spaces. "Latent space communication" can be achieved in two ways: i) by independently mapping the original spaces to a shared or relative one; ii) by directly estimating a transformation from a source latent space to a target one. In this work, we combine the two into a novel method to obtain latent space translation through the relative space. By formalizing the invertibility of angle-preserving relative representations and assuming the scale invariance of decoder modules in neural models, we can effectively use the relative space as an intermediary, independently projecting onto and from other semantically similar spaces. Extensive experiments over various architectures and datasets validate our scale invariance assumption and demonstrate the high accuracy of our method in latent space translation. We also apply our method to zero-shot stitching between arbitrary pre-trained text and image encoders and their classifiers, even across modalities. Our method has significant potential for facilitating the reuse of models in a practical manner via compositionality., Comment: arXiv admin note: text overlap with arXiv:2311.00664, arXiv:2406.11014

As NNs permeate various scientific and industrial domains, understanding the universality and reusability of their representations becomes crucial. At their core, these networks create intermediate neural representations, indicated as latent spaces, of the input data and subsequently leverage them to perform specific downstream tasks. This dissertation focuses on the universality and reusability of neural representations. Do the latent representations crafted by a NN remain exclusive to a particular trained instance, or can they generalize across models, adapting to factors such as randomness during training, model architecture, or even data domain? This adaptive quality introduces the notion of Latent Communication -- a phenomenon that describes when representations can be unified or reused across neural spaces. A salient observation from our research is the emergence of similarities in latent representations, even when these originate from distinct or seemingly unrelated NNs. By exploiting a partial correspondence between the two data distributions that establishes a semantic link, we found that these representations can either be projected into a universal representation, coined as Relative Representation, or be directly translated from one space to another. Latent Communication allows for a bridge between independently trained NN, irrespective of their training regimen, architecture, or the data modality they were trained on -- as long as the data semantic content stays the same (e.g., images and their captions). This holds true for both generation, classification and retrieval downstream tasks; in supervised, weakly supervised, and unsupervised settings; and spans various data modalities including images, text, audio, and graphs -- showcasing the universality of the Latent Communication phenomenon. [...], Comment: Doctoral Thesis: https://iris.uniroma1.it/handle/11573/1711827

Visual Reinforcement Learning is a popular and powerful framework that takes full advantage of the Deep Learning breakthrough. However, it is also known that variations in the input (e.g., different colors of the panorama due to the season of the year) or the task (e.g., changing the speed limit for a car to respect) could require complete retraining of the agents. In this work, we leverage recent developments in unifying latent representations to demonstrate that it is possible to combine the components of an agent, rather than retrain it from scratch. We build upon the recent relative representations framework and adapt it for Visual RL. This allows us to create completely new agents capable of handling environment-task combinations never seen during training. Our work paves the road toward a more accessible and flexible use of reinforcement learning., Comment: 13 pages, 10 figures, 4 tables

We review the role of the spectral condition as a characteristic feature unifying Minkowski, de Sitter and anti de Sitter Quantum Field Theory. In this context, we highlight the role of an important class of plane waves which are either de Sitter or anti de Sitter covariant and are compatible with the relevant analyticity domains linked to the spectral condition(s). We show again how to expand the two-point functions and propagators in terms of them and some of the advantages of doing so rather than using special coordinate systems and separated variables., Comment: THE FRIEDMANN COSMOLOGY: A CENTURY LATER. Invited paper

We discuss general one and two-loops banana diagrams with arbitrary masses on the de Sitter spacetime by using direct methods of dS quantum field theory in the dimensional regularization approach. In the one-loop case we also compute the effective potential for an $O(N)$ model in $d=4$ dimension as an explicit function of the cosmological constant $\Lambda$, both exactly and perturbatively up to order $\Lambda$. For the two-loop case we show that the calculation is made easy thanks to a remarkable Kallen-Lehmann formula that has been in the literature for a while. We discuss the divergent cases at d=3 using a contiguity formula for generalized hypergeometric functions and we extract the dominant term at d=4 proving a general formula to deal with a divergent hypergeometric series.

We discuss general one and two-loop banana diagrams and one-loop diagrams with external lines with arbitrary masses on the anti de Sitter spacetime by using methods of AdS quantum field theory in the dimensional regularization approach. The banana diagrams explicitly computed in this paper are indeed the necessary ingredients for the evaluation of the two-loop effective potential of the Standard Model and can be used to extend the flat space results in in presence of a negative cosmological constant. In the one-loop case we also compute the effective potential for an $O(N)$ model in $d=4$ dimension as an explicit function of the cosmological constant $\Lambda$, both exactly and perturbatively up to order $\Lambda$. In the two-loop case we show the explicit calculation is possible thanks to a remarkable discrete K\"all\'en-Lehmann formula which we found and proved sometimes ago and whose domain of applicability we extend in the present paper., Comment: New introduction. Extended appendix A and new appendix B extending the Kallen-Lehmann formula generalizing our original formula and other subsequent results. Extended Section 5 including a general formula for the bubble in any background geometry. New section 6 with some comments about the relation among bulk diagrams and Witten diagrams. New section 8 including the tadpole and the sunset

Models trained on semantically related datasets and tasks exhibit comparable inter-sample relations within their latent spaces. We investigate in this study the aggregation of such latent spaces to create a unified space encompassing the combined information. To this end, we introduce Relative Latent Space Aggregation, a two-step approach that first renders the spaces comparable using relative representations, and then aggregates them via a simple mean. We carefully divide a classification problem into a series of learning tasks under three different settings: sharing samples, classes, or neither. We then train a model on each task and aggregate the resulting latent spaces. We compare the aggregated space with that derived from an end-to-end model trained over all tasks and show that the two spaces are similar. We then observe that the aggregated space is better suited for classification, and empirically demonstrate that it is due to the unique imprints left by task-specific embedders within the representations. We finally test our framework in scenarios where no shared region exists and show that it can still be used to merge the spaces, albeit with diminished benefits over naive merging., Comment: To appear in the NeurReps workshop @ NeurIPS 2023

While different neural models often exhibit latent spaces that are alike when exposed to semantically related data, this intrinsic similarity is not always immediately discernible. Towards a better understanding of this phenomenon, our work shows how representations learned from these neural modules can be translated between different pre-trained networks via simpler transformations than previously thought. An advantage of this approach is the ability to estimate these transformations using standard, well-understood algebraic procedures that have closed-form solutions. Our method directly estimates a transformation between two given latent spaces, thereby enabling effective stitching of encoders and decoders without additional training. We extensively validate the adaptability of this translation procedure in different experimental settings: across various trainings, domains, architectures (e.g., ResNet, CNN, ViT), and in multiple downstream tasks (classification, reconstruction). Notably, we show how it is possible to zero-shot stitch text encoders and vision decoders, or vice-versa, yielding surprisingly good classification performance in this multimodal setting., Comment: Accepted at NeurIPS 2023. 21 pages, 13 figures, 8 tables

It has been observed that representations learned by distinct neural networks conceal structural similarities when the models are trained under similar inductive biases. From a geometric perspective, identifying the classes of transformations and the related invariances that connect these representations is fundamental to unlocking applications, such as merging, stitching, and reusing different neural modules. However, estimating task-specific transformations a priori can be challenging and expensive due to several factors (e.g., weights initialization, training hyperparameters, or data modality). To this end, we introduce a versatile method to directly incorporate a set of invariances into the representations, constructing a product space of invariant components on top of the latent representations without requiring prior knowledge about the optimal invariance to infuse. We validate our solution on classification and reconstruction tasks, observing consistent latent similarity and downstream performance improvements in a zero-shot stitching setting. The experimental analysis comprises three modalities (vision, text, and graphs), twelve pretrained foundational models, nine benchmarks, and several architectures trained from scratch., Comment: 41 pages, 14 figures and 31 tables