Back to Search Start Over

Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game.

Authors :
Hoang, Nguyen Le
Taniguchi, Tadahiro
Hagiwara, Yoshinobu
Taniguchi, Akira
Filos, Angelos
Dessi, Roberto
Source :
Frontiers in Robotics & AI; 2024, p1-18, 18p
Publication Year :
2024

Abstract

Deep generative models (DGM) are increasingly employed in emergent communication systems. However, their application in multimodal data contexts is limited. This study proposes a novel model that combines multimodal DGM with the Metropolis-Hastings (MH) naming game, enabling two agents to focus jointly on a shared subject and develop common vocabularies. The model proves that it can handle multimodal data, even in cases of missing modalities. Integrating the MH naming game with multimodal variational autoencoders (VAE) allows agents to form perceptual categories and exchange signs within multimodal contexts. Moreover, fine-tuning the weight ratio to favor a modality that the model could learn and categorize more readily improved communication. Our evaluation of three multimodal approaches -- mixture-of-experts (MoE), product-of-experts (PoE), and mixture-of-product- of-experts (MoPoE)-suggests an impact on the creation of latent spaces, the internal representations of agents. Our results from experiments with the MNIST + SVHN and Multimodal165 datasets indicate that combining the Gaussian mixture model (GMM), PoE multimodal VAE, and MH naming game substantially improved information sharing, knowledge formation, and data reconstruction. [ABSTRACT FROM AUTHOR]

Details

Language :
English
ISSN :
22969144
Database :
Complementary Index
Journal :
Frontiers in Robotics & AI
Publication Type :
Academic Journal
Accession number :
175537866
Full Text :
https://doi.org/10.3389/frobt.2023.1290604