Search

Multimodal data provide complementary information of a natural phenomenon by integrating data from various domains with very different statistical properties. Capturing the intramodality and cross-modality information of multimodal data is the essential capability of multimodal learning methods. The geometry-aware data analysis approaches provide these capabilities by implicitly representing data in various modalities based on their geometric underlying structures. Also, in many applications, data are explicitly defined on an intrinsic geometric structure. Generalizing deep learning methods to the non-Euclidean domains is an emerging research field, which has recently been investigated in many studies. Most of those popular methods are developed for unimodal data. In this article, a multimodal graph wavelet convolutional network (M-GWCN) is proposed as an end-to-end network. M-GWCN simultaneously finds intramodality representation by applying the multiscale graph wavelet transform to provide helpful localization properties in the graph domain of each modality and cross-modality representation by learning permutations that encode correlations among various modalities. M-GWCN is not limited to either the homogeneous modalities with the same number of data or any prior knowledge indicating correspondences between modalities. Several semisupervised node classification experiments have been conducted on three popular unimodal explicit graph-based datasets and five multimodal implicit ones. The experimental results indicate the superiority and effectiveness of the proposed methods compared with both spectral graph domain convolutional neural networks and state-of-the-art multimodal methods.

International audience; Multimodal learning, also known as multi-view learning, data integration, or data fusion, is an emerging field in signal processing, machine learning, and pattern recognition domains. It aims at building models, learned from several related and complementary modalities, in order to increase the generalization performances of a predictive learning model. Multimodal manifold learning extends spectral or diffusion geometry-aware data analysis to multiple modalities. This can be performed through the definition of undirected graph Laplacian matrices in different modalities. However, finding common eigenbasis of multiple Laplacians is not always a relevant solution for multimodal manifold learning problems. As a matter of fact, the Laplacians of all modalities are not simultaneously diagonalizable in many real-world problems due to the major differences between the different modalities. In this paper, we propose a multimodal manifold learning approach based on intrinsic local tangent spaces of underlying data manifolds in order to discover the local geometrical structure around matching and mismatching samples in different modalities in sparse diagonalization problems. This approach searches for approximate common eigenbasis of Laplacian matrices by expanding the signal of limited existing information about matching and mismatching samples of different modalities to their on-manifold neighbors. Experiments on synthetic and real-world datasets in supervised, unsupervised, and semi-supervised problems demonstrate the superiority of our proposed approach over existing state-of-the-art related methods.

Multimodal data provide complementary information of a natural phenomenon by integrating data from various domains with very different statistical properties. Capturing the intra-modality and cross-modality information of multimodal data is the essential capability of multimodal learning methods. The geometry-aware data analysis approaches provide these capabilities by implicitly representing data in various modalities based on their geometric underlying structures. Also, in many applications, data are explicitly defined on an intrinsic geometric structure. Generalizing deep learning methods to the non-Euclidean domains is an emerging research field, which has recently been investigated in many studies. Most of those popular methods are developed for unimodal data. In this paper, a multimodal multi-scaled graph wavelet convolutional network (M-GWCN) is proposed as an end-to-end network. M-GWCN simultaneously finds intra-modality representation by applying the multiscale graph wavelet transform to provide helpful localization properties in the graph domain of each modality, and cross-modality representation by learning permutations that encode correlations among various modalities. M-GWCN is not limited to either the homogeneous modalities with the same number of data, or any prior knowledge indicating correspondences between modalities. Several semi-supervised node classification experiments have been conducted on three popular unimodal explicit graph-based datasets and five multimodal implicit ones. The experimental results indicate the superiority and effectiveness of the proposed methods compared with both spectral graph domain convolutional neural networks and state-of-the-art multimodal methods., Comment: 13 pages