Semantic-specific multimodal relation learning for sentiment analysis.

Multimodal sentiment analysis (MSA) seeks to understand human affection by leveraging signals from multiple modalities. A core challenge in MSA is the effective extraction of sentimental relations between these signals, as this can enhance a model's consistency and accuracy. Existing studies typically use multimodal matching tasks to learn all semantic relations between modalities and then use downstream task to obtain the specific semantics from the multimodal representation. However, there are multiple semantics between modalities, such as action semantics, scene semantics and sentiment semantics. Relying solely on specific tasks to filter these semantics often results in a surplus of redundant information in the multimodal representation, potentially degrading MSA accuracy. In addition, the unimodal semantic expression is also important. In this paper, we propose a semantic-specific multimodal relation learning method to correlate modalities with specific semantics. Specifically, with smaller computational resources, we enhance unimodal sentimental semantic expression while diminishing non-sentimental semantic information in the multimodal representation. We conducted experiments on multimodal sentiment analysis datasets, CMU-MOSI, CMU-MOSEI and CH-SIMS. The results show that our method outperforms the current state-of-the-art. Notably, on the Acc2 evaluation metric, our approach exhibits an average accuracy improvement of 0.75 compared to the best baseline. [ABSTRACT FROM AUTHOR]