Back to Search
Start Over
Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition
- Source :
- Affective Computing and Intelligent Interaction (ACII), Sep 2023, Cambridge (MA), United States
- Publication Year :
- 2023
-
Abstract
- Decades of research indicate that emotion recognition is more effective when drawing information from multiple modalities. But what if some modalities are sometimes missing? To address this problem, we propose a novel Transformer-based architecture for recognizing valence and arousal in a time-continuous manner even with missing input modalities. We use a coupling of cross-attention and self-attention mechanisms to emphasize relationships between modalities during time and enhance the learning process on weak salient inputs. Experimental results on the Ulm-TSST dataset show that our model exhibits an improvement of the concordance correlation coefficient evaluation of 37% when predicting arousal values and 30% when predicting valence values, compared to a late-fusion baseline approach.
Details
- Database :
- arXiv
- Journal :
- Affective Computing and Intelligent Interaction (ACII), Sep 2023, Cambridge (MA), United States
- Publication Type :
- Report
- Accession number :
- edsarx.2311.10119
- Document Type :
- Working Paper