1. Multimodal Embeddings From Language Models for Emotion Recognition in the Wild
- Author
-
Shao-Yen Tseng, Panayiotis G. Georgiou, and Shrikanth S. Narayanan
- Subjects
Context model ,Computer science ,business.industry ,Applied Mathematics ,Feature extraction ,020206 networking & telecommunications ,02 engineering and technology ,computer.software_genre ,Paralanguage ,Signal Processing ,0202 electrical engineering, electronic engineering, information engineering ,Task analysis ,Word usage ,Language model ,Artificial intelligence ,Electrical and Electronic Engineering ,business ,computer ,Natural language processing ,Word (computer architecture) ,Spoken language - Abstract
Word embeddings such as ELMo and BERT have been shown to model word usage in language with greater efficacy through contextualized learning on large-scale language corpora, resulting in significant performance improvement across many natural language processing tasks. In this work we integrate acoustic information into contextualized lexical embeddings through the addition of a parallel stream to the bidirectional language model. This multimodal language model is trained on spoken language data that includes both text and audio modalities. We show that embeddings extracted from this model integrate paralinguistic cues into word meanings and can provide vital affective information by applying these multimodal embeddings to the task of speaker emotion recognition.
- Published
- 2021