Back to Search
Start Over
End-to-end listening agent for audiovisual emotional and naturalistic interactions
- Source :
- Journal of Science and Technology of the Arts; Vol 10 No 2 (2018): eNTERFACE 2017; 49-61, Journal of Science and Technology of the Arts; v. 10 n. 2 (2018): eNTERFACE 2017; 49-61, Repositório Científico de Acesso Aberto de Portugal, Repositório Científico de Acesso Aberto de Portugal (RCAAP), instacron:RCAAP, Journal of Science and Technology of the Arts; Vol 10 No 2 (2018): Volume 10-Number 2, 2018 (Special Issue); 2-49-61, Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos), Agência para a Sociedade do Conhecimento (UMIC)-FCT-Sociedade da Informação, Journal of Science and Technology of the Arts, 10(2), 49-61. Catholic University of Portugal, Journal of Science and Technology of the Arts, 10(2), 49-61. Portuguese Catholic University, Journal of Science and Technology of the Arts, Vol 10, Iss 2 (2018)
- Publication Year :
- 2018
- Publisher :
- Portuguese Catholic University, 2018.
-
Abstract
- In this work, we established the foundations of a framework with the goal to build an end-to-end naturalistic expressive listening agent. The project was split into modules for recognition of the user’s paralinguistic and nonverbal expressions, prediction of the agent’s reactions, synthesis of the agent’s expressions and data recordings of nonverbal conversation expressions. First, a multimodal multitask deep learning-based emotion classification system was built along with a rule-based visual expression detection system. Then several sequence prediction systems for nonverbal expressions were implemented and compared. Also, an audiovisual concatenation-based synthesis system was implemented. Finally, a naturalistic, dyadic emotional conversation database was collected. We report here the work made for each of these modules and our planned future improvements.<br />Journal of Science and Technology of the Arts, v. 10 n. 2 (2018): eNTERFACE 2017
- Subjects :
- Head movement
Multimodal synthesis
lcsh:Fine Arts
Computer science
02 engineering and technology
Nonverbal Expression Detection
Smile
Nonverbal Expression Synthesis
Listening agent
Human–computer interaction
0202 electrical engineering, electronic engineering, information engineering
media_common
Sequence-to-Sequence Prediction Systems
Eyebrow movement
Eyebrow Movement
06 humanities and the arts
Non-verbal expression synthesis
Computer Science Applications
0602 languages and literature
020201 artificial intelligence & image processing
Visual Arts and Performing Arts
Nonverbal expression detection
Speech emotion recognition
media_common.quotation_subject
Emotion classification
Head Movement
Concatenation
Nonverbal expression synthesis
Conservation
Paralanguage
Laughter
Nonverbal communication
ComputerApplications_MISCELLANEOUS
Active listening
Conversation
Dyadic conversation database
060201 languages & linguistics
Speech Emotion Recognition
Emotion database
lcsh:NX1-820
business.industry
Sequence-to-sequence prediction systems
Deep learning
Non-verbal communication
Dyadic Conversa
lcsh:Arts in general
Emotion Database
Listening Agent
Multimodal Synthesis
lcsh:N
Artificial intelligence
business
Music
Subjects
Details
- Language :
- English
- ISSN :
- 16469798 and 21830088
- Volume :
- 10
- Issue :
- 2
- Database :
- OpenAIRE
- Journal :
- Journal of Science and Technology of the Arts
- Accession number :
- edsair.doi.dedup.....52ae531d01b3be08c241b4b804c1a5bd
- Full Text :
- https://doi.org/10.34632/citarj.2018.10.2