Back to Search
Start Over
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos
- Source :
- Computer Vision – ECCV 2020 ISBN: 9783030585471, ECCV (4)
- Publication Year :
- 2020
- Publisher :
- Springer International Publishing, 2020.
-
Abstract
- Automatically generating sentences to describe events and temporally localizing sentences in a video are two important tasks that bridge language and videos. Recent techniques leverage the multimodal nature of videos by using off-the-shelf features to represent videos, but interactions between modalities are rarely explored. Inspired by the fact that there exist cross-modal interactions in the human brain, we propose a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos and thus improve performances on both tasks. We model modality interaction in both the sequence and channel levels in a pairwise fashion, and the pairwise interaction also provides some explainability for the predictions of target tasks. We demonstrate the effectiveness of our method and validate specific design choices through extensive ablation studies. Our method turns out to achieve state-of-the-art performances on four standard benchmark datasets: MSVD and MSR-VTT (event captioning task), and Charades-STA and ActivityNet Captions (temporal sentence localization task).
- Subjects :
- Closed captioning
Modality (human–computer interaction)
Modalities
business.industry
Computer science
Event (computing)
02 engineering and technology
010501 environmental sciences
computer.software_genre
01 natural sciences
Task (project management)
0202 electrical engineering, electronic engineering, information engineering
Leverage (statistics)
020201 artificial intelligence & image processing
Pairwise comparison
Artificial intelligence
business
computer
Sentence
Natural language processing
0105 earth and related environmental sciences
Subjects
Details
- ISBN :
- 978-3-030-58547-1
- ISBNs :
- 9783030585471
- Database :
- OpenAIRE
- Journal :
- Computer Vision – ECCV 2020 ISBN: 9783030585471, ECCV (4)
- Accession number :
- edsair.doi...........37a521434e8b2163149b20a6f4224c43