Search

Your search keyword '"Rouditchenko, Andrew"' showing total 6 results
Start Over Author "Rouditchenko, Andrew" Publisher ieee
6 results on '"Rouditchenko, Andrew"'

3. Self-Supervised Audio-Visual Co-Segmentation

Author

MIT-IBM Watson AI Lab, Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory, Rouditchenko, Andrew, Zhao, Hang, Gan, Chuang, McDermott, Joshua Hartman, Torralba, Antonio, MIT-IBM Watson AI Lab, Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory, Rouditchenko, Andrew, Zhao, Hang, Gan, Chuang, McDermott, Joshua Hartman, and Torralba, Antonio

Abstract

© 2019 IEEE. Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this paper we develop a neural network model for visual object segmentation and sound source separation that learns from natural videos through self-supervision. The model is an extension of recently proposed work that maps image pixels to sounds [1]. Here, we introduce a learning approach to disentangle concepts in the neural networks, and assign semantic categories to network feature channels to enable independent image segmentation and sound source separation after audio-visual training on videos. Our evaluations show that the disentangled model outperforms several baselines in semantic segmentation and sound source separation.

Published
2022

4. Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos

Author

Chen, Brian, Rouditchenko, Andrew, Duarte, Kevin, Kuehne, Hilde, Thomas, Samuel, Boggust, Angie, Panda, Rameswar, Kingsbury, Brian, Feris, Rogerio, Harwath, David, Glass, James, Picheny, Michael, and Chang, Shih-Fu

Subjects
FOS: Computer and information sciences, Computer Vision and Pattern Recognition (cs.CV), Computer Science - Computer Vision and Pattern Recognition
Abstract

Multimodal self-supervised learning is getting more and more attention as it allows not only to train large networks without human supervision but also to search and retrieve data across various modalities. In this context, this paper proposes a self-supervised training framework that learns a common multimodal embedding space that, in addition to sharing representations across different modalities, enforces a grouping of semantically similar instances. To this end, we extend the concept of instance-level contrastive learning with a multimodal clustering step in the training pipeline to capture semantic similarities across modalities. The resulting embedding space enables retrieval of samples across all modalities, even from unseen datasets and different domains. To evaluate our approach, we train our model on the HowTo100M dataset and evaluate its zero-shot retrieval capabilities in two challenging domains, namely text-to-video retrieval, and temporal action localization, showing state-of-the-art results on four different datasets., Comment: To be presented at ICCV 2021

Published
2021

5. Self-Supervised Audio-Visual Co-Segmentation

Author

MIT-IBM Watson AI Lab, Rouditchenko, Andrew, Zhao, Hang, Gan, Chuang, McDermott, Josh, Torralba, Antonio, MIT-IBM Watson AI Lab, Rouditchenko, Andrew, Zhao, Hang, Gan, Chuang, McDermott, Josh, and Torralba, Antonio

Abstract

© 2019 IEEE. Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this paper we develop a neural network model for visual object segmentation and sound source separation that learns from natural videos through self-supervision. The model is an extension of recently proposed work that maps image pixels to sounds [1]. Here, we introduce a learning approach to disentangle concepts in the neural networks, and assign semantic categories to network feature channels to enable independent image segmentation and sound source separation after audio-visual training on videos. Our evaluations show that the disentangled model outperforms several baselines in semantic segmentation and sound source separation.

Published
2021

Catalog

Books, media, physical & digital resources
See catalog results