1. VMCML: Video and Music Matching via Cross-Modality Lifting
- Author
-
Lee, Yi-Shan, Tseng, Wei-Cheng, Wang, Fu-En, and Sun, Min
- Subjects
Computer Science - Computer Vision and Pattern Recognition - Abstract
We propose a content-based system for matching video and background music. The system aims to address the challenges in music recommendation for new users or new music give short-form videos. To this end, we propose a cross-modal framework VMCML that finds a shared embedding space between video and music representations. To ensure the embedding space can be effectively shared by both representations, we leverage CosFace loss based on margin-based cosine similarity loss. Furthermore, we establish a large-scale dataset called MSVD, in which we provide 390 individual music and the corresponding matched 150,000 videos. We conduct extensive experiments on Youtube-8M and our MSVD datasets. Our quantitative and qualitative results demonstrate the effectiveness of our proposed framework and achieve state-of-the-art video and music matching performance.
- Published
- 2023