Search

The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.

Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA), capable of answering music-related questions and generating captions for music files. Our model utilizes audio representations from a pretrained MERT model to extract music features. However, obtaining a suitable dataset for training the MU-LLaMA model remains challenging, as existing publicly accessible audio question answering datasets lack the necessary depth for open-ended music question answering. To fill this gap, we present a methodology for generating question-answer pairs from existing audio captioning datasets and introduce the MusicQA Dataset designed for answering open-ended music-related questions. The experiments demonstrate that the proposed MU-LLaMA model, trained on our designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.

Digital contact tracing and analysis of social distancing from smartphone location data are two prime examples of non-therapeutic interventions used in many countries to mitigate the impact of the COVID-19 pandemic. While many understand the importance of trading personal privacy for the public good, others have been alarmed at the potential for surveillance via measures enabled through location tracking on smartphones. In our research, we analyzed massive yet atomic individual-level location data containing over 22 billion records from ten Blue (Democratic) and ten Red (Republican) cities in the U.S., based on which we present, herein, some of the first evidence of how Americans responded to the increasing concerns that government authorities, the private sector, and public health experts might use individual-level location data to track the COVID-19 spread. First, we found a significant decreasing trend of mobile-app location-sharing opt-out. Whereas areas with more Democrats were more privacy-concerned than areas with more Republicans before the advent of the COVID-19 pandemic, there was a significant decrease in the overall opt-out rates after COVID-19, and this effect was more salient among Democratic than Republican cities. Second, people who practiced social distancing (i.e., those who traveled less and interacted with fewer close contacts during the pandemic) were also less likely to opt-out, whereas the converse was true for people who practiced less social-distancing. This relationship also was more salient among Democratic than Republican cities. Third, high-income populations and males, compared with low-income populations and females, were more privacy-conscientious and more likely to opt-out of location tracking.