Search

Recent advances in text-to-music editing, which employ text queries to modify music (e.g.\ by changing its style or adjusting instrumental components), present unique challenges and opportunities for AI-assisted music creation. Previous approaches in this domain have been constrained by the necessity to train specific editing models from scratch, which is both resource-intensive and inefficient; other research uses large language models to predict edited music, resulting in imprecise audio reconstruction. To Combine the strengths and address these limitations, we introduce Instruct-MusicGen, a novel approach that finetunes a pretrained MusicGen model to efficiently follow editing instructions such as adding, removing, or separating stems. Our approach involves a modification of the original MusicGen architecture by incorporating a text fusion module and an audio fusion module, which allow the model to process instruction texts and audio inputs concurrently and yield the desired edited music. Remarkably, Instruct-MusicGen only introduces 8% new parameters to the original MusicGen model and only trains for 5K steps, yet it achieves superior performance across all tasks compared to existing baselines, and demonstrates performance comparable to the models trained for specific tasks. This advancement not only enhances the efficiency of text-to-music editing but also broadens the applicability of music language models in dynamic music production environments., Comment: Code and demo are available at: https://github.com/ldzhangyx/instruct-musicgen

Generating novel views from a single image remains a challenging task due to the complexity of 3D scenes and the limited diversity in the existing multi-view datasets to train a model on. Recent research combining large-scale text-to-image (T2I) models with monocular depth estimation (MDE) has shown promise in handling in-the-wild images. In these methods, an input view is geometrically warped to novel views with estimated depth maps, then the warped image is inpainted by T2I models. However, they struggle with noisy depth maps and loss of semantic details when warping an input view to novel viewpoints. In this paper, we propose a novel approach for single-shot novel view synthesis, a semantic-preserving generative warping framework that enables T2I generative models to learn where to warp and where to generate, through augmenting cross-view attention with self-attention. Our approach addresses the limitations of existing methods by conditioning the generative model on source view images and incorporating geometric warping signals. Qualitative and quantitative evaluations demonstrate that our model outperforms existing methods in both in-domain and out-of-domain scenarios. Project page is available at https://GenWarp-NVS.github.io/., Comment: Accepted to NeurIPS 2024 / Project page: https://GenWarp-NVS.github.io

The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training. We propose Progressive Growing of Diffusion Autoencoder (PaGoDA), a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a $64\times$ reduced cost in training its diffusion model on 8x downsampled data; while at the inference, with the single-step, it performs state-of-the-art on ImageNet across all resolutions from 64x64 to 512x512, and text-to-image. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models (e.g., Stable Diffusion). The code is available at https://github.com/sony/pagoda., Comment: NeurIPS 2024

We develop a C++ package of the STOchastic LAttice Simulation (STOLAS) of cosmic inflation. It performs the numerical lattice simulation in the application of the stochastic-$\delta N$ formalism. STOLAS can directly compute the three-dimensional map of the observable curvature perturbation without estimating its statistical properties. In its application to two toy models of inflation, chaotic inflation and Starobinsky's linear-potential inflation, we confirm that STOLAS is well-consistent with the standard perturbation theory. Furthermore, by introducing the importance sampling technique, we have success in numerically sampling the current abundance of primordial black holes in a non-perturbative way. The package is available at https://github.com/STOchasticLAtticeSimulation/STOLAS_dist., Comment: 34 pages, 10 figures; v2 matches the accepted version in JCAP

In typical multimodal contrastive learning, such as CLIP, encoders produce one point in the latent representation space for each input. However, one-point representation has difficulty in capturing the relationship and the similarity structure of a huge amount of instances in the real world. For richer classes of the similarity, we propose the use of weighted point clouds, namely, sets of pairs of weight and vector, as representations of instances. In this work, we theoretically show the benefit of our proposed method through a new understanding of the contrastive loss of CLIP, which we call symmetric InfoNCE. We clarify that the optimal similarity that minimizes symmetric InfoNCE is the pointwise mutual information, and show an upper bound of excess risk on downstream classification tasks of representations that achieve the optimal similarity. In addition, we show that our proposed similarity based on weighted point clouds consistently achieves the optimal similarity. To verify the effectiveness of our proposed method, we demonstrate pretraining of text-image representation models and classification tasks on common benchmarks., Comment: Preprint. Under review

We have developed a two-dimensional neutron imager based on a semiconductor pixelated sensor, especially designed for experiments measuring of a spatial and a temporal behavior of quantum bound states of ultra-cold neutrons. Through these measurements, we expect to measure the ratio between the inertial and gravitational masses of neutrons and to test the equivalence principle in the quantum regime. As one of the principal neutron imagers, we fabricated a sensor with a sandwich configuration, named 10B-INTPIX4-sw, and tested its response to ultra-cold neutrons at the Los Alamos Neutron Science Center (LANSCE). We observed simultaneous events on both sandwiching sensors without significant loss of detection efficiency. The efficiency was evaluated to be about 16%, relative to the 10B/ZnS reference detector. The coincidence condition reduces its efficiency by a factor of about 3.

Graph Neural Networks (GNNs) have achieved notable success in various applications over graph data. However, recent research has revealed that real-world graphs often contain noise, and GNNs are susceptible to noise in the graph. To address this issue, several Graph Structure Learning (GSL) models have been introduced. While GSL models are tailored to enhance robustness against edge noise through edge reconstruction, a significant limitation surfaces: their high reliance on node features. This inherent dependence amplifies their susceptibility to noise within node features. Recognizing this vulnerability, we present DEGNN, a novel GNN model designed to adeptly mitigate noise in both edges and node features. The core idea of DEGNN is to design two separate experts: an edge expert and a node feature expert. These experts utilize self-supervised learning techniques to produce modified edges and node features. Leveraging these modified representations, DEGNN subsequently addresses downstream tasks, ensuring robustness against noise present in both edges and node features of real-world graphs. Notably, the modification process can be trained end-to-end, empowering DEGNN to adjust dynamically and achieves optimal edge and node representations for specific tasks. Comprehensive experiments demonstrate DEGNN's efficacy in managing noise, both in original real-world graphs and in graphs with synthetic noise., Comment: PAKDD 2024, the code is available at https://github.com/TaiHasegawa/DEGNN

Exemplar-Free Class Incremental Learning is a highly challenging setting where replay memory is unavailable. Methods relying on frozen feature extractors have drawn attention recently in this setting due to their impressive performances and lower computational costs. However, those methods are highly dependent on the data used to train the feature extractor and may struggle when an insufficient amount of classes are available during the first incremental step. To overcome this limitation, we propose to use a pre-trained text-to-image diffusion model in order to generate synthetic images of future classes and use them to train the feature extractor. Experiments on the standard benchmarks CIFAR100 and ImageNet-Subset demonstrate that our proposed method can be used to improve state-of-the-art methods for exemplar-free class incremental learning, especially in the most difficult settings where the first incremental step only contains few classes. Moreover, we show that using synthetic samples of future classes achieves higher performance than using real data from different classes, paving the way for better and less costly pre-training methods for incremental learning.

Prompt engineering is effective for controlling the output of text-to-image (T2I) generative models, but it is also laborious due to the need for manually crafted prompts. This challenge has spurred the development of algorithms for automated prompt generation. However, these methods often struggle with transferability across T2I models, require white-box access to the underlying model, and produce non-intuitive prompts. In this work, we introduce PRISM, an algorithm that automatically identifies human-interpretable and transferable prompts that can effectively generate desired concepts given only black-box access to T2I models. Inspired by large language model (LLM) jailbreaking, PRISM leverages the in-context learning ability of LLMs to iteratively refine the candidate prompts distribution for given reference images. Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney.

A neural network for software compensation was developed for the highly granular CALICE Analogue Hadronic Calorimeter (AHCAL). The neural network uses spatial and temporal event information from the AHCAL and energy information, which is expected to improve sensitivity to shower development and the neutron fraction of the hadron shower. The neural network method produced a depth-dependent energy weighting and a time-dependent threshold for enhancing energy deposits consistent with the timescale of evaporation neutrons. Additionally, it was observed to learn an energy-weighting indicative of longitudinal leakage correction. In addition, the method produced a linear detector response and outperformed a published control method regarding resolution for every particle energy studied.

The Crab Nebula, also known as Tau A, is a polarized astronomical source at millimeter wavelengths. It has been used as a stable light source for polarization angle calibration in millimeter-wave astronomy. However, it is known that its intensity and polarization vary as a function of time at a variety of wavelengths. Thus, it is of interest to verify the stability of the millimeter-wave polarization. If detected, polarization variability may be used to better understand the dynamics of Tau~A, and for understanding the validity of Tau~A as a calibrator. One intriguing application of such observation is to use it for the search of axion-like particles (ALPs). Ultralight ALPs couple to photons through a Chern-Simons term, and induce a temporal oscillation in the polarization angle of linearly polarized sources. After assessing a number of systematic errors and testing for internal consistency, we evaluate the variability of the polarization angle of the Crab Nebula using 2015 and 2016 observations with the 150 GHz POLARBEAR instrument. We place a median 95% upper bound of polarization oscillation amplitude $A < 0.065^\circ$ over the oscillation frequencies from $0.75~\mathrm{year}^{-1}$ to $0.66~\mathrm{hour}^{-1}$. Assuming that no sources other than ALP are causing Tau A's polarization angle variation, that the ALP constitutes all the dark matter, and that the ALP field is a stochastic Gaussian field, this bound translates into a median 95% upper bound of ALP-photon coupling $g_{a\gamma\gamma} < 2.16\times10^{-12}\,\mathrm{GeV}^{-1}\times(m_a/10^{-21} \mathrm{eV})$ in the mass range from $9.9\times10^{-23} \mathrm{eV}$ to $7.7\times10^{-19} \mathrm{eV}$. This demonstrates that this type of analysis using bright polarized sources is as competitive as those using the polarization of cosmic microwave background in constraining ALPs., Comment: 24 pages, 19 figures, 5 tables