Search

We investigate what kind of images lie in the high-density regions of diffusion models. We introduce a theoretical mode-tracking process capable of pinpointing the exact mode of the denoising distribution, and we propose a practical high-probability sampler that consistently generates images of higher likelihood than usual samplers. Our empirical findings reveal the existence of significantly higher likelihood samples that typical samplers do not produce, often manifesting as cartoon-like drawings or blurry images depending on the noise level. Curiously, these patterns emerge in datasets devoid of such examples. We also present a novel approach to track sample likelihoods in diffusion SDEs, which remarkably incurs no additional computational cost.

The covariance for clean data given a noisy observation is an important quantity in many conditional generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations. We propose a new framework that sidesteps these issues by using covariance information that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's formula. We integrate these sources of information using {\em (i)} a novel method to transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps., Comment: 24 pages, 11 figures

In turbulent flows, tracking the source of a passive scalar cue requires exploiting the limited information that can be gleaned from rare, stochastic encounters with the cue. When crafting a search policy, the most challenging and important decision is what to do in the absence of an encounter. In this work, we perform high-fidelity direct numerical simulations of a turbulent flow with a stationary source of tracer particles, and obtain quasi-optimal policies (in the sense of minimal average search time) with respect to the empirical encounter statistics. We study the trajectories under such policies and compare the results to those of the infotaxis heuristic. In the presence of a strong mean wind, the optimal motion in the absence of an encounter is zigzagging (akin to the well-known insect behavior "casting") followed by a return to the starting location. The zigzag motion generates characteristic $t^{1/2}$ scaling of the rms displacement envelope. By passing to the limit where the probability of detection vanishes, we connect these results to the classical linear search problem and derive an estimate of the tail of the arrival time pdf as a stretched exponential, which agrees with Monte Carlo results. We also discuss what happens as the wind speed decreases., Comment: 7 pages, 6 figures, plus supplement

Several generative models with elaborate training and sampling procedures have been proposed recently to accelerate structure-based drug design (SBDD); however, perplexingly, their empirical performance turns out to be suboptimal. We seek to better understand this phenomenon from both theoretical and empirical perspectives. Since most of these models apply graph neural networks (GNNs), one may suspect that they inherit the representational limitations of GNNs. We analyze this aspect, establishing the first such results for protein-ligand complexes. A plausible counterview may attribute the underperformance of these models to their excessive parameterizations, inducing expressivity at the expense of generalization. We also investigate this possibility with a simple metric-aware approach that learns an economical surrogate for affinity to infer an unlabelled molecular graph and optimizes for labels conditioned on this graph and molecular properties. The resulting model achieves state-of-the-art results using 100x fewer trainable parameters and affords up to 1000x speedup. Collectively, our findings underscore the need to reassess and redirect the existing paradigm and efforts for SBDD., Comment: 25 pages, 11 figures

The transition towards carbon neutral transportation and energy sources increases the global demand for mineral raw materials while easy-to-find near-surface (\< 200 m) ore deposits are unlikely discovered in well-explored areas such as Europe. In order to increase the mineral exploration success rate, the project SEEMS DEEP (SEismic and ElectroMagnetic methodS for DEEP mineral exploration) develops geophysical deep exploration workflow capable of imaging the bedrock from the surface down to several kilometres depth. In this paper, we present first results from ground electrical and electromagnetic surveys conducted at the SEEM DEEP geological test site, namely the Koillismaa Layered Intrusion Complex in north-eastern Finland. Here, a 1.7 km long hole drilled by GTK intersected mafic-ultramafic rocks with anomalous electrical and chargeability properties at ~1400 m depth, making it an interesting case study to test the ability of such technologies for imaging resistivity and chargeability contrasts at several kilometre depth.

The problem of locating an odor source in turbulent flows is central to key applications such as environmental monitoring and disaster response. We address this challenge by designing an algorithm based on Bayesian inference, which uses odor measurements from an ensemble of static sensors to estimate the source position through a stochastic model of the environment. The problem is hard because of the multi-scale and out-of-equilibrium properties of turbulent transport, which lacks accurate analytical and phenomenological modeling, thus preventing a guaranteed convergence for Bayesian approaches. To overcome the risk of relying on a single unavoidably wrong model approximation, we propose a method to rank "many wrong models" and to blend their predictions. We evaluate our weighted Bayesian update algorithm by its ability to estimate the source location with predefined accuracy and/or within a specified time frame, and compare it to standard Monte Carlo sampling methods. To demonstrate the robustness and potential applications of both approaches under realistic environmental conditions, we use high-quality direct numerical simulations of the Navier-Stokes equations to mimic the transport of odors in the atmospheric boundary layer. Despite minimal prior information about the source and environmental conditions, our proposed approach consistently proves to be more accurate, reliable, and robust than Monte Carlo methods, thus showing promise as a new tool for addressing the odor source localization problem in real-world scenarios.

Deep neural networks (DNNs) excel on clean images but struggle with corrupted ones. Incorporating specific corruptions into the data augmentation pipeline can improve robustness to those corruptions but may harm performance on clean images and other types of distortion. In this paper, we introduce an alternative approach that improves the robustness of DNNs to a wide range of corruptions without compromising accuracy on clean images. We first demonstrate that input perturbations can be mimicked by multiplicative perturbations in the weight space. Leveraging this, we propose Data Augmentation via Multiplicative Perturbation (DAMP), a training method that optimizes DNNs under random multiplicative weight perturbations. We also examine the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) and show that it optimizes DNNs under adversarial multiplicative weight perturbations. Experiments on image classification datasets (CIFAR-10/100, TinyImageNet and ImageNet) and neural network architectures (ResNet50, ViT-S/16, ViT-B/16) show that DAMP enhances model generalization performance in the presence of corruptions across different settings. Notably, DAMP is able to train a ViT-S/16 on ImageNet from scratch, reaching the top-1 error of 23.7% which is comparable to ResNet50 without extensive data augmentations., Comment: Published at NeurIPS 2024 (spotlight). Code is available at https://github.com/trungtrinh44/DAMP

Introducing training-time augmentations is a key technique to enhance generalization and prepare deep neural networks against test-time corruptions. Inspired by the success of generative diffusion models, we propose a novel approach of coupling data mollification, in the form of image noising and blurring, with label smoothing to align predicted label confidences with image degradation. The method is simple to implement, introduces negligible overheads, and can be combined with existing augmentations. We demonstrate improved robustness and uncertainty quantification on the corrupted image benchmarks of the CIFAR and TinyImageNet datasets., Comment: Under review

In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset. This work paves the way for more advances in discrete diffusion models with potentially significant enhancements in performance., Comment: 10 pages, 7 figures

Retrosynthesis, the task of identifying precursors for a given molecule, can be naturally framed as a conditional graph generation task. Diffusion models are a particularly promising modelling approach, enabling post-hoc conditioning and trading off quality for speed during generation. We show mathematically that permutation equivariant denoisers severely limit the expressiveness of graph diffusion models and thus their adaptation to retrosynthesis. To address this limitation, we relax the equivariance requirement such that it only applies to aligned permutations of the conditioning and the generated graphs obtained through atom mapping. Our new denoiser achieves the highest top-$1$ accuracy ($54.7$\%) across template-free and template-based methods on USPTO-50k. We also demonstrate the ability for flexible post-training conditioning and good sample quality with small diffusion step counts, highlighting the potential for interactive applications and additional controls for multi-step planning., Comment: 28 pages, 9 figures

Climate and weather prediction traditionally relies on complex numerical simulations of atmospheric physics. Deep learning approaches, such as transformers, have recently challenged the simulation paradigm with complex network forecasts. However, they often act as data-driven black-box models that neglect the underlying physics and lack uncertainty quantification. We address these limitations with ClimODE, a spatiotemporal continuous-time process that implements a key principle of advection from statistical mechanics, namely, weather changes due to a spatial movement of quantities over time. ClimODE models precise weather evolution with value-conserving dynamics, learning global weather transport as a neural flow, which also enables estimating the uncertainty in predictions. Our approach outperforms existing data-driven methods in global and regional forecasting with an order of magnitude smaller parameterization, establishing a new state of the art., Comment: Accepted as ICLR 2024 Oral. Project website: https://yogeshverma1998.github.io/ClimODE/

This work introduces FMG, a field-based model for drug-like molecule generation. We show how the flexibility of this method provides crucial advantages over the prevalent, point-cloud based methods, and achieves competitive molecular stability generation. We tackle optical isomerism (enantiomers), a previously omitted molecular property that is crucial for drug safety and effectiveness, and thus account for all molecular geometry aspects. We demonstrate how previous methods are invariant to a group of transformations that includes enantiomer pairs, leading them invariant to the molecular R and S configurations, while our field-based generative model captures this property., Comment: 15 pages, 14 figures

As global groundwater levels continue to decline rapidly, there is a growing need for advanced techniques to monitor and manage aquifers effectively. This study focuses on validating a numerical model using seismic data from a small-scale experimental setup designed to estimate water volume in a porous reservoir. Expanding on previous work with synthetic data, we analyze seismic data acquired from a controlled experimental site in Laukaa, Finland. By employing neural networks, we directly estimate water volume from seismic responses, bypassing the traditional need for separate determinations, for example, of reservoir water table level and porosity. The study models wave propagation through a coupled poroviscoelastic-viscoelastic medium using a three-dimensional discontinuous Galerkin method. The proposed methodology is validated against experimental data, aiming to improve precision in mapping current water volumes and contributing to the development of sustainable groundwater management practices.

Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available for pretraining. In this work, we study the challenges of creating LLMs for Finnish, a language spoken by less than 0.1% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI. For model evaluation, we introduce FIN-bench, a version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at https://turkunlp.org/gpt3-finnish., Comment: 17 pages (10 main), 7 figures, 5 tables

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep, Comment: Code available at https://github.com/qbouniot/AffScoreDeep