Search

We investigate what kind of images lie in the high-density regions of diffusion models. We introduce a theoretical mode-tracking process capable of pinpointing the exact mode of the denoising distribution, and we propose a practical high-probability sampler that consistently generates images of higher likelihood than usual samplers. Our empirical findings reveal the existence of significantly higher likelihood samples that typical samplers do not produce, often manifesting as cartoon-like drawings or blurry images depending on the noise level. Curiously, these patterns emerge in datasets devoid of such examples. We also present a novel approach to track sample likelihoods in diffusion SDEs, which remarkably incurs no additional computational cost.

The covariance for clean data given a noisy observation is an important quantity in many conditional generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations. We propose a new framework that sidesteps these issues by using covariance information that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's formula. We integrate these sources of information using {\em (i)} a novel method to transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps., Comment: 24 pages, 11 figures

In turbulent flows, tracking the source of a passive scalar cue requires exploiting the limited information that can be gleaned from rare, stochastic encounters with the cue. When crafting a search policy, the most challenging and important decision is what to do in the absence of an encounter. In this work, we perform high-fidelity direct numerical simulations of a turbulent flow with a stationary source of tracer particles, and obtain quasi-optimal policies (in the sense of minimal average search time) with respect to the empirical encounter statistics. We study the trajectories under such policies and compare the results to those of the infotaxis heuristic. In the presence of a strong mean wind, the optimal motion in the absence of an encounter is zigzagging (akin to the well-known insect behavior "casting") followed by a return to the starting location. The zigzag motion generates characteristic $t^{1/2}$ scaling of the rms displacement envelope. By passing to the limit where the probability of detection vanishes, we connect these results to the classical linear search problem and derive an estimate of the tail of the arrival time pdf as a stretched exponential, which agrees with Monte Carlo results. We also discuss what happens as the wind speed decreases., Comment: 7 pages, 6 figures, plus supplement

Several generative models with elaborate training and sampling procedures have been proposed recently to accelerate structure-based drug design (SBDD); however, perplexingly, their empirical performance turns out to be suboptimal. We seek to better understand this phenomenon from both theoretical and empirical perspectives. Since most of these models apply graph neural networks (GNNs), one may suspect that they inherit the representational limitations of GNNs. We analyze this aspect, establishing the first such results for protein-ligand complexes. A plausible counterview may attribute the underperformance of these models to their excessive parameterizations, inducing expressivity at the expense of generalization. We also investigate this possibility with a simple metric-aware approach that learns an economical surrogate for affinity to infer an unlabelled molecular graph and optimizes for labels conditioned on this graph and molecular properties. The resulting model achieves state-of-the-art results using 100x fewer trainable parameters and affords up to 1000x speedup. Collectively, our findings underscore the need to reassess and redirect the existing paradigm and efforts for SBDD., Comment: 25 pages, 11 figures

The transition towards carbon neutral transportation and energy sources increases the global demand for mineral raw materials while easy-to-find near-surface (\< 200 m) ore deposits are unlikely discovered in well-explored areas such as Europe. In order to increase the mineral exploration success rate, the project SEEMS DEEP (SEismic and ElectroMagnetic methodS for DEEP mineral exploration) develops geophysical deep exploration workflow capable of imaging the bedrock from the surface down to several kilometres depth. In this paper, we present first results from ground electrical and electromagnetic surveys conducted at the SEEM DEEP geological test site, namely the Koillismaa Layered Intrusion Complex in north-eastern Finland. Here, a 1.7 km long hole drilled by GTK intersected mafic-ultramafic rocks with anomalous electrical and chargeability properties at ~1400 m depth, making it an interesting case study to test the ability of such technologies for imaging resistivity and chargeability contrasts at several kilometre depth.

The problem of locating an odor source in turbulent flows is central to key applications such as environmental monitoring and disaster response. We address this challenge by designing an algorithm based on Bayesian inference, which uses odor measurements from an ensemble of static sensors to estimate the source position through a stochastic model of the environment. The problem is hard because of the multi-scale and out-of-equilibrium properties of turbulent transport, which lacks accurate analytical and phenomenological modeling, thus preventing a guaranteed convergence for Bayesian approaches. To overcome the risk of relying on a single unavoidably wrong model approximation, we propose a method to rank "many wrong models" and to blend their predictions. We evaluate our weighted Bayesian update algorithm by its ability to estimate the source location with predefined accuracy and/or within a specified time frame, and compare it to standard Monte Carlo sampling methods. To demonstrate the robustness and potential applications of both approaches under realistic environmental conditions, we use high-quality direct numerical simulations of the Navier-Stokes equations to mimic the transport of odors in the atmospheric boundary layer. Despite minimal prior information about the source and environmental conditions, our proposed approach consistently proves to be more accurate, reliable, and robust than Monte Carlo methods, thus showing promise as a new tool for addressing the odor source localization problem in real-world scenarios.

Deep neural networks (DNNs) excel on clean images but struggle with corrupted ones. Incorporating specific corruptions into the data augmentation pipeline can improve robustness to those corruptions but may harm performance on clean images and other types of distortion. In this paper, we introduce an alternative approach that improves the robustness of DNNs to a wide range of corruptions without compromising accuracy on clean images. We first demonstrate that input perturbations can be mimicked by multiplicative perturbations in the weight space. Leveraging this, we propose Data Augmentation via Multiplicative Perturbation (DAMP), a training method that optimizes DNNs under random multiplicative weight perturbations. We also examine the recently proposed Adaptive Sharpness-Aware Minimization (ASAM) and show that it optimizes DNNs under adversarial multiplicative weight perturbations. Experiments on image classification datasets (CIFAR-10/100, TinyImageNet and ImageNet) and neural network architectures (ResNet50, ViT-S/16, ViT-B/16) show that DAMP enhances model generalization performance in the presence of corruptions across different settings. Notably, DAMP is able to train a ViT-S/16 on ImageNet from scratch, reaching the top-1 error of 23.7% which is comparable to ResNet50 without extensive data augmentations., Comment: Published at NeurIPS 2024 (spotlight). Code is available at https://github.com/trungtrinh44/DAMP

Introducing training-time augmentations is a key technique to enhance generalization and prepare deep neural networks against test-time corruptions. Inspired by the success of generative diffusion models, we propose a novel approach of coupling data mollification, in the form of image noising and blurring, with label smoothing to align predicted label confidences with image degradation. The method is simple to implement, introduces negligible overheads, and can be combined with existing augmentations. We demonstrate improved robustness and uncertainty quantification on the corrupted image benchmarks of the CIFAR and TinyImageNet datasets., Comment: Under review

In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset. This work paves the way for more advances in discrete diffusion models with potentially significant enhancements in performance., Comment: 10 pages, 7 figures

Retrosynthesis, the task of identifying precursors for a given molecule, can be naturally framed as a conditional graph generation task. Diffusion models are a particularly promising modelling approach, enabling post-hoc conditioning and trading off quality for speed during generation. We show mathematically that permutation equivariant denoisers severely limit the expressiveness of graph diffusion models and thus their adaptation to retrosynthesis. To address this limitation, we relax the equivariance requirement such that it only applies to aligned permutations of the conditioning and the generated graphs obtained through atom mapping. Our new denoiser achieves the highest top-$1$ accuracy ($54.7$\%) across template-free and template-based methods on USPTO-50k. We also demonstrate the ability for flexible post-training conditioning and good sample quality with small diffusion step counts, highlighting the potential for interactive applications and additional controls for multi-step planning., Comment: 28 pages, 9 figures

Climate and weather prediction traditionally relies on complex numerical simulations of atmospheric physics. Deep learning approaches, such as transformers, have recently challenged the simulation paradigm with complex network forecasts. However, they often act as data-driven black-box models that neglect the underlying physics and lack uncertainty quantification. We address these limitations with ClimODE, a spatiotemporal continuous-time process that implements a key principle of advection from statistical mechanics, namely, weather changes due to a spatial movement of quantities over time. ClimODE models precise weather evolution with value-conserving dynamics, learning global weather transport as a neural flow, which also enables estimating the uncertainty in predictions. Our approach outperforms existing data-driven methods in global and regional forecasting with an order of magnitude smaller parameterization, establishing a new state of the art., Comment: Accepted as ICLR 2024 Oral. Project website: https://yogeshverma1998.github.io/ClimODE/

This work introduces FMG, a field-based model for drug-like molecule generation. We show how the flexibility of this method provides crucial advantages over the prevalent, point-cloud based methods, and achieves competitive molecular stability generation. We tackle optical isomerism (enantiomers), a previously omitted molecular property that is crucial for drug safety and effectiveness, and thus account for all molecular geometry aspects. We demonstrate how previous methods are invariant to a group of transformations that includes enantiomer pairs, leading them invariant to the molecular R and S configurations, while our field-based generative model captures this property., Comment: 15 pages, 14 figures

As global groundwater levels continue to decline rapidly, there is a growing need for advanced techniques to monitor and manage aquifers effectively. This study focuses on validating a numerical model using seismic data from a small-scale experimental setup designed to estimate water volume in a porous reservoir. Expanding on previous work with synthetic data, we analyze seismic data acquired from a controlled experimental site in Laukaa, Finland. By employing neural networks, we directly estimate water volume from seismic responses, bypassing the traditional need for separate determinations, for example, of reservoir water table level and porosity. The study models wave propagation through a coupled poroviscoelastic-viscoelastic medium using a three-dimensional discontinuous Galerkin method. The proposed methodology is validated against experimental data, aiming to improve precision in mapping current water volumes and contributing to the development of sustainable groundwater management practices.

Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available for pretraining. In this work, we study the challenges of creating LLMs for Finnish, a language spoken by less than 0.1% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI. For model evaluation, we introduce FIN-bench, a version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at https://turkunlp.org/gpt3-finnish., Comment: 17 pages (10 main), 7 figures, 5 tables

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep, Comment: Code available at https://github.com/qbouniot/AffScoreDeep

Answering a question of Wright, we show that spheres of any radius are always connected in the curve graph of surfaces $\Sigma_{2,0}, \Sigma_{1,3},$ and $\Sigma_{0,6}$, and the union of two consecutive spheres is always connected for $\Sigma_{0, 5}$ and $\Sigma_{1,2}$. We also classify the connected components of spheres of radius 2 in the curve graph of $\Sigma_{0, 5}$ and $\Sigma_{1,2}$.

Gestational diabetes (GDM) poses a growing health risk to both pregnant women and their offspring. While telehealth interventions for GDM management have proven effective, they have traditionally relied on healthcare professionals for guidance and feedback. Our aim was to explore self-tracking in GDM with wearable sensors from self-discovery (i.e., learning associations between glucose levels and lifestyle) and user experience perspectives. We conducted a mixed-methods study with women diagnosed with GDM, utilizing continuous glucose monitor and three types of physical activity sensors (activity bracelet, hip-worn sensor, and electrocardiography sensor) for a week. Data from the sensors was collected, and participants were later interviewed about their experience with the wearable sensors. Additionally, we gathered maternal nutrition data through a 3-day food diary and recorded self-reported physical activity using a logbook. We discovered that continuous glucose monitors were especially valuable for self-discovery, particularly when establishing links between glucose levels and nutritional intake. Challenges associated with using wearable sensors data for self-discovery in GDM included: (1) Separation of glucose and physical activity data in different applications, (2) Missing key trackable features, such as light physical activity and non-walking activities, (3) Discrepancies in data, and (4) Differences in perceived versus measured physical activity. The placement of sensors on the body emerged as a critical factor influencing data quality and personal preferences. To conclude, an app where glucose, nutrition, and physical activity data are combined is needed to support self-discovery. This app should enable tracking of essential features for women with GDM, including light physical activity, with data originating from a single sensor to ensure consistency and eliminate redundancy.

Arrays of artificial spin ices exhibit reconfigurable ferromagnetic resonance frequencies that can be leveraged and designed for potential applications.However, analytical and numerical studies of the frequency response of artificial spin ices have remained somewhat limited due to the need of take into account nonlocal dipole fields in theoretical calculations or by long computation times in micromagnetic simulations. Here, we introduce Gaenice, a framework to compute magnon dispersion relations of arbitrary artificial spin ice configurations. Gaenice makes use of a tight-binding approach to compute the magnon bands. It also provides the user complete control of the interaction terms included, e.g., external field, anisotropy, exchange, and dipole, making it useful also to compute ferromagnetic resonances for a variety of structures, such as multilayers and ensembles of weakly or non-interacting nanoparticles. Because it relies on a semi-analytical model, Gaenice is computationally inexpensive and efficient, making it an attractive tool for the exploration of large parameter spaces.

Based on observations obtained with the Nordic Optical Telescope we investigate the spectral variability of the Herbig Ae star RR Tau. This star belongs to the UX Ori family, characterized by very deep fadings caused by the screening of the star with opaque fragments (clouds) of the protoplanetary discs. At the moments of such minima one observes strong spectral variability due to the fact that the dust cloud occults, for an observer, not only the star but also a part of the region where the emission spectrum originates. We calculated a series of obscuration models to interpret the observed variability of the H-alpha line parameters. We consider two main obscuration scenarios: (1) the dust screen rises vertically above the circumstellar disc, and (2) the screen intersects the line-of-sight moving azimuthally with the disc. In both cases the model of the emission region consists of a compact magnetosphere and a magneto-centrifugal disc wind. Comparison with observations shows that the first scenario explains well the variability of the radiation flux, the equivalent width, as well as the asymmetry of the H-alpha line during eclipses, while the second scenario explains them only partly. This permits us to suggest that in the case of RR Tau, the main causes of the eclipses are either a structured disc wind, or the charged dust lifted along the field lines of the poloidal component of the magnetic field of the circumstellar disc., Comment: 16 pages, 17 figures

We introduce a novel grid-independent model for learning partial differential equations (PDEs) from noisy and partial observations on irregular spatiotemporal grids. We propose a space-time continuous latent neural PDE model with an efficient probabilistic framework and a novel encoder design for improved data efficiency and grid independence. The latent state dynamics are governed by a PDE model that combines the collocation method and the method of lines. We employ amortized variational inference for approximate posterior estimation and utilize a multiple shooting technique for enhanced training speed and stability. Our model demonstrates state-of-the-art performance on complex synthetic and real-world datasets, overcoming limitations of previous approaches and effectively handling partially-observed data. The proposed model outperforms recent methods, showing its potential to advance data-driven PDE modeling and enabling robust, grid-independent modeling of complex partially-observed dynamic processes.

Deep Ensembles (DEs) demonstrate improved accuracy, calibration and robustness to perturbations over single neural networks partly due to their functional diversity. Particle-based variational inference (ParVI) methods enhance diversity by formalizing a repulsion term based on a network similarity kernel. However, weight-space repulsion is inefficient due to over-parameterization, while direct function-space repulsion has been found to produce little improvement over DEs. To sidestep these difficulties, we propose First-order Repulsive Deep Ensemble (FoRDE), an ensemble learning method based on ParVI, which performs repulsion in the space of first-order input gradients. As input gradients uniquely characterize a function up to translation and are much smaller in dimension than the weights, this method guarantees that ensemble members are functionally different. Intuitively, diversifying the input gradients encourages each network to learn different features, which is expected to improve the robustness of an ensemble. Experiments on image classification datasets and transfer learning tasks show that FoRDE significantly outperforms the gold-standard DEs and other ensemble methods in accuracy and calibration under covariate shift due to input perturbations., Comment: Published at ICLR 2024 (spotlight presentation). Code is available at https://github.com/AaltoPML/FoRDE

A collective excitation of the spin structure in a magnetic insulator can transmit spin-angular momentum with negligible dissipation. This quantum of a spin wave, introduced more than nine decades ago, has always been manipulated through magnetic dipoles, (i.e., timereversal symmetry). Here, we report the experimental observation of chiral-spin transport in multiferroic BiFeO3, where the spin transport is controlled by reversing the ferroelectric polarization (i.e., spatial inversion symmetry). The ferroelectrically controlled magnons produce an unprecedented ratio of up to 18% rectification at room temperature. The spin torque that the magnons in BiFeO3 carry can be used to efficiently switch the magnetization of adja-cent magnets, with a spin-torque efficiency being comparable to the spin Hall effect in heavy metals. Utilizing such a controllable magnon generation and transmission in BiFeO3, an alloxide, energy-scalable logic is demonstrated composed of spin-orbit injection, detection, and magnetoelectric control. This observation opens a new chapter of multiferroic magnons and paves an alternative pathway towards low-dissipation nanoelectronics.

Antibodies are Y-shaped proteins that neutralize pathogens and constitute the core of our adaptive immune system. De novo generation of new antibodies that target specific antigens holds the key to accelerating vaccine discovery. However, this co-design of the amino acid sequence and the 3D structure subsumes and accentuates some central challenges from multiple tasks, including protein folding (sequence to structure), inverse folding (structure to sequence), and docking (binding). We strive to surmount these challenges with a new generative model AbODE that extends graph PDEs to accommodate both contextual information and external interactions. Unlike existing approaches, AbODE uses a single round of full-shot decoding and elicits continuous differential attention that encapsulates and evolves with latent interactions within the antibody as well as those involving the antigen. We unravel fundamental connections between AbODE and temporal networks as well as graph-matching networks. The proposed model significantly outperforms existing methods on standard metrics across benchmarks., Comment: Accepted at ICML 2023

Optimal transport (OT) is a powerful geometric tool used to compare and align probability measures following the least effort principle. Despite its widespread use in machine learning (ML), OT problem still bears its computational burden, while at the same time suffering from the curse of dimensionality for measures supported on general high-dimensional spaces. In this paper, we propose to tackle these challenges using representation learning. In particular, we seek to learn an embedding space such that the samples of the two input measures become alignable in it with a simple affine mapping that can be calculated efficiently in closed-form. We then show that such approach leads to results that are comparable to solving the original OT problem when applied to the transfer learning task on which many OT baselines where previously evaluated in both homogeneous and heterogeneous DA settings. The code for our contribution is available at \url{https://github.com/Oleffa/LaOT}.

Magnetic droplets are nanoscale, non-topological, magnetodynamical solitons that can be nucleated in spin torque nano-oscillators (STNOs) or spin Hall nano-oscillators (SHNOs). All theoretical, numerical, and experimental droplet studies have so far focused on the free layer (FL), and any additional dynamics in the reference layer (RL) have been entirely ignored. Here we show, using all-perpendicular STNOs, that there is not only significant magnetodynamics in the RL, but the reference layer itself can host a droplet coexisting with the FL droplet. Both droplets are observed experimentally as stepwise changes and sharp peaks in the dc and differential resistance, respectively. Whereas the single FL droplet is highly stable, the coexistence state exhibits high-power broadband microwave noise. Micromagnetic simulations corroborate the experimental results and reveal a strong interaction between the droplets. Our demonstration of strongly interacting and closely spaced droplets offers a unique platform for fundamental studies of highly non-linear soliton pair dynamics., Comment: 23 pages, 6 figures

This paper introduces Bayesian uncertainty modeling using Stochastic Weight Averaging-Gaussian (SWAG) in Natural Language Understanding (NLU) tasks. We apply the approach to standard tasks in natural language inference (NLI) and demonstrate the effectiveness of the method in terms of prediction accuracy and correlation with human annotation disagreements. We argue that the uncertainty representations in SWAG better reflect subjective interpretation and the natural variation that is also present in human language understanding. The results reveal the importance of uncertainty modeling, an often neglected aspect of neural language modeling, in NLU tasks., Comment: NoDaLiDa 2023 camera ready

A continuum approach to study magnetoelectric multiferroic $\mathrm{BiFeO}_3$ (BFO) is proposed. Our modeling effort marries the ferroelectric (FE) phase field method and micromagnetic simulations in order to describe the entire multiferroic order parameter sector (polarization, oxygen antiphase tilts, strain, and magnetism) self-consistently on the same time and length scale. In this paper, we discuss our choice of ferroelectric and magnetic energy terms and demonstrate benchmarks against known behavior. We parameterize the lowest order couplings of the structural distortions against previous predictions from density functional theory calculations giving access to simulations of the FE domain wall (DW) topology. This allows us to estimate the energetic hierarchy and thicknesses of the numerous structural DWs. We then extend the model to the canted antiferromagnetic order and demonstrate how the ferroelectric domain boundaries influence the resulting magnetic DWs. We also highlight some capabilities of this model by providing two examples relevant for applications. We demonstrate spin wave transmission through the multiferroic domain boundaries which identify rectification in qualitative agreement with recent experimental observations. As a second example of application, we model fully-dynamical magnetoelectric switching, where we find a sensitivity on the Gilbert damping with respect to switching pathways. We envision that this modeling effort will set the basis for further work on properties of arbitrary 3D nanostructures of BFO (and related multiferroics) at the mesoscale., Comment: 15 pages (double column), 8 figures

To better understand the equilibrium $\gamma^\prime$(L1$_2$) precipitate morphology in Co-based superalloys, a phase field modeling sensitivity analysis is conducted to examine how four phase-field parameters [initial Co concentration ($c_0$), double-well barrier height ($\omega$), gradient energy density coefficient ($\kappa$), and lattice misfit strain ($\epsilon_{\rm misfit}$)] influence the $\gamma^\prime$(L1$_2$) precipitate size and morphology. Gaussian Process Regression (GPR) models are used to fit the sample points and to generate surrogate models for both precipitate size and morphology. In an Active Learning approach, a Bayesian Optimization algorithm is coupled with the GPR models to suggest new sample points to calculate and efficiently update the models based on a reduction of uncertainty. The algorithm has a user-defined objective, which controls the balance between exploration and exploitation for new suggested points. Our methodology provides a qualitative and quantitative relationship between the $\gamma^\prime$(L1$_2$) precipitate size and morphology and the four phase-field parameters, and concludes that the most sensitive phase-field parameter for precipitate size and morphology is the initial Co concentration ($c_0$) and the double-well barrier height ($\omega$), respectively. We note that the GPR model for precipitate morphology required adding a noise tolerance in order to avoid overfitting due to irregularities in some of the simulated equilibrium $\gamma^\prime$(L1$_2$) precipitate morphology., Comment: 24 pages, 10 figures

Hamiltonian mechanics is one of the cornerstones of natural sciences. Recently there has been significant interest in learning Hamiltonian systems in a free-form way directly from trajectory data. Previous methods have tackled the problem of learning from many short, low-noise trajectories, but learning from a small number of long, noisy trajectories, whilst accounting for model uncertainty has not been addressed. In this work, we present a Gaussian process model for Hamiltonian systems with efficient decoupled parameterisation, and introduce an energy-conserving shooting method that allows robust inference from both short and long trajectories. We demonstrate the method's success in learning Hamiltonian systems in various data settings., Comment: Accepted in TMLR (March 2023)

We introduce Functional Diffusion Processes (FDPs), which generalize score-based diffusion models to infinite-dimensional function spaces. FDPs require a new mathematical framework to describe the forward and backward dynamics, and several extensions to derive practical training objectives. These include infinite-dimensional versions of Girsanov theorem, in order to be able to compute an ELBO, and of the sampling theorem, in order to guarantee that functional evaluations in a countable set of points are equivalent to infinite-dimensional functions. We use FDPs to build a new breed of generative models in function spaces, which do not require specialized network architectures, and that can work with any kind of continuous data. Our results on real data show that FDPs achieve high-quality image generation, using a simple MLP architecture with orders of magnitude fewer parameters than existing diffusion models., Comment: Under review

The olfactory search POMDP (partially observable Markov decision process) is a sequential decision-making problem designed to mimic the task faced by insects searching for a source of odor in turbulence, and its solutions have applications to sniffer robots. As exact solutions are out of reach, the challenge consists in finding the best possible approximate solutions while keeping the computational cost reasonable. We provide a quantitative benchmarking of a solver based on deep reinforcement learning against traditional POMDP approximate solvers. We show that deep reinforcement learning is a competitive alternative to standard methods, in particular to generate lightweight policies suitable for robots.

Bose-Einstein condensation happens as a gas of bosons is cooled below its transition temperature, and the ground state becomes macroscopically occupied. The phase transition occurs in the thermodynamic limit of many particles. However, recent experimental progress has made it possible to assemble quantum many-body systems from the bottom up, for example, by adding single atoms to an optical lattice one at a time. Here, we show how one can predict the condensation temperature of a Bose gas from the energy fluctuations of a small number of bosons. To this end, we make use of recent advances in Lee-Yang theories of phase transitions, which allow us to determine the zeros and the poles of the partition function in the complex plane of the inverse temperature from the high cumulants of the energy fluctuations. By increasing the number of bosons in the trapping potential, we can predict the convergence point of the partition function zeros in the thermodynamic limit, where they reach the inverse critical temperature on the real axis. Using less than 100 bosons, we can estimate the condensation temperature for a Bose gas in a harmonic potential in two and three dimensions, and we also find that there is no phase transition in one dimension as one would expect., Comment: 11 pages, 6 figures

Programmers' mental models represent their knowledge and understanding of programs, programming concepts, and programming in general. They guide programmers' work and influence their task performance. Understanding mental models is important for designing work systems and practices that support programmers. Although the importance of programmers' mental models is widely acknowledged, research on mental models has decreased over the years. The results are scattered and do not take into account recent developments in software engineering. We analyze the state of research into programmers' mental models and provide an overview of existing research. We connect results on mental models from different strands of research to form a more unified knowledge base on the topic. We conducted a systematic literature review on programmers' mental models. We analyzed literature addressing mental models in different contexts, including mental models of programs, programming tasks, and programming concepts. Using nine search engines, we found 3678 articles (excluding duplicates). 84 were selected for further analysis. Using the snowballing technique, we obtained a final result set containing 187 articles. We show that the literature shares a kernel of shared understanding of mental models. By collating and connecting results on mental models from different fields of research, we uncovered some well-researched aspects, which we argue are fundamental characteristics of programmers' mental models. This work provides a basis for future work on mental models. The research field on programmers' mental models still faces many challenges rising from a lack of a shared knowledge base and poorly defined constructs. We created a unified knowledge base on the topic. We also point to directions for future studies. In particular, we call for studies that examine programmers working with modern practices and tools., Comment: Submitted to Information and Software Technology

Optical control of spin in semiconductors has been pioneered using nanostructures of III-V and II-VI semiconductors, but the emergence of two-dimensional van der Waals materials offers an alternative low-dimensional platform for spintronic phenomena. Indium selenide (InSe), a group-III monochalcogenide van der Waals material, has shown promise for opto-electronics due to its high electron mobility, tunable direct bandgap, and quantum transport. There are predictions of spin-dependent optical selection rules suggesting potential for all-optical excitation and control of spin in a two-dimensional layered material. Despite these predictions, layer-dependent optical spin phenomena in InSe have yet to be explored. Here, we present measurements of layer-dependent optical spin dynamics in few-layer and bulk InSe. Polarized photoluminescence reveals layer-dependent optical orientation of spin, thereby demonstrating the optical selection rules in few-layer InSe. Spin dynamics are also studied in many-layer InSe using time-resolved Kerr rotation spectroscopy. By applying out-of-plane and in-plane static magnetic fields for polarized emission measurements and Kerr measurements, respectively, the $g$-factor for InSe was extracted. Further investigations are done by calculating precession values using a $\textbf{k} \cdot \textbf{p}$ model, which is supported by \textit{ab-initio} density functional theory. Comparison of predicted precession rates with experimental measurements highlights the importance of excitonic effects in InSe for understanding spin dynamics. Optical orientation of spin is an important prerequisite for opto-spintronic phenomena and devices, and these first demonstrations of layer-dependent optical excitation of spins in InSe lay the foundation for combining layer-dependent spin properties with advantageous electronic properties found in this material., Comment: 11 pages, 6 figures, supplemental material

Generating new molecules is fundamental to advancing critical applications such as drug discovery and material synthesis. Flows can generate molecules effectively by inverting the encoding process, however, existing flow models either require artifactual dequantization or specific node/edge orderings, lack desiderata such as permutation invariance, or induce discrepancy between the encoding and the decoding steps that necessitates post hoc validity correction. We circumvent these issues with novel continuous normalizing E(3)-equivariant flows, based on a system of node ODEs coupled as a graph PDE, that repeatedly reconcile locally toward globally aligned densities. Our models can be cast as message-passing temporal networks, and result in superlative performance on the tasks of density estimation and molecular generation. In particular, our generated samples achieve state-of-the-art on both the standard QM9 and ZINC250K benchmarks., Comment: Accepted to NeurIPS 2022. More info at: https://yogeshverma1998.github.io/ModFlow/

Training dynamic models, such as neural ODEs, on long trajectories is a hard problem that requires using various tricks, such as trajectory splitting, to make model training work in practice. These methods are often heuristics with poor theoretical justifications, and require iterative manual tuning. We propose a principled multiple shooting technique for neural ODEs that splits the trajectories into manageable short segments, which are optimised in parallel, while ensuring probabilistic control on continuity over consecutive segments. We derive variational inference for our shooting-based latent neural ODE models and propose amortized encodings of irregularly sampled trajectories with a transformer-based recognition network with temporal attention and relative positional encoding. We demonstrate efficient and stable training, and state-of-the-art performance on multiple large-scale benchmark datasets.

In many practical scenarios, a flying insect must search for the source of an emitted cue which is advected by the atmospheric wind. On the macroscopic scales of interest, turbulence tends to mix the cue into patches of relatively high concentration over a background of very low concentration, so that the insect will only detect the cue intermittently and cannot rely on chemotactic strategies which simply climb the concentration gradient. In this work, we cast this search problem in the language of a partially observable Markov decision process (POMDP) and use the Perseus algorithm to compute strategies that are near-optimal with respect to the arrival time. We test the computed strategies on a large two-dimensional grid, present the resulting trajectories and arrival time statistics, and compare these to the corresponding results for several heuristic strategies, including (space-aware) infotaxis, Thompson sampling, and QMDP. We find that the near-optimal policy found by our implementation of Perseus outperforms all heuristics we test by several measures. We use the near-optimal policy to study how the search difficulty depends on the starting location. We discuss additionally the choice of initial belief and the robustness of the policies to changes in the environment. Finally, we present a detailed and pedagogical discussion about the implementation of the Perseus algorithm, including the benefits -- and pitfalls -- of employing a reward shaping function., Comment: 33 pages, 12 figures

Bayesian neural networks (BNNs) can account for both aleatoric and epistemic uncertainty. However, in BNNs the priors are often specified over the weights which rarely reflects true prior knowledge in large and complex neural network architectures. We present a simple approach to incorporate prior knowledge in BNNs based on external summary information about the predicted classification probabilities for a given dataset. The available summary information is incorporated as augmented data and modeled with a Dirichlet process, and we derive the corresponding \emph{Summary Evidence Lower BOund}. The approach is founded on Bayesian principles, and all hyperparameters have a proper probabilistic interpretation. We show how the method can inform the model about task difficulty and class imbalance. Extensive experiments show that, with negligible computational overhead, our method parallels and in many cases outperforms popular alternatives in accuracy, uncertainty calibration, and robustness against corruptions with both balanced and imbalanced data., Comment: Accepted in AISTATS 2023

We have studied the temperature and magnetic field dependence of the total magnetic moment of large-area permalloy artificial square spin ice arrays. The temperature dependence and hysteresis behavior are consistent with the coherent magnetization reversal expected in the Stoner-Wohlfarth model, with clear deviations due to inter-island interactions at small lattice spacing. Through micromagnetic simulations, we explore this behavior and demonstrate that the deviations result from increasingly complex magnetization reversal at small lattice spacing, induced by inter-island interactions, and depending critically on details of the island shapes. These results establish new means to tune the physical properties of artificial spin ice structures and other interacting nanomagnet systems, such as patterned magnetic media., Comment: 17 pages, 5 figures

While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the empirical success of coarse-to-fine modelling, we propose a new diffusion-like model that generates images through stochastically reversing the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. We interpret the solution of the forward heat equation with constant additive noise as a variational approximation in the diffusion latent variable model. Our new model shows emergent qualitative properties not seen in standard diffusion models, such as disentanglement of overall colour and shape in images. Spectral analysis on natural images highlights connections to diffusion models and reveals an implicit coarse-to-fine inductive bias in them.

Bayesian neural networks (BNNs) promise improved generalization under covariate shift by providing principled probabilistic representations of epistemic uncertainty. However, weight-based BNNs often struggle with high computational complexity of large-scale architectures and datasets. Node-based BNNs have recently been introduced as scalable alternatives, which induce epistemic uncertainty by multiplying each hidden node with latent random variables, while learning a point-estimate of the weights. In this paper, we interpret these latent noise variables as implicit representations of simple and domain-agnostic data perturbations during training, producing BNNs that perform well under covariate shift due to input corruptions. We observe that the diversity of the implicit corruptions depends on the entropy of the latent variables, and propose a straightforward approach to increase the entropy of these variables during training. We evaluate the method on out-of-distribution image classification benchmarks, and show improved uncertainty estimation of node-based BNNs under covariate shift due to input perturbations. As a side effect, the method also provides robustness against noisy training labels., Comment: Published at ICML 2022 (long oral presentation). Code is available at https://github.com/AaltoPML/node-BNN-covariate-shift

The transition-metal intercalated dichalcogenide CoNb$_3$S$_6$ is a triangular antiferromagnet that has recently been shown to exhibit a large anomalous Hall effect (AHE) below the N\'eel temperature, even though the response to an external field is very small. This suggests that there is an interesting magnetic structure that interacts with the electronic structure to yield the AHE, as collinear antiferromagnets cannot exhibit a nonzero AHE. We propose a model for magnetic transition-metal intercalated dichalcogenides and examine its ground state as function of interaction parameters. The model exhibits transitions between planar spin spirals, non-planar spin spirals, and a particular non-coplanar so-called $3q$ state. This latter state must exhibit a nonzero AHE, while the spin spirals do not.

Pattern-forming nonequilibrium systems are ubiquitous in nature, from driven quantum matter and biological life forms to atmospheric and interstellar gases. Identifying universal aspects of their far-from-equilibrium dynamics and statistics poses major conceptual and practical challenges due to the absence of energy and momentum conservation laws. Here, we experimentally and theoretically investigate the statistics of prototypical nonequilibrium systems in which inherent length-scale selection confines the dynamics near a mean energy hypersurface. Guided by spectral analysis of the field modes and scaling arguments, we derive a universal nonequilibrium distribution for kinetic field observables. We confirm the predicted energy distributions in experimental observations of Faraday surface waves, and in quantum chaos and active turbulence simulations. Our results indicate that pattern dynamics and transport in driven physical and biological matter can often be described through monochromatic random fields, suggesting a path towards a unified statistical field theory of nonequilibrium systems with length-scale selection.

Twisted bilayer graphene (TBG) near "magic angles" has emerged as a rich platform for strongly correlated states of two-dimensional Dirac semimetals. Here we show that twisted bilayers of thin-film magnetic topological insulators (MTI) with large in-plane magnetization can realize flat bands near 2D Dirac nodes. Using a simple model for thin films of MTIs, we derive a continuum model for two such MTIs, twisted by a small angle with respect to each other. When the magnetization is in-plane, we show that interlayer tunneling terms act as effective $SU(2)$ vector potentials, which are known to lead to flat bands in TBG. We show that by changing the in-plane magnetization, it is possible to tune the twisted bilayer MTI band dispersion to quadratic band touching or to flat bands, similar to the TBG. If realized, this system can be a highly tunable platform for strongly correlated phases of two-dimensional Dirac semimetals., Comment: Version accepted in Phys Rev Research

As the only semimetallic $d^{10}$-based delafossite, AgNiO$_2$ has received a great deal of attention due to both its unique semimetallicity and its antiferromagnetism in the NiO$_2$ layer that is coupled with a lattice distortion. In contrast, other delafossites such as AgCoO$_2$ are insulating. Here we study how the electronic structure of AgNi$_{1-x}$Co$_{x}$O$_2$ alloys vary with Ni/Co concentration, in order to investigate the electronic properties and phase stability of the intermetallics. While the electronic and magnetic structure of delafossites have been studied using Density Functional Theory (DFT), earlier studies have not included corrections for strong on-site Coulomb interactions. In order to treat these interactions accurately, in this study we use Quantum Monte Carlo (QMC) simulations to obtain accurate estimates for the electronic and magnetic properties of AgNiO$_2$. By comparison to DFT results we show that these electron correlations are critical to account for. We show that Co doping on the magnetic Ni sites results in a metal-insulator transition near $x\sim 0.33$, and reentrant behavior near $x\sim 0.66$, Comment: 25 Pages, 12 figures

Accurately predicting the formation energy of a compound, which describes its thermodynamic stability, is a key challenge in materials physics. Here, we employ many-body quantum Monte Carlo (QMC) with single-reference trial functions to compute the formation energy of two electronically disparate compounds, the intermetallic VPt$_2$ and the semiconductor CuI, for which standard density functional theory (DFT) predictions using both the Perdew-Burke Ernzerhof (PBE) and the strongly constrained and appropriately normed (SCAN) density functional approximations deviate markedly from available experimental values. For VPt$_2$, we find an agreement between QMC, SCAN, and PBE0 estimates, which therefore remain in disagreement with the much less exothermic experimental value. For CuI, the QMC result agrees with neither SCAN nor PBE pointing towards DFT exchange-correlation biases, likely related to the localized Cu $3d$ electrons. Compared to the behavior of some density functional approximations within DFT, spin-averaged QMC exhibits a smaller but still appreciable deviation when compared to experiment. The QMC result is slightly improved by incorporating spin-orbit corrections for CuI and solid I$_2$, so that experiment and theory are brought into imperfect but reasonable agreement within about 120~meV/atom.

We perform classical Monte Carlo and stochastic Landau-Lifshitz-Gilbert simulations to study temperature dependent magnetism of Kagome antiferromagnet (AFM) Weyl metal $\text{Mn}_3\text{Ge}$ and find that a long range chiral order sets in at a transition temperature well below the N{\'e}el temperature ($T_N$). Based on the crystalline symmetries, imposed by the chiral magnetic order, we argue for the presence of multiple iso-energetic Weyl nodes (nodes that are at same energy and with congruent Fermi surface around them) near chemical potential. Using the semi-classical Boltzmann equations, we show that the combined contribution to the net longitudinal magnetoconductance (LMC) and the planar Hall conductance (PHC) from tilted Weyl nodes can lead to signatures, qualitatively distinct from that of a single pair of Weyl nodes. In particular, we show that magnetic orders with different chiralities can give rise to different periods in LMC and PHC as a function of the in-plane magnetic field direction. This is ultimately related to differences in the symmetry-imposed constraints on the Weyl nodes.

The first magnetic 2D material discovered, monolayer (ML) CrI$_3$, is particularly fascinating due to its ground state ferromagnetism. Yet, because monolayer materials are difficult to probe experimentally, much remains unresolved about ML CrI$_{3}$'s structural, electronic, and magnetic properties. Here, we leverage Density Functional Theory (DFT) and high-accuracy Diffusion Monte Carlo (DMC) simulations to predict lattice parameters, magnetic moments, and spin-phonon and spin-lattice coupling of ML CrI$_{3}$. We exploit a recently developed surrogate Hessian DMC line search technique to determine CrI$_{3}$'s monolayer geometry with DMC accuracy, yielding lattice parameters in good agreement with recently-published STM measurements - an accomplishment given the $\sim 10$% variability in previous DFT-derived estimates depending upon the functional. Strikingly, we find previous DFT predictions of ML CrI$_3$'s magnetic spin moments are correct on average across a unit cell, but miss critical local spatial fluctuations in the spin density revealed by more accurate DMC. DMC predicts magnetic moments in ML CrI$_3$ are 3.62 $\mu_B$ per chromium and -0.145 $\mu_B$ per iodine; both larger than previous DFT predictions. The large disparate moments together with the large spin-orbit coupling of CrI$_3$'s I-$\textit{p}$ orbital suggests a ligand superexchange-dominated magnetic anisotropy in ML CrI$_3$, corroborating recent observations of magnons in its 2D limit. We also find ML CrI$_3$ exhibits a substantial spin-phonon coupling of $\sim$3.32 cm$^{-1}$. Our work thus establishes many of ML CrI$_{3}$'s key properties, while also continuing to demonstrate the pivotal role DMC can assume in the study of magnetic and other 2D materials., Comment: 16 pages, 13 figures

Approximate Bayesian inference estimates descriptors of an intractable target distribution - in essence, an optimization problem within a family of distributions. For example, Langevin dynamics (LD) extracts asymptotically exact samples from a diffusion process because the time evolution of its marginal distributions constitutes a curve that minimizes the KL-divergence via steepest descent in the Wasserstein space. Parallel to LD, Stein variational gradient descent (SVGD) similarly minimizes the KL, albeit endowed with a novel Stein-Wasserstein distance, by deterministically transporting a set of particle samples, thus de-randomizes the stochastic diffusion process. We propose de-randomized kernel-based particle samplers to all diffusion-based samplers known as MCMC dynamics. Following previous work in interpreting MCMC dynamics, we equip the Stein-Wasserstein space with a fiber-Riemannian Poisson structure, with the capacity of characterizing a fiber-gradient Hamiltonian flow that simulates MCMC dynamics. Such dynamics discretizes into generalized SVGD (GSVGD), a Stein-type deterministic particle sampler, with particle updates coinciding with applying the diffusion Stein operator to a kernel function. We demonstrate empirically that GSVGD can de-randomize complex MCMC dynamics, which combine the advantages of auxiliary momentum variables and Riemannian structure, while maintaining the high sample quality from an interacting particle system., Comment: 22 pages, 6 figures. NeurIPS 2021

Inspired by the observation of the extremely large anomalous Hall effect in the absence of applied magnetic fields or uniform magnetization in CoNb$_3$S$_6$ [Nature Comm. 9, 3280 (2018); Phys. Rev. Research 2, 023051 (2020)], we perform a first-principles study of this and related compounds of the $M$Nb$_3$S$_6$ type with different transition metal $M$ ions to determine their magnetic orders and the anomalous Hall conductivity (AHC). We find that non-coplanar antiferromagnetic ordering is favored relative to collinear or coplanar order in the case of $M$=Co, Fe and Ni, while ferromagnetic ordering is favored in MnNb$_3$S$_6$ at low temperatures. The AHC in these materials with non-coplanar spin ordering can reach about $e^2/h$ per crystalline layer, while being negligible for coplanar and collinear cases. We also find that the AHC depends sensitively on doping and reaches a maximum for intermediate values of the local spin exchange potential between 0.3 and 0.8 eV. Our AHC results are consistent with the reported Hall measurements in CoNb$_3$S$_6$ and suggest a possibility of similarly large anomalous Hall effects in related compounds., Comment: 9 pages, 6 figures