Search

Robustness is essential for deep neural networks, especially in security-sensitive applications. To this end, randomized smoothing provides theoretical guarantees for certifying robustness against adversarial perturbations. Recently, diffusion models have been successfully employed for randomized smoothing to purify noise-perturbed samples before making predictions with a standard classifier. While these methods excel at small perturbation radii, they struggle with larger perturbations and incur a significant computational overhead during inference compared to classical methods. To address this, we reformulate the generative modeling task along the diffusion trajectories in pixel space as a discriminative task in the latent space. Specifically, we use instance discrimination to achieve consistent representations along the trajectories by aligning temporally adjacent points. After fine-tuning based on the learned representations, our model enables implicit denoising-then-classification via a single prediction, substantially reducing inference costs. We conduct extensive experiments on various datasets and achieve state-of-the-art performance with minimal computation budget during inference. For example, our method outperforms the certified accuracy of diffusion-based methods on ImageNet across all perturbation radii by 5.3% on average, with up to 11.6% at larger radii, while reducing inference costs by 85$\times$ on average. Codes are available at: https://github.com/jiachenlei/rRCM.

We study the problem of training neural stochastic differential equations, or diffusion models, to sample from a Boltzmann distribution without access to target samples. Existing methods for training such models enforce time-reversal of the generative and noising processes, using either differentiable simulation or off-policy reinforcement learning (RL). We prove equivalences between families of objectives in the limit of infinitesimal discretization steps, linking entropic RL methods (GFlowNets) with continuous-time objects (partial differential equations and path space measures). We further show that an appropriate choice of coarse time discretization during training allows greatly improved sample efficiency and the use of time-local objectives, achieving competitive performance on standard sampling benchmarks with reduced computational cost., Comment: code: https://github.com/GFNOrg/gfn-diffusion/tree/stagger

The suite of datasets commonly used to train and evaluate the mathematical capabilities of AI-based mathematical copilots (primarily large language models) exhibit several shortcomings. These limitations include a restricted scope of mathematical complexity, typically not exceeding lower undergraduate-level mathematics, binary rating protocols and other issues, which makes comprehensive proof-based evaluation suites difficult. We systematically explore these limitations and contend that enhancing the capabilities of large language models, or any forthcoming advancements in AI-based mathematical assistants (copilots or "thought partners"), necessitates a paradigm shift in the design of mathematical datasets and the evaluation criteria of mathematical ability: It is necessary to move away from result-based datasets (theorem statement to theorem proof) and convert the rich facets of mathematical research practice to data LLMs can train on. Examples of these are mathematical workflows (sequences of atomic, potentially subfield-dependent tasks that are often performed when creating new mathematics), which are an important part of the proof-discovery process. Additionally, we advocate for mathematical dataset developers to consider the concept of "motivated proof", introduced by G. P\'olya in 1949, which can serve as a blueprint for datasets that offer a better proof learning signal, alleviating some of the mentioned limitations. Lastly, we introduce math datasheets for datasets, extending the general, dataset-agnostic variants of datasheets: We provide a questionnaire designed specifically for math datasets that we urge dataset creators to include with their datasets. This will make creators aware of potential limitations of their datasets while at the same time making it easy for readers to assess it from the point of view of training and evaluating mathematical copilots., Comment: 40 pages

We present NeuralOperator, an open-source Python library for operator learning. Neural operators generalize neural networks to maps between function spaces instead of finite-dimensional Euclidean spaces. They can be trained and inferenced on input and output functions given at various discretizations, satisfying a discretization convergence properties. Built on top of PyTorch, NeuralOperator provides all the tools for training and deploying neural operator models, as well as developing new ones, in a high-quality, tested, open-source package. It combines cutting-edge models and customizability with a gentle learning curve and simple user interface for newcomers.

An effective approach for sampling from unnormalized densities is based on the idea of gradually transporting samples from an easy prior to the complicated target distribution. Two popular methods are (1) Sequential Monte Carlo (SMC), where the transport is performed through successive annealed densities via prescribed Markov chains and resampling steps, and (2) recently developed diffusion-based sampling methods, where a learned dynamical transport is used. Despite the common goal, both approaches have different, often complementary, advantages and drawbacks. The resampling steps in SMC allow focusing on promising regions of the space, often leading to robust performance. While the algorithm enjoys asymptotic guarantees, the lack of flexible, learnable transitions can lead to slow convergence. On the other hand, diffusion-based samplers are learned and can potentially better adapt themselves to the target at hand, yet often suffer from training instabilities. In this work, we present a principled framework for combining SMC with diffusion-based samplers by viewing both methods in continuous time and considering measures on path space. This culminates in the new Sequential Controlled Langevin Diffusion (SCLD) sampling method, which is able to utilize the benefits of both methods and reaches improved performance on multiple benchmark problems, in many cases using only 10% of the training budget of previous diffusion-based samplers.

In this paper we study 2D Fourier expansions for a general class of planar measures $\mu$, generally singular, but assumed compactly supported in $\mathbb{R}^2$. We focus on the following question: When does $L^2(\mu)$ admit a 2D system of Fourier expansions? We offer concrete conditions allowing an affirmative answer to the question for a large class of Borel probability measures, and we present an explicit Fourier duality for these cases. Our 2D Fourier analysis relies on a detailed conditioning-analysis. For a given $\mu$, it is based on the corresponding systems of 1D measures consisting of a marginal measure and associated family of conditional measures computed from $\mu$ by the Rokhlin Disintegration Theorem. Our identified $L^2(\mu)$-Fourier expansions are special in two ways: For our measures $\mu$, the Fourier expansions are generally non-orthogonal, but nonetheless, they lend themselves to algorithmic computations. Second, we further stress that our class of 2D measures $\mu$ considered here go beyond what exists in the literature. In particular, our measures do not require affine iterated function system (IFS) properties, but we do study grid IFS measures in detail and provide some technical criteria guaranteeing their admission of Fourier expansions. Our analyses make use of estimates for the Hausdorff dimensions of the measure supports. An important class of examples addressed in this paper is fractal Bedford-McMullen carpets.

Recent advancements in artificial intelligence (AI) for numerical weather prediction (NWP) have significantly transformed atmospheric modeling. AI NWP models outperform traditional physics-based systems, such as the Integrated Forecast System (IFS), across several global metrics while requiring fewer computational resources. However, existing AI NWP models face limitations related to training datasets and timestep choices, often resulting in artifacts that reduce model performance. To address these challenges, we introduce the Community Research Earth Digital Intelligence Twin (CREDIT) framework, developed at NSF NCAR. CREDIT provides a flexible, scalable, and user-friendly platform for training and deploying AI-based atmospheric models on high-performance computing systems. It offers an end-to-end pipeline for data preprocessing, model training, and evaluation, democratizing access to advanced AI NWP capabilities. We demonstrate CREDIT's potential through WXFormer, a novel deterministic vision transformer designed to predict atmospheric states autoregressively, addressing common AI NWP issues like compounding error growth with techniques such as spectral normalization, padding, and multi-step training. Additionally, to illustrate CREDIT's flexibility and state-of-the-art model comparisons, we train the FUXI architecture within this framework. Our findings show that both FUXI and WXFormer, trained on six-hourly ERA5 hybrid sigma-pressure levels, generally outperform IFS HRES in 10-day forecasts, offering potential improvements in efficiency and forecast accuracy. CREDIT's modular design enables researchers to explore various models, datasets, and training configurations, fostering innovation within the scientific community.

Cosmological neutral hydrogen (HI) surveys provide a promising tomographic probe of the post-reionization era and of the standard model of cosmology. Simulations of this signal are crucial for maximizing the utility of these surveys. We present a fast method for simulating the cosmological distribution of HI based on a halo model approach. Employing the approximate $\texttt{PINOCCHIO}$ code, we generate the past light cone of dark matter halos. Subsequently, the halos are populated with HI according to a HI-halo mass relation. The nature of 21 cm intensity mapping demands large-volume simulations with a high halo mass resolution. To fulfill both requirements, we simulate a past light cone for declinations between -15{\deg} and -35{\deg} in the frequency range from 700 to 800 MHz, matching HIRAX, the Hydrogen Intensity and Real-time Analysis eXperiment. We run $\texttt{PINOCCHIO}$ for a 1 h$^{-3}$Gpc$^3$ box with 6700$^3$ simulation particles. With this configuration, halos with masses as low as M$_\text{min}$ = 4.3 $\times$ 10$^{9}$M$_{\odot}$ are simulated, resulting in the recovery of more than 97% of the expected HI density. From the dark matter and HI past light cone, maps with a width of 5 MHz are created. To validate the simulations, we have implemented and present here an analytical dark matter and HI halo model in $\texttt{PyCosmo}$, a Python package tailored for theoretical cosmological predictions. We perform extensive comparisons between analytical predictions and the simulations for the mass function, mass density, power spectrum, and angular power spectrum for dark matter and HI. We find close agreement in the mass function and mass densities, with discrepancies within a few percent. For the three-dimensional power spectra and angular power spectra, we observe an agreement better than 10%., Comment: Submitted to JCAP. 29 pages, 10 figures, 1 table. The PyCosmo package is available on https://cosmology.ethz.ch/research/software-lab/PyCosmo.html and can be installed from the Python Package Index at https://pypi.org. The simulation dataset will be made available at the time of publication

Frames in a Hilbert space that are generated by operator orbits are vastly studied because of the applications in dynamic sampling and signal recovery. We demonstrate in this paper a representation theory for frames generated by operator orbits that provides explicit constructions of the frame and the operator when the operators are not surjective. It is known that the Kaczmarz algorithm for stationary sequences in Hilbert spaces generates a frame that arises from an operator orbit where the operator is not surjective. In this paper, we show that every frame generated by a not surjective operator in any Hilbert space arises from the Kaczmarz algorithm. Furthermore, we show that the operators generating these frames are similar to rank one perturbations of unitary operators. After this, we describe a large class of operator orbit frames that arise from Fourier expansions for singular measures. Moreover, we classify all measures that possess frame-like Fourier expansions arising from two-sided operator orbit frames. Finally, we show that measures that possess frame-like Fourier expansions arising from two-sided operator orbits are weighted Lebesgue measure with weight satisfying a weak $A_{2}$ condition, even in the non-frame case. We also use these results to classify measures with other types of frame-like Fourier expansions.

Fourier Neural Operators (FNOs) excel on tasks using functional data, such as those originating from partial differential equations. Such characteristics render them an effective approach for simulating the time evolution of quantum wavefunctions, which is a computationally challenging, yet coveted task for understanding quantum systems. In this manuscript, we use FNOs to model the evolution of random quantum spin systems, so chosen due to their representative quantum dynamics and minimal symmetry. We explore two distinct FNO architectures and examine their performance for learning and predicting time evolution using both random and low-energy input states. Additionally, we apply FNOs to a compact set of Hamiltonian observables ($\sim\text{poly}(n)$) instead of the entire $2^n$ quantum wavefunction, which greatly reduces the size of our inputs and outputs and, consequently, the requisite dimensions of the resulting FNOs. Moreover, this Hamiltonian observable-based method demonstrates that FNOs can effectively distill information from high-dimensional spaces into lower-dimensional spaces. The extrapolation of Hamiltonian observables to times later than those used in training is of particular interest, as this stands to fundamentally increase the simulatability of quantum systems past both the coherence times of contemporary quantum architectures and the circuit-depths of tractable tensor networks., Comment: 9 pages, 4 figures

Accurately predicting the long-term behavior of chaotic systems is crucial for various applications such as climate modeling. However, achieving such predictions typically requires iterative computations over a dense spatiotemporal grid to account for the unstable nature of chaotic systems, which is expensive and impractical in many real-world situations. An alternative approach to such a full-resolved simulation is using a coarse grid and then correcting its errors through a \textit{closure model}, which approximates the overall information from fine scales not captured in the coarse-grid simulation. Recently, ML approaches have been used for closure modeling, but they typically require a large number of training samples from expensive fully-resolved simulations (FRS). In this work, we prove an even more fundamental limitation, i.e., the standard approach to learning closure models suffers from a large approximation error for generic problems, no matter how large the model is, and it stems from the non-uniqueness of the mapping. We propose an alternative end-to-end learning approach using a physics-informed neural operator (PINO) that overcomes this limitation by not using a closure model or a coarse-grid solver. We first train the PINO model on data from a coarse-grid solver and then fine-tune it with (a small amount of) FRS and physics-based losses on a fine grid. The discretization-free nature of neural operators means that they do not suffer from the restriction of a coarse grid that closure models face, and they can provably approximate the long-term statistics of chaotic systems. In our experiments, our PINO model achieves a 330x speedup compared to FRS with a relative error $\sim 10\%$. In contrast, the closure model coupled with a coarse-grid solver is $60$x slower than PINO while having a much higher error $\sim186\%$ when the closure model is trained on the same FRS dataset.

Canard cascading (CC) is observed in dynamical networks with global adaptive coupling. It is a fast-slow phenomenon characterized by a recurrent sequence of fast transitions between distinct and slowly evolving quasi-stationary states. In this letter, we uncover the dynamical mechanisms behind CC, using an illustrative example of globally and adaptively coupled semiconductor lasers, where CC represents sequential switching on and off the lasers. Firstly, we show that CC is a robust and truly adaptive network effect that is scalable with network size and does not occur without adaptation. Secondly, we uncover multiple saddle slow manifolds (unstable quasi-stationary states) linked by heteroclinic orbits (fast transitions) in the phase space of the system. This allows us to identify CC with a novel heteroclinic canard orbit that organises different unstable quasi-stationary states into an intricate fast-slow limit cycle. Although individual quasi-stationary states are unstable (saddles), the CC cycle as a whole is attractive and robust to parameter changes.

Assessing and improving the robustness of a graph $G$ are critical steps in network design and analysis. To this end, we consider the optimisation problem of removing $k$ edges from $G$ such that the resulting graph has minimal robustness, simulating attacks or failures. In this paper, we propose total harmonic resistance as a new robustness measure for this purpose - and compare it to the recently proposed forest index [Zhu et al., IEEE Trans.\ Inf.\ Forensics and Security, 2023]. Both measures are related to the established total effective resistance measure, but their advantage is that they can handle disconnected graphs. This is also important for originally connected graphs due to the removal of the $k$ edges. To compare our measure with the forest index, we first investigate exact solutions for small examples. The best $k$ edges to select when optimizing for the forest index lie at the periphery. Our proposed measure, in turn, prioritizes more central edges, which should be beneficial for most applications. Furthermore, we adapt a generic greedy algorithm to our optimization problem with the total harmonic resistance. With this algorithm, we perform a case study on the Berlin road network and also apply the algorithm to established benchmark graphs. The results are similar as for the small example graphs above and indicate the higher suitability of the new measure., Comment: accepted at the research track of ECML PKDD 2024

The task of sampling from a probability density can be approached as transporting a tractable density function to the target, known as dynamical measure transport. In this work, we tackle it through a principled unified framework using deterministic or stochastic evolutions described by partial differential equations (PDEs). This framework incorporates prior trajectory-based sampling methods, such as diffusion models or Schr\"odinger bridges, without relying on the concept of time-reversals. Moreover, it allows us to propose novel numerical methods for solving the transport task and thus sampling from complicated targets without the need for the normalization constant or data samples. We employ physics-informed neural networks (PINNs) to approximate the respective PDE solutions, implying both conceptional and computational advantages. In particular, PINNs allow for simulation- and discretization-free optimization and can be trained very efficiently, leading to significantly better mode coverage in the sampling task compared to alternative methods. Moreover, they can readily be fine-tuned with Gauss-Newton methods to achieve high accuracy in sampling.

We propose accelerating the simulation of Lagrangian dynamics, such as fluid flows, granular flows, and elastoplasticity, with neural-operator-based reduced-order modeling. While full-order approaches simulate the physics of every particle within the system, incurring high computation time for dense inputs, we propose to simulate the physics on sparse graphs constructed by sampling from the spatially discretized system. Our discretization-invariant reduced-order framework trains on any spatial discretizations and computes temporal dynamics on any sparse sampling of these discretizations through neural operators. Our proposed approach is termed Graph Informed Optimized Reduced-Order Modeling or \textit{GIOROM}. Through reduced order modeling, we ensure lower computation time by sparsifying the system by 6.6-32.0$\times$, while ensuring high-fidelity full-order inference via neural fields. We show that our model generalizes to a range of initial conditions, resolutions, and materials. The code and the demos are provided at \url{https://github.com/HrishikeshVish/GIOROM}

Diffusion models have recently achieved success in solving Bayesian inverse problems with learned data priors. Current methods build on top of the diffusion sampling process, where each denoising step makes small modifications to samples from the previous step. However, this process struggles to correct errors from earlier sampling steps, leading to worse performance in complicated nonlinear inverse problems, such as phase retrieval. To address this challenge, we propose a new method called Decoupled Annealing Posterior Sampling (DAPS) that relies on a novel noise annealing process. Specifically, we decouple consecutive steps in a diffusion sampling trajectory, allowing them to vary considerably from one another while ensuring their time-marginals anneal to the true posterior as we reduce noise levels. This approach enables the exploration of a larger solution space, improving the success rate for accurate reconstructions. We demonstrate that DAPS significantly improves sample quality and stability across multiple image restoration tasks, particularly in complicated nonlinear inverse problems.

We propose Neural Walk-on-Spheres (NWoS), a novel neural PDE solver for the efficient solution of high-dimensional Poisson equations. Leveraging stochastic representations and Walk-on-Spheres methods, we develop novel losses for neural networks based on the recursive solution of Poisson equations on spheres inside the domain. The resulting method is highly parallelizable and does not require spatial gradients for the loss. We provide a comprehensive comparison against competing methods based on PINNs, the Deep Ritz method, and (backward) stochastic differential equations. In several challenging, high-dimensional numerical examples, we demonstrate the superiority of NWoS in accuracy, speed, and computational costs. Compared to commonly used PINNs, our approach can reduce memory usage and errors by orders of magnitude. Furthermore, we apply NWoS to problems in PDE-constrained optimization and molecular dynamics to show its efficiency in practical applications., Comment: Accepted at ICML 2024

Existing neural operator architectures face challenges when solving multiphysics problems with coupled partial differential equations (PDEs) due to complex geometries, interactions between physical variables, and the limited amounts of high-resolution training data. To address these issues, we propose Codomain Attention Neural Operator (CoDA-NO), which tokenizes functions along the codomain or channel space, enabling self-supervised learning or pretraining of multiple PDE systems. Specifically, we extend positional encoding, self-attention, and normalization layers to function spaces. CoDA-NO can learn representations of different PDE systems with a single model. We evaluate CoDA-NO's potential as a backbone for learning multiphysics PDEs over multiple systems by considering few-shot learning settings. On complex downstream tasks with limited data, such as fluid flow simulations, fluid-structure interactions, and Rayleigh-B\'enard convection, we found CoDA-NO to outperform existing methods by over 36%.

Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at \url{https://github.com/thu-ml/DPOT}.

Neural operators learn mappings between function spaces, which is practical for learning solution operators of PDEs and other scientific modeling applications. Among them, the Fourier neural operator (FNO) is a popular architecture that performs global convolutions in the Fourier space. However, such global operations are often prone to over-smoothing and may fail to capture local details. In contrast, convolutional neural networks (CNN) can capture local features but are limited to training and inference at a single resolution. In this work, we present a principled approach to operator learning that can capture local features under two frameworks by learning differential operators and integral operators with locally supported kernels. Specifically, inspired by stencil methods, we prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions. Both these approaches preserve the properties of operator learning and, hence, the ability to predict at any resolution. Adding our layers to FNOs significantly improves their performance, reducing the relative L2-error by 34-72% in our experiments, which include a turbulent 2D Navier-Stokes and the spherical shallow water equations., Comment: Accepted at 2024 International Conference on Machine Learning

For multi-variable finite measure spaces, we present in this paper a new framework for non-orthogonal $L^2$ Fourier expansions. Our results hold for probability measures $\mu$ with finite support in $\mathbb{R}^d$ that satisfy a certain disintegration condition that we refer to as ``slice-singular''. In this general framework, we present explicit $L^{2}(\mu)$-Fourier expansions, with Fourier exponentials having positive Fourier frequencies in each of the d coordinates. Our Fourier representations apply to every $f \in L^2(\mu)$, are based on an extended Kaczmarz algorithm, and use a new recursive $\mu$ Rokhlin disintegration representation. In detail, our Fourier series expansion for $f$ is in terms of the multivariate Fourier exponentials $\{e_n\}$, but the associated Fourier coefficients for $f$ are now computed from a Kaczmarz system $\{g_n\}$ in $L^{2}(\mu)$ which is dual to the Fourier exponentials. The $\{g_n\}$ system is shown to be a Parseval frame for $L^{2}(\mu)$. Explicit computations for our new Fourier expansions entail a detailed analysis of subspaces of the Hardy space on the polydisk, dual to $L^{2}(\mu)$, and an associated d-variable Normalized Cauchy Transform. Our results extend earlier work for measures $\mu$ in one and two dimensions, i.e., $d=1 (\mu $ singular), and $d=2 (\mu$ assumed slice-singular). Here our focus is the extension to the cases of measures $\mu$ in dimensions $d >2$. Our results are illustrated with the use of explicit iterated function systems (IFSs), including the IFS generated Menger sponge for $d=3$.

Phase transitions are crucial in shaping the collective dynamics of a broad spectrum of natural systems across disciplines. Here, we report two distinct heterogeneous nucleation facilitating single-step and multi-step phase transitions to global synchronization in a finite-size adaptive network due to the trade-off between time scale adaptation and coupling strength disparities. Specifically, small intracluster nucleations coalesce either at the population interface or within the populations resulting in the two distinct phase transitions depending on the degree of the disparities. We find that the coupling strength disparity largely controls the nature of phase transition in the phase diagram irrespective of the adaptation disparity. We provide a mesoscopic description for the cluster dynamics using the collective coordinates approach that brilliantly captures the multicluster dynamics among the populations leading to distinct phase transitions. Further, we also deduce the upper bound for the coupling strength for the existence of two intraclusters explicitly in terms of adaptation and coupling strength disparities. These insights may have implications across domains ranging from neurological disorders to segregation dynamics in social networks.

Partially synchronized solitary states occur frequently when a synchronized system of networked oscillators is perturbed locally. Several asymptotic states of different frequencies can coexist at the same node. Here, we reveal the mechanism behind this multistability: additional solitary frequencies arise from the coupling between network modes and the solitary oscillator's frequency, leading to significant energy transfer. This can cause the solitary node's frequency to resonate with a Laplacian eigenvalue. We analyze which network structures enable this resonance and explain longstanding numerical observations. Another solitary state is characterized by the effective decoupling of the synchronized network and the solitary node at the natural frequency. Our framework unifies the description of solitary states near and far from resonance, allowing to predict the behavior of complex networks.