Search

We present a natural framework for constructing energy-stable time discretization schemes. By leveraging the Onsager principle, we demonstrate its efficacy in formulating partial differential equation models for diverse gradient flow systems. Furthermore, this principle provides a robust basis for developing numerical schemes that uphold crucial physical properties. Within this framework, several widely used schemes emerge naturally, showing its versatility and applicability.

This paper is concerned with structure-preserving numerical approximations for a class of nonlinear nonlocal Fokker-Planck equations, which admit a gradient flow structure and find application in diverse contexts. The solutions, representing density distributions, must be non-negative and satisfy a specific energy dissipation law. We design an arbitrary high-order discontinuous Galerkin (DG) method tailored for these model problems. Both semi-discrete and fully discrete schemes are shown to admit the energy dissipation law for non-negative numerical solutions. To ensure the preservation of positivity in cell averages at all time steps, we introduce a local flux correction applied to the DDG diffusive flux. Subsequently, a hybrid algorithm is presented, utilizing a positivity-preserving limiter, to generate positive and energy-dissipating solutions. Numerical examples are provided to showcase the high resolution of the numerical solutions and the verified properties of the DG schemes.

In this paper, we present several unconditionally energy-stable invariant energy quadratization (IEQ) finite element methods (FEMs) with linear, first- and second-order accuracy for solving both the Cahn-Hilliard equation and the Allen-Cahn equation. For time discretization, we compare three distinct IEQ-FEM schemes that position the intermediate function introduced by the IEQ approach in different function spaces: finite element space, continuous function space, or a combination of these spaces. Rigorous proofs establishing the existence and uniqueness of the numerical solution, along with analyses of energy dissipation for both equations and mass conservation for the Cahn-Hilliard equation, are provided. The proposed schemes' accuracy, efficiency, and solution properties are demonstrated through numerical experiments., Comment: 33 pages, 15 figures

Solving high dimensional partial differential equations (PDEs) has historically posed a considerable challenge when utilizing conventional numerical methods, such as those involving domain meshes. Recent advancements in the field have seen the emergence of neural PDE solvers, leveraging deep networks to effectively tackle high dimensional PDE problems. This study introduces Inf-SupNet, a model-based unsupervised learning approach designed to acquire solutions for a specific category of elliptic PDEs. The fundamental concept behind Inf-SupNet involves incorporating the inf-sup formulation of the underlying PDE into the loss function. The analysis reveals that the global solution error can be bounded by the sum of three distinct errors: the numerical integration error, the duality gap of the loss function (training error), and the neural network approximation error for functions within Sobolev spaces. To validate the efficacy of the proposed method, numerical experiments conducted in high dimensions demonstrate its stability and accuracy across various boundary conditions, as well as for both semi-linear and nonlinear PDEs.

Extracting physical laws from observation data is a central challenge in many diverse areas of science and engineering. We propose Optimal Control Neural Networks (OCN) to learn the laws of vector fields in dynamical systems, with no assumption on their analytical form, given data consisting of sampled trajectories. The OCN framework consists of a neural network representation and an optimal control formulation. We provide error bounds for both the solution and the vector field. The bounds are shown to depend on both the training error and the time step between the observation data. We also demonstrate the effectiveness of OCN, as well as its generalization ability, by testing on several canonical systems, including the chaotic Lorenz system., Comment: 28 pages, 8 figures

Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area., Comment: Accepted by NeurIPS 2023

We propose an adaptive step size with an energy approach for a suitable class of preconditioned gradient descent methods. We focus on settings where the preconditioning is applied to address the constraints in optimization problems, such as the Hessian-Riemannian and natural gradient descent methods. More specifically, we incorporate these preconditioned gradient descent algorithms in the recently introduced Adaptive Energy Gradient Descent (AEGD) framework. In particular, we discuss theoretical results on the unconditional energy-stability and convergence rates across three classes of objective functions. Furthermore, our numerical results demonstrate excellent performance of the proposed method on several test bed optimization problems., Comment: 32 pages, 3 figures

We review the theoretical development in the study of critical thresholds for hyperbolic balance laws. The emphasis is on two classes of systems: Euler-Poisson-alignment (EPA) systems and hyperbolic relaxation systems. We start with an introduction to the `Critical Threshold Phenomena' and study some nonlocal PDE systems, which are important from modeling point of view., Comment: 10 pages, 2 figures

The Euler-Poisson system describes the dynamic behavior of many important physical flows including charge transport, plasma with collision and cosmological waves. We prove sharp threshold conditions for the global existence/finite-time-breakdown of solutions to the multidimensional pressureless Euler-Poisson (EP) system with or without background and general initial data. In particular, the initial data could include points where velocity is negative, that is, the flow is directed towards the origin. Obtaining threshold conditions for such systems is extremely hard due to the coupling of various local/nonlocal forces. Remarkably, we are able to achieve a sharp threshold for the zero background case and most importantly, the positive background case, which is quite delicate due to the oscillations present in the solutions. We discover a completely novel nonlinear quantity that helps to analyze the system. In the case of positive background, if the initial data results in a global-in-time solution, then we show that the density is periodic along any single characteristic path. We use the Floquet Theorem to prove periodicity., Comment: 48 pages, 7 figures

Anderson acceleration (AA) as an efficient technique for speeding up the convergence of fixed-point iterations may be designed for accelerating an optimization method. We propose a novel optimization algorithm by adapting Anderson acceleration to the energy adaptive gradient method (AEGD) [arXiv:2010.05109]. The feasibility of our algorithm is examined in light of convergence results for AEGD, though it is not a fixed-point iteration. We also quantify the accelerated convergence rate of AA for gradient descent by a factor of the gain at each implementation of the Anderson mixing. Our experimental results show that the proposed algorithm requires little tuning of hyperparameters and exhibits superior fast convergence., Comment: 18 pages, 4 figures

The state space for solutions of the compressible Euler equations with a general equation of state is examined. An arbitrary equation of state is allowed, subject only to the physical requirements of thermodynamics. An invariant region of the resulting Euler system is identified and the convexity property of this region is justified by using only very minimal thermodynamical assumptions. Finally, we show how an invariant-region-preserving (IRP) limiter can be constructed for use in high order finite-volume type schemes to solve the compressible Euler equations with a general constitutive relation.

The Kompaneets equation governs dynamics of the photon energy spectrum in certain high temperature (or low density) plasmas. We prove several results concerning the long-time convergence of solutions to Bose--Einstein equilibria and the failure of photon conservation. In particular, we show the total photon number can decrease with time via an outflux of photons at the zero-energy boundary. The ensuing accumulation of photons at zero energy is analogous to Bose--Einstein condensation. We provide two conditions that guarantee that photon loss occurs, and show that once loss is initiated then it persists forever. We prove that as $t\to \infty$, solutions necessarily converge to equilibrium and we characterize the limit in terms of the total photon loss. Additionally, we provide a few results concerning the behavior of the solution near the zero-energy boundary, an Oleinik inequality, a comparison principle, and show that the solution operator is a contraction in $L^1$. None of these results impose a boundary condition at the zero-energy boundary., Comment: 36 pages, 3 figures

In this paper, we propose SGEM, Stochastic Gradient with Energy and Momentum, to solve a large class of general non-convex stochastic optimization problems, based on the AEGD method that originated in the work [AEGD: Adaptive Gradient Descent with Energy. arXiv: 2010.05109]. SGEM incorporates both energy and momentum at the same time so as to inherit their dual advantages. We show that SGEM features an unconditional energy stability property, and derive energy-dependent convergence rates in the general nonconvex stochastic setting, as well as a regret bound in the online convex setting. A lower threshold for the energy variable is also provided. Our experimental results show that SGEM converges faster than AEGD and generalizes better or at least as well as SGDM in training some deep neural networks., Comment: 24 pages, 4 figures

This paper investigates a novel gradient algorithm, AGEM, using both energy and momentum, for addressing general non-convex optimization problems. The solution properties of the AGEM algorithm, including aspects such as uniformly boundedness and convergence to critical points, are examined. The dynamic behavior is studied through a comprehensive analysis of a high-resolution ODE system. This ODE system, being nonlinear, is derived by taking the limit of the discrete scheme while preserving the momentum effect through a rescaling of the momentum parameter. The paper emphasizes the global well-posedness of the ODE system and the time-asymptotic convergence of solution trajectories. Furthermore, we establish a linear convergence rate for objective functions that adhere to the Polyak-{\L}ojasiewicz condition., Comment: 30 pages, 2 figures

We introduce a novel algorithm for gradient-based optimization of stochastic objective functions. The method may be seen as a variant of SGD with momentum equipped with an adaptive learning rate automatically adjusted by an 'energy' variable. The method is simple to implement, computationally efficient, and well suited for large-scale machine learning problems. The method exhibits unconditional energy stability for any size of the base learning rate. We provide a regret bound on the convergence rate under the online convex optimization framework. We also establish the energy-dependent convergence rate of the algorithm to a stationary point in the stochastic non-convex setting. In addition, a sufficient condition is provided to guarantee a positive lower threshold for the energy variable. Our experiments demonstrate that the algorithm converges fast while generalizing better than or as well as SGD with momentum in training deep neural networks, and compares also favorably to Adam., Comment: 29 pages, 6 figures

This paper is concerned with the global wellposedness of the Euler-Poisson-alignment (EPA) system. This system arises from collective dynamics, and features two types of nonlocal interactions: the repulsive electric force and the alignment force. It is known that the repulsive electric force generates oscillatory solutions, which is difficult to be controlled by the nonlocal alignment force using conventional comparison principles. We construct \emph{invariant regions} such that the solution trajectories cannot exit, and therefore obtain global wellposedness for subcritical initial data that lie in the invariant regions. Supercritical regions of initial data are also derived which leads to finite-time singularity formations. To handle the oscillation and the nonlocality, we introduce a new way to construct invariant regions piece by piece in the phase plane of a reformulation of the EPA system. Our result is extended to the case when the alignment force is weakly singular. The singularity leads to the loss of a priori bounds crucial in our analysis. With the help of improved estimates on the nonlocal quantities, we design non-trivial invariant regions that guarantee global wellposedness of the EPA system with weakly singular alignment interactions., Comment: 33 pages, 11 figures

This letter studies the Euler-alignment system with weakly singular influence functions by introducing a novel technique to bound the density. Instead of resorting to a nonlinear maximum principle used in [C. Tan, Nonlinearity, 33: 1907--1924, 2020] to bound the interaction term $\psi\ast\rho$ by $\rho^s$ with $s\in (0, 1)$ for $\psi$ with an algebraic singularity at origin, we bound $\psi*\rho$ by a relaxed constant for any $\psi \in L^1$. We thus establish the global-in-time existence results with weaker assumptions on $\psi$ and refined solution bounds, characterized by the structure of $\psi$., Comment: 8 pages

Implicit deep learning has received increasing attention recently due to the fact that it generalizes the recursive prediction rules of many commonly used neural network architectures. Its prediction rule is provided implicitly based on the solution of an equilibrium equation. Although a line of recent empirical studies has demonstrated its superior performances, the theoretical understanding of implicit neural networks is limited. In general, the equilibrium equation may not be well-posed during the training. As a result, there is no guarantee that a vanilla (stochastic) gradient descent (SGD) training nonlinear implicit neural networks can converge. This paper fills the gap by analyzing the gradient flow of Rectified Linear Unit (ReLU) activated implicit neural networks. For an $m$-width implicit neural network with ReLU activation and $n$ training samples, we show that a randomly initialized gradient descent converges to a global minimum at a linear rate for the square loss function if the implicit neural network is \textit{over-parameterized}. It is worth noting that, unlike existing works on the convergence of (S)GD on finite-layer over-parameterized neural networks, our convergence results hold for implicit neural networks, where the number of layers is \textit{infinite}., Comment: Accepted by ICLR 2022

In this work, we study the behavior of blow-up solutions to the multidimensional restricted Euler--Poisson equations which are the localized version of the full Euler--Poisson system. We provide necessary conditions for the existence of finite-time blow-up solutions in terms of the initial data, and describe the asymptotic behavior of the solutions near blow up times. We also identify a rich set of the initial data which yields global bounded solutions., Comment: 32 pages