Search

The pre-training cost of large language models (LLMs) is prohibitive. One cutting-edge approach to reduce the cost is zero-shot weight transfer, also known as model growth for some cases, which magically transfers the weights trained in a small model to a large model. However, there are still some theoretical mysteries behind the weight transfer. In this paper, inspired by prior applications of mean field theory to neural network dynamics, we introduce a mean field ansatz to provide a theoretical explanation for weight transfer. Specifically, we propose the row-column (RC) ansatz under the mean field point of view, which describes the measure structure of the weights in the neural network (NN) and admits a close measure dynamic. Thus, the weights of different sizes NN admit a common distribution under proper assumptions, and weight transfer methods can be viewed as sampling methods. We empirically validate the RC ansatz by exploring simple MLP examples and LLMs such as GPT-3 and Llama-3.1. We show the mean-field point of view is adequate under suitable assumptions which can provide theoretical support for zero-shot weight transfer., Comment: 40 pages, 6 Figures, 1 table

We study the weak convergence behaviour of the Leimkuhler--Matthews method, a non-Markovian Euler-type scheme with the same computational cost as the Euler scheme, for the approximation of the stationary distribution of a one-dimensional McKean--Vlasov Stochastic Differential Equation (MV-SDE). The particular class under study is known as mean-field (overdamped) Langevin equations (MFL). We provide weak and strong error results for the scheme in both finite and infinite time. We work under a strong convexity assumption. Based on a careful analysis of the variation processes and the Kolmogorov backward equation for the particle system associated with the MV-SDE, we show that the method attains a higher-order approximation accuracy in the long-time limit (of weak order convergence rate $3/2$) than the standard Euler method (of weak order $1$). While we use an interacting particle system (IPS) to approximate the MV-SDE, we show the convergence rate is independent of the dimension of the IPS and this includes establishing uniform-in-time decay estimates for moments of the IPS, the Kolmogorov backward equation and their derivatives. The theoretical findings are supported by numerical tests., Comment: 73 pages, 2 Figure

We propose a novel approach to numerically approximate McKean-Vlasov stochastic differential equations (MV-SDE) using stochastic gradient descent (SGD) while avoiding the use of interacting particle systems. The technique of SGD is deployed to solve a Euclidean minimization problem, which is obtained by first representing the MV-SDE as a minimization problem over the set of continuous functions of time, and then by approximating the domain with a finite-dimensional subspace. Convergence is established by proving certain intermediate stability and moment estimates of the relevant stochastic processes (including the tangent ones). Numerical experiments illustrate the competitive performance of our SGD based method compared to the IPS benchmarks. This work offers a theoretical foundation for using the SGD method in the context of numerical approximation of MV-SDEs, and provides analytical tools to study its stability and convergence., Comment: 25 pages, 6 figures

The complex Helmholtz equation $(\Delta + k^2)u=f$ (where $k\in{\mathbb R},u(\cdot),f(\cdot)\in{\mathbb C}$) is a mainstay of computational wave simulation. Despite its apparent simplicity, efficient numerical methods are challenging to design and, in some applications, regarded as an open problem. Two sources of difficulty are the large number of degrees of freedom and the indefiniteness of the matrices arising after discretisation. Seeking to meet them within the novel framework of probabilistic domain decomposition, we set out to rewrite the Helmholtz equation into a form amenable to the Feynman-Kac formula for elliptic boundary value problems. We consider two typical scenarios, the scattering of a plane wave and the propagation inside a cavity, and recast them as a sequence of Poisson equations. By means of stochastic arguments, we find a sufficient and simulatable condition for the convergence of the iterations. Upon discretisation a necessary condition for convergence can be derived by adding up the iterates using the harmonic series for the matrix inverse -- we illustrate the procedure in the case of finite differences. From a practical point of view, our results are ultimately of limited scope. Nonetheless, this unexpected -- even paradoxical -- new direction of attack on the Helmholtz equation proposed by this work offers a fresh perspective on this classical and difficult problem. Our results show that there indeed exists a predictable range $k

We study a class of McKean--Vlasov Stochastic Differential Equations (MV-SDEs) with drifts and diffusions having super-linear growth in measure and space -- the maps have general polynomial form but also satisfy a certain monotonicity condition. The combination of the drift's super-linear growth in measure (by way of a convolution) and the super-linear growth in space and measure of the diffusion coefficient requires novel technical elements in order to obtain the main results. We establish wellposedness, propagation of chaos (PoC), and under further assumptions on the model parameters, we show an exponential ergodicity property alongside the existence of an invariant distribution. No differentiability or non-degeneracy conditions are required. Further, we present a particle system based Euler-type split-step scheme (SSM) for the simulation of this type of MV-SDEs. The scheme attains, in stepsize, the strong error rate $1/2$ in the non-path-space root-mean-square error metric and we demonstrate the property of mean-square contraction. Our results are illustrated by numerical examples including: estimation of PoC rates across dimensions, preservation of periodic phase-space, and the observation that taming appears to be not a suitable method unless strong dissipativity is present., Comment: 42 pages, 5 figures, accepted final Author version (with corected typos; to appear in EJP)

In this paper, we introduce a new simple approach to developing and establishing the convergence of splitting methods for a large class of stochastic differential equations (SDEs), including additive, diagonal and scalar noise types. The central idea is to view the splitting method as a replacement of the driving signal of an SDE, namely Brownian motion and time, with a piecewise linear path that yields a sequence of ODEs $-$ which can be discretized to produce a numerical scheme. This new way of understanding splitting methods is inspired by, but does not use, rough path theory. We show that when the driving piecewise linear path matches certain iterated stochastic integrals of Brownian motion, then a high order splitting method can be obtained. We propose a general proof methodology for establishing the strong convergence of these approximations that is akin to the general framework of Milstein and Tretyakov. That is, once local error estimates are obtained for the splitting method, then a global rate of convergence follows. This approach can then be readily applied in future research on SDE splitting methods. By incorporating recently developed approximations for iterated integrals of Brownian motion into these piecewise linear paths, we propose several high order splitting methods for SDEs satisfying a certain commutativity condition. In our experiments, which include the Cox-Ingersoll-Ross model and additive noise SDEs (noisy anharmonic oscillator, stochastic FitzHugh-Nagumo model, underdamped Langevin dynamics), the new splitting methods exhibit convergence rates of $O(h^{3/2})$ and outperform schemes previously proposed in the literature., Comment: 49 pages, 9 figures

We consider in this work the convergence of a split-step Euler type scheme (SSM) for the numerical simulation of interacting particle Stochastic Differential Equation (SDE) systems and McKean-Vlasov Stochastic Differential Equations (MV-SDEs) with full super-linear growth in the spatial and the interaction component in the drift, and non-constant Lipschitz diffusion coefficient. The super-linear growth in the interaction (or measure) component stems from convolution operations with super-linear growth functions allowing in particular application to the granular media equation with multi-well confining potentials. From a methodological point of view, we avoid altogether functional inequality arguments (as we allow for non-constant non-bounded diffusion maps). The scheme attains, in stepsize, a near-optimal classical (path-space) root mean-square error rate of $1/2-\varepsilon$ for $\varepsilon>0$ and an optimal rate $1/2$ in the non-path-space mean-square error metric. Numerical examples illustrate all findings. In particular, the testing raises doubts if taming is a suitable methodology for this type of problem (with convolution terms and non-constant diffusion coefficients)., Comment: 40 pages, 3 figures; Final author accepted version (to appear in IMA J. of Num. Analysis)

We propose an explicit drift-randomised Milstein scheme for both McKean--Vlasov stochastic differential equations and associated high-dimensional interacting particle systems with common noise. By using a drift-randomisation step in space and measure, we establish the scheme's strong convergence rate of $1$ under reduced regularity assumptions on the drift coefficient: no classical (Euclidean) derivatives in space or measure derivatives (e.g., Lions/Fr\'echet) are required. The main result is established by enriching the concepts of bistability and consistency of numerical schemes used previously for standard SDE. We introduce certain Spijker-type norms (and associated Banach spaces) to deal with the interaction of particles present in the stochastic systems being analysed. A discussion of the scheme's complexity is provided., Comment: 36 pages including appendix; minor revision from earlier version

In this paper, we develop a natural operator-splitting variational scheme for a general class of non-local, degenerate conservative-dissipative evolutionary equations. The splitting-scheme consists of two phases: a conservative (transport) phase and a dissipative (diffusion) phase. The first phase is solved exactly using the method of characteristic and DiPerna-Lions theory while the second phase is solved approximately using a JKO-type variational scheme that minimizes an energy functional with respect to a certain Kantorovich optimal transport cost functional. In addition, we also introduce an entropic-regularisation of the scheme. We prove the convergence of both schemes to a weak solution of the evolutionary equation. We illustrate the generality of our work by providing a number of examples, including the kinetic Fokker-Planck equation and the (regularized) Vlasov-Poisson-Fokker-Planck equation., Comment: 26 pages. significant revision from the previous versions

We present an implicit Split-Step explicit Euler type Method (dubbed SSM) for the simulation of McKean-Vlasov Stochastic Differential Equations (MV-SDEs) with drifts of superlinear growth in space, Lipschitz in measure and non-constant Lipschitz diffusion coefficient. The scheme is designed to leverage the structure induced by the interacting particle approximation system, including parallel implementation and the solvability of the implicit equation. The scheme attains the classical $1/2$ root mean square error (rMSE) convergence rate in stepsize and closes the gap left by [18, "Simulation of McKean-Vlasov SDEs with super-linear growth" in IMA Journal of Numerical Analysis, 01 2021. draa099] regarding efficient implicit methods and their convergence rate for this class of McKean-Vlasov SDEs. A sufficient condition for mean-square contractivity of the scheme is presented. Several numerical examples are presented, including a comparative analysis to other known algorithms for this class (Taming and Adaptive time-stepping) across parallel and non-parallel implementations., Comment: 30 pages, 15 figures

We prove the well-posedness of solutions to McKean-Vlasov stochastic differential equations driven by L\'evy noise under mild assumptions where, in particular, the L\'evy measure is not required to be finite. The drift, diffusion and jump coefficients are allowed to be random, can grow super-linearly in the state variable, and all may depend on the marginal law of the solution process. We provide a propagation of chaos result under more relaxed conditions than those existing in the literature, and consistent with our well-posedness result. We propose a tamed Euler scheme for the associated interacting particle system and prove that the rate of its strong convergence is arbitrarily close to $1/2$. As a by-product, we also obtain the corresponding results on well-posedness, propagation of chaos and strong convergence of the tamed Euler scheme for McKean-Vlasov stochastic delay differential equations (SDDE) and McKean-Vlasov stochastic differential equations with Markovian switching (SDEwMS), both driven by L\'evy noise. Furthermore, our results on tamed Euler schemes are new even for ordinary SDEs driven by L\'evy noise and with super-linearly growing coefficients., Comment: 33

We propose a stochastic representation for a simple class of transport PDEs based on Ito representations. We detail an algorithm using an estimator stemming for the representation that, unlike regularization by noise estimators, is unbiased. We rely on recent developments on branching diffusions, regime switching processes and their representations of PDEs. There is a loose relation between our technique and regularization by noise, but contrary to the latter, we add a perturbation and immediately its correction. The method is only possible through a judicious choice of the diffusion coefficient $\sigma$. A key feature is that our approach does not rely on the smallness of $\sigma$, in fact, our $\sigma$ is strictly bounded from below which is in stark contrast with standard perturbation techniques. This is critical for extending this method to non-toy PDEs which have nonlinear terms in the first derivative where the usual perturbation technique breaks down. The examples presented show the algorithm outperforming alternative approaches. Moreover, the examples point toward a potential algorithm for the fully nonlinear case where the method of characteristics breaks down., Comment: 31 pages, 2 figures; To appear in "Frontiers in Stochastic Analysis - BSDEs, SPDEs and their Applications"

The numerical solution of large-scale PDEs, such as those occurring in data-driven applications, unavoidably require powerful parallel computers and tailored parallel algorithms to make the best possible use of them. In fact, considerations about the parallelization and scalability of realistic problems are often critical enough to warrant acknowledgement in the modelling phase. The purpose of this paper is to spread awareness of the Probabilistic Domain Decomposition (PDD) method, a fresh approach to the parallelization of PDEs with excellent scalability properties. The idea exploits the stochastic representation of the PDE and its approximation via Monte Carlo in combination with deterministic high-performance PDE solvers. We describe the ingredients of PDD and its applicability in the scope of data science. In particular, we highlight recent advances in stochastic representations for nonlinear PDEs using branching diffusions, which have significantly broadened the scope of PDD. We envision this work as a dictionary giving large-scale PDE practitioners references on the very latest algorithms and techniques of a non-standard, yet highly parallelizable, methodology at the interface of deterministic and probabilistic numerical methods. We close this work with an invitation to the fully nonlinear case and open research questions., Comment: 23 pages, 7 figures; Final SMUR version; To appear in the European Journal of Applied Mathematics (EJAM)

The theory of Forward-Backward Stochastic Differential Equations (FBSDEs) paves a way to probabilistic numerical methods for nonlinear parabolic PDEs. The majority of the results on the numerical methods for FBSDEs relies on the global Lipschitz assumption, which is not satisfied for a number of important cases such as the Fisher--KPP or the FitzHugh--Nagumo equations. Furthermore, it has been shown in \cite{LionnetReisSzpruch2015} that for BSDEs with monotone drivers having polynomial growth in the primary variable $y$, only the (sufficiently) implicit schemes converge. But these require an additional computational effort compared to explicit schemes. This article develops a general framework that allows the analysis, in a systematic fashion, of the integrability properties, convergence and qualitative properties (e.g.~comparison theorem) for whole families of modified explicit schemes. The framework yields the convergence of some modified explicit scheme with the same rate as implicit schemes and with the computational cost of the standard explicit scheme. To illustrate our theory, we present several classes of easily implementable modified explicit schemes that can computationally outperform the implicit one and preserve the qualitative properties of the solution to the BSDE. These classes fit into our developed framework and are tested in computational experiments., Comment: 49 pages, 3 figures

In this paper, we undertake the error analysis of the time discretization of systems of Forward-Backward Stochastic Differential Equations (FBSDEs) with drivers having polynomial growth and that are also monotone in the state variable. We show with a counter-example that the natural explicit Euler scheme may diverge, unlike in the canonical Lipschitz driver case. This is due to the lack of a certain stability property of the Euler scheme which is essential to obtain convergence. However, a thorough analysis of the family of $\theta$-schemes reveals that this required stability property can be recovered if the scheme is sufficiently implicit. As a by-product of our analysis, we shed some light on higher order approximation schemes for FBSDEs under non-Lipschitz condition. We then return to fully explicit schemes and show that an appropriately tamed version of the explicit Euler scheme enjoys the required stability property and as a consequence converges. In order to establish convergence of the several discretizations, we extend the canonical path- and first-order variational regularity results to FBSDEs with polynomial growth drivers which are also monotone. These results are of independent interest for the theory of FBSDEs., Comment: Published at http://dx.doi.org/10.1214/14-AAP1056 in the Annals of Applied Probability (http://www.imstat.org/aap/) by the Institute of Mathematical Statistics (http://www.imstat.org)