Search

Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive, hindering their further development and adoption for applications that need high fidelity. In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model. In particular, we derive the dynamical system governing the continuous-time limit of the Mamba model and characterize the asymptotic behavior of its solutions. In the one-dimensional case, we prove that only one of the following two scenarios happens: either all tokens converge to zero, or all tokens diverge to infinity. We provide criteria based on model parameters to determine when each scenario occurs. For the convergent scenario, we empirically verify that this scenario negatively impacts the model's performance. For the divergent scenario, we prove that different tokens will diverge to infinity at different rates, thereby contributing unequally to the updates during model training. Based on these investigations, we propose two refinements for the model: excluding the convergent scenario and reordering tokens based on their importance scores, both aimed at improving practical performance. Our experimental results validate these refinements, offering insights into enhancing Mamba's effectiveness in real-world applications.

Stochastic gradient descent (SGD) is a powerful optimization technique that is particularly useful in online learning scenarios. Its convergence analysis is relatively well understood under the assumption that the data samples are independent and identically distributed (iid). However, applying SGD to policy optimization problems in operations research involves a distinct challenge: the policy changes the environment and thereby affects the data used to update the policy. The adaptively generated data stream involves samples that are non-stationary, no longer independent from each other, and affected by previous decisions. The influence of previous decisions on the data generated introduces bias in the gradient estimate, which presents a potential source of instability for online learning not present in the iid case. In this paper, we introduce simple criteria for the adaptively generated data stream to guarantee the convergence of SGD. We show that the convergence speed of SGD with adaptive data is largely similar to the classical iid setting, as long as the mixing time of the policy-induced dynamics is factored in. Our Lyapunov-function analysis allows one to translate existing stability analysis of stochastic systems studied in operations research into convergence rates for SGD, and we demonstrate this for queueing and inventory management problems. We also showcase how our result can be applied to study the sample complexity of an actor-critic policy gradient algorithm.

Dynamic mode decomposition (DMD) is a data-driven method of extracting spatial-temporal coherent modes from complex systems and providing an equation-free architecture to model and predict systems. However, in practical applications, the accuracy of DMD can be limited in extracting dynamical features due to sensor noise in measurements. We develop an adaptive method to constantly update dynamic modes and eigenvalues from noisy measurements arising from discrete systems. Our method is based on the Ensemble Kalman filter owing to its capability of handling time-varying systems and nonlinear observables. Our method can be extended to non-autonomous dynamical systems, accurately recovering short-time eigenvalue-eigenvector pairs and observables. Theoretical analysis shows that the estimation is accurate in long term data misfit. We demonstrate the method on both autonomous and non-autonomous dynamical systems to show its effectiveness.

We consider Bayesian inference for image deblurring with total variation (TV) prior. Since the posterior is analytically intractable, we resort to Markov chain Monte Carlo (MCMC) methods. However, since most MCMC methods significantly deteriorate in high dimensions, they are not suitable to handle high resolution imaging problems. In this paper, we show how low-dimensional sampling can still be facilitated by exploiting the sparse conditional structure of the posterior. To this end, we make use of the local structures of the blurring operator and the TV prior by partitioning the image into rectangular blocks and employing a blocked Gibbs sampler with proposals stemming from the Metropolis-Hastings adjusted Langevin Algorithm (MALA). We prove that this MALA-within-Gibbs (MLwG) sampling algorithm has dimension-independent block acceptance rates and dimension-independent convergence rate. In order to apply the MALA proposals, we approximate the TV by a smoothed version, and show that the introduced approximation error is evenly distributed and dimension-independent. Since the posterior is a Gibbs density, we can use the Hammersley-Clifford Theorem to identify the posterior conditionals which only depend locally on the neighboring blocks. We outline computational strategies to evaluate the conditionals, which are the target densities in the Gibbs updates, locally and in parallel. In two numerical experiments, we validate the dimension-independent properties of the MLwG algorithm and demonstrate its superior performance over MALA.

Analyzing dynamical data often requires information of the temporal labels, but such information is unavailable in many applications. Recovery of these temporal labels, closely related to the seriation or sequencing problem, becomes crucial in the study. However, challenges arise due to the nonlinear nature of the data and the complexity of the underlying dynamical system, which may be periodic or non-periodic. Additionally, noise within the feature space complicates the theoretical analysis. Our work develops spectral algorithms that leverage manifold learning concepts to recover temporal labels from noisy data. We first construct the graph Laplacian of the data, and then employ the second (and the third) Fiedler vectors to recover temporal labels. This method can be applied to both periodic and aperiodic cases. It also does not require monotone properties on the similarity matrix, which are commonly assumed in existing spectral seriation algorithms. We develop the $\ell_{\infty}$ error of our estimators for the temporal labels and ranking, without assumptions on the eigen-gap. In numerical analysis, our method outperforms spectral seriation algorithms based on a similarity matrix. The performance of our algorithms is further demonstrated on a synthetic biomolecule data example., Comment: 20 pages, 4 figures

We examine the infinite-dimensional optimization problem of finding a decomposition of a probability measure into K probability sub-measures to minimize specific loss functions inspired by applications in clustering and user grouping. We analytically explore the structures of the support of optimal sub-measures and introduce algorithms based on Wasserstein gradient flow, demonstrating their convergence. Numerical results illustrate the implementability of our algorithms and provide further insights.

In inverse problems, the goal is to estimate unknown model parameters from noisy observational data. Traditionally, inverse problems are solved under the assumption of a fixed forward operator describing the observation model. In this article, we consider the extension of this approach to situations where we have a dynamic forward model, motivated by applications in scientific computation and engineering. We specifically consider this extension for a derivative-free optimizer, the ensemble Kalman inversion (EKI). We introduce and justify a new methodology called dynamic-EKI, which is a particle-based method with a changing forward operator. We analyze our new method, presenting results related to the control of our particle system through its covariance structure. This analysis includes moment bounds and an ensemble collapse, which are essential for demonstrating a convergence result. We establish convergence in expectation and validate our theoretical findings through experiments with dynamic-EKI applied to a 2D Darcy flow partial differential equation.

Ensemble Kalman inversion (EKI) is an ensemble-based method to solve inverse problems. Its gradient-free formulation makes it an attractive tool for problems with involved formulation. However, EKI suffers from the ''subspace property'', i.e., the EKI solutions are confined in the subspace spanned by the initial ensemble. It implies that the ensemble size should be larger than the problem dimension to ensure EKI's convergence to the correct solution. Such scaling of ensemble size is impractical and prevents the use of EKI in high dimensional problems. To address this issue, we propose a novel approach using dropout regularization to mitigate the subspace problem. We prove that dropout-EKI converges in the small ensemble settings, and the computational cost of the algorithm scales linearly with dimension. We also show that dropout-EKI reaches the optimal query complexity, up to a constant factor. Numerical examples demonstrate the effectiveness of our approach.

Principal component analysis (PCA) is a simple and popular tool for processing high-dimensional data. We investigate its effectiveness for matrix denoising. We consider the clean data are generated from a low-dimensional subspace, but masked by independent high-dimensional sub-Gaussian noises with standard deviation $\sigma$. Under the low-rank assumption on the clean data with a mild spectral gap assumption, we prove that the distance between each pair of PCA-denoised data point and the clean data point is uniformly bounded by $O(\sigma \log n)$. To illustrate the spectral gap assumption, we show it can be satisfied when the clean data are independently generated with a non-degenerate covariance matrix. We then provide a general lower bound for the error of the denoised data matrix, which indicates PCA denoising gives a uniform error bound that is rate-optimal. Furthermore, we examine how the error bound impacts downstream applications such as clustering and manifold learning. Numerical results validate our theoretical findings and reveal the importance of the uniform error., Comment: 33 pages, 2 figures

Sampling methods, as important inference and learning techniques, are typically designed for unconstrained domains. However, constraints are ubiquitous in machine learning problems, such as those on safety, fairness, robustness, and many other properties that must be satisfied to apply sampling results in real-life applications. Enforcing these constraints often leads to implicitly-defined manifolds, making efficient sampling with constraints very challenging. In this paper, we propose a new variational framework with a designed orthogonal-space gradient flow (O-Gradient) for sampling on a manifold $\mathcal{G}_0$ defined by general equality constraints. O-Gradient decomposes the gradient into two parts: one decreases the distance to $\mathcal{G}_0$ and the other decreases the KL divergence in the orthogonal space. While most existing manifold sampling methods require initialization on $\mathcal{G}_0$, O-Gradient does not require such prior knowledge. We prove that O-Gradient converges to the target constrained distribution with rate $\widetilde{O}(1/\text{the number of iterations})$ under mild conditions. Our proof relies on a new Stein characterization of conditional measure which could be of independent interest. We implement O-Gradient through both Langevin dynamics and Stein variational gradient descent and demonstrate its effectiveness in various experiments, including Bayesian deep neural networks., Comment: NeurIPS 2022

Many organizations have access to abundant data but lack the computational power to process the data. While they can outsource the computational task to other facilities, there are various constraints on the amount of data that can be shared. It is natural to ask what can data outsourcing accomplish under such constraints. We address this question from a machine learning perspective. When training a model with optimization algorithms, the quality of the results often relies heavily on the points where the algorithms are initialized. Random start is one of the most popular methods to tackle this issue, but it can be computationally expensive and not feasible for organizations lacking computing resources. Based on three different scenarios, we propose simulation-based algorithms that can utilize a small amount of outsourced data to find good initial points accordingly. Under suitable regularity conditions, we provide theoretical guarantees showing the algorithms can find good initial points with high probability. We also conduct numerical experiments to demonstrate that our algorithms perform significantly better than the random start approach., Comment: 22 pages, 5 figures

When implementing Markov Chain Monte Carlo (MCMC) algorithms, perturbation caused by numerical errors is sometimes inevitable. This paper studies how perturbation of MCMC affects the convergence speed and Monte Carlo estimation accuracy. Our results show that when the original Markov chain converges to stationarity fast enough and the perturbed transition kernel is a good approximation to the original transition kernel, the corresponding perturbed sampler has similar convergence speed and high approximation accuracy as well. We discuss two different analysis frameworks: ergodicity and spectral gap, both are widely used in the literature. Our results can be easily extended to obtain non-asymptotic error bounds for MCMC estimators. We also demonstrate how to apply our convergence and approximation results to the analysis of specific sampling algorithms, including Random walk Metropolis and Metropolis adjusted Langevin algorithm with perturbed target densities, and parallel tempering Monte Carlo with perturbed densities. Finally we present some simple numerical examples to verify our theoretical claims., Comment: 26 pages, 5 figures

In reinforcement learning (RL), offline learning decoupled learning from data collection and is useful in dealing with exploration-exploitation tradeoff and enables data reuse in many applications. In this work, we study two offline learning tasks: policy evaluation and policy learning. For policy evaluation, we formulate it as a stochastic optimization problem and show that it can be solved using approximate stochastic gradient descent (aSGD) with time-dependent data. We show aSGD achieves $\tilde O(1/t)$ convergence when the loss function is strongly convex and the rate is independent of the discount factor $\gamma$. This result can be extended to include algorithms making approximately contractive iterations such as TD(0). The policy evaluation algorithm is then combined with the policy iteration algorithm to learn the optimal policy. To achieve an $\epsilon$ accuracy, the complexity of the algorithm is $\tilde O(\epsilon^{-2}(1-\gamma)^{-5})$, which matches the complexity bound for classic online RL algorithms such as Q-learning.

Ensemble Kalman inversion (EKI) is a technique for the numerical solution of inverse problems. A great advantage of the EKI's ensemble approach is that derivatives are not required in its implementation. But theoretically speaking, EKI's ensemble size needs to surpass the dimension of the problem. This is because of EKI's "subspace property", i.e., that the EKI solution is a linear combination of the initial ensemble it starts off with. We show that the ensemble can break out of this initial subspace when "localization" is applied. In essence, localization enforces an assumed correlation structure onto the problem, and is heavily used in ensemble Kalman filtering and data assimilation. We describe and analyze how to apply localization to the EKI, and how localization helps the EKI ensemble break out of the initial subspace. Specifically, we show that the localized EKI (LEKI) ensemble will collapse to a single point (as intended) and that the LEKI ensemble mean will converge to the global optimum at a sublinear rate. Under strict assumptions on the localization procedure and observation process, we further show that the data misfit decays uniformly. We illustrate our ideas and theoretical developments with numerical examples with simplified toy problems, a Lorenz model, and an inversion of electromagnetic data, where some of our mathematical assumptions may only be approximately valid., Comment: 37 pages, 7 figures

Ensemble Kalman inversion (EKI) is a derivative-free optimizer aimed at solving inverse problems, taking motivation from the celebrated ensemble Kalman filter. The purpose of this article is to consider the introduction of adaptive Tikhonov strategies for EKI. This work builds upon Tikhonov EKI (TEKI) which was proposed for a fixed regularization constant. By adaptively learning the regularization parameter, this procedure is known to improve the recovery of the underlying unknown. For the analysis, we consider a continuous-time setting where we extend known results such as well-posdeness and convergence of various loss functions, but with the addition of noisy observations. Furthermore, we allow a time-varying noise and regularization covariance in our presented convergence result which mimic adaptive regularization schemes. In turn we present three adaptive regularization schemes, which are highlighted from both the deterministic and Bayesian approaches for inverse problems, which include bilevel optimization, the MAP formulation and covariance learning. We numerically test these schemes and the theory on linear and nonlinear partial differential equations, where they outperform the non-adaptive TEKI and EKI.

Nature-inspired swarm-based algorithms have been widely applied to tackle high-dimensional and complex optimization problems across many disciplines. They are general purpose optimization algorithms, easy to use and implement, flexible and assumption-free. A common drawback of these algorithms is premature convergence and the solution found is not a global optimum. We provide sufficient conditions for an algorithm to converge almost surely (a.s.) to a global optimum. We then propose a general, simple and effective strategy, called Perturbation-Projection (PP), to enhance an algorithm's exploration capability so that our convergence conditions are guaranteed to hold. We illustrate this approach using three widely used nature-inspired swarm-based optimization algorithms: particle swarm optimization (PSO), bat algorithm (BAT) and competitive swarm optimizer (CSO). Extensive numerical experiments show that each of the three algorithms with the enhanced PP strategy outperforms the original version in a number of notable ways., Comment: 20 pages, 9 figures

The field distribution inside the superconducting radiofrequency (SRF) film with different mean free path is studied using niobium (Nb) as an example. The surface resistance of clean Nb film with different substrate and different film thickness is calculated. We also show the study of a special structured multilayer superconducting film called Superconductor-Insulator-Superconductor (SIS) structure.

Core-collapse supernovae are among the most magnificent events in the observable universe. They produce many of the chemical elements necessary for life to exist and their remnants -- neutron stars and black holes -- are interesting astrophysical objects in their own right. However, despite millennia of observations and almost a century of astrophysical study, the explosion mechanism of core-collapse supernovae is not yet well understood. Hyper-Kamiokande is a next-generation neutrino detector that will be able to observe the neutrino flux from the next galactic core-collapse supernova in unprecedented detail. We focus on the first 500 ms of the neutrino burst, corresponding to the accretion phase, and use a newly-developed, high-precision supernova event generator to simulate Hyper-Kamiokande's response to five different supernova models. We show that Hyper-Kamiokande will be able to distinguish between these models with high accuracy for a supernova at a distance of up to 100 kpc. Once the next galactic supernova happens, this ability will be a powerful tool for guiding simulations towards a precise reproduction of the explosion mechanism observed in nature., Comment: 21 pages, 7 figures. Article based on thesis published as arXiv:2002.01649. v2: added references and some explanations in response to reviewer comments

The likelihood-informed subspace (LIS) method offers a viable route to reducing the dimensionality of high-dimensional probability distributions arising in Bayesian inference. LIS identifies an intrinsic low-dimensional linear subspace where the target distribution differs the most from some tractable reference distribution. Such a subspace can be identified using the leading eigenvectors of a Gram matrix of the gradient of the log-likelihood function. Then, the original high-dimensional target distribution is approximated through various forms of marginalization of the likelihood function, in which the approximated likelihood only has support on the intrinsic low-dimensional subspace. This approximation enables the design of inference algorithms that can scale sub-linearly with the apparent dimensionality of the problem. Intuitively, the accuracy of the approximation, and hence the performance of the inference algorithms, are influenced by three factors -- the dimension truncation error in identifying the subspace, Monte Carlo error in estimating the Gram matrices, and Monte Carlo error in constructing marginalizations. %This work establishes a unified framework to analyze each of these three factors and their interplay. Under mild technical assumptions, we establish error bounds for a range of existing dimension reduction techniques based on the principle of LIS. Our error bounds also provide useful insights into the accuracy of these methods. In addition, we analyze the integration of LIS with sampling methods such as Markov Chain Monte Carlo (MCMC) and sequential Monte Carlo (SMC). We also demonstrate the applicability of our analysis on a linear inverse problem with Gaussian prior, which shows that all the estimates can be dimension-independent if the prior covariance is a trace-class operator., Comment: 51 pages, 8 figures

One fundamental problem when solving inverse problems is how to find regularization parameters. This article considers solving this problem using data-driven bilevel optimization, i.e. we consider the adaptive learning of the regularization parameter from data by means of optimization. This approach can be interpreted as solving an empirical risk minimization problem, and we analyze its performance in the large data sample size limit for general nonlinear problems. We demonstrate how to implement our framework on linear inverse problems, where we can further show the inverse accuracy does not depend on the ambient space dimension. To reduce the associated computational cost, online numerical schemes are derived using the stochastic gradient descent method. We prove convergence of these numerical schemes under suitable assumptions on the forward problem. Numerical experiments are presented illustrating the theoretical results and demonstrating the applicability and efficiency of the proposed approaches for various linear and nonlinear inverse problems, including Darcy flow, the eikonal equation, and an image denoising example.

Langevin diffusion (LD) is one of the main workhorses for sampling problems. However, its convergence rate can be significantly reduced if the target distribution is a mixture of multiple densities, especially when each component concentrates around a different mode. Replica exchange Langevin diffusion (ReLD) is a sampling method that can circumvent this issue. In particular, ReLD adds another LD sampling a high-temperature version of the target density, and exchange the locations of two LDs according to a Metropolis-Hasting type of law. This approach can be further extended to multiple replica exchange Langevin diffusion (mReLD), where $K$ additional LDs are added to sample distributions at different temperatures and exchanges take place between neighboring-temperature processes. While ReLD and mReLD have been used extensively in statistical physics, molecular dynamics, and other applications, there is little existing analysis on its convergence rate and choices of temperatures. This paper closes these gaps assuming the target distribution is a mixture of log-concave densities. We show ReLD can obtain constant or even better convergence rates even when the density components of the mixture concentrate around isolated modes. We also show using mReLD with $K$ additional LDs can achieve the same result while the exchange frequency only needs to be $(1/K)$-th power of the one in ReLD.

For the radiofrequency (RF) applications of normal conducting film with large mean free path at high frequency and low temperature, the anomalous skin effect differs considerably from the normal skin effect with field decaying exponentially in the film. Starting from the relationship between the current and the electric field (E field) in the film, the amplitude of E field along the film depth is calculated, and is found to be non-monotonic. The surface impedance is found to have a minimum value at certain film thickness. We apply this calculation into a Cu coated S.S. beam pipe used in an accelerator to reduce the ohmic power loss to determine the minimum thickness that should be applied., Comment: 6 pages, 5 figures

One classical canon of statistics is that large models are prone to overfitting, and model selection procedures are necessary for high dimensional data. However, many overparameterized models, such as neural networks, perform very well in practice, although they are often trained with simple online methods and regularization. The empirical success of overparameterized models, which is often known as benign overfitting, motivates us to have a new look at the statistical generalization theory for online optimization. In particular, we present a general theory on the generalization error of stochastic gradient descent (SGD) solutions for both convex and locally convex loss functions. We further discuss data and model conditions that lead to a ``low effective dimension". Under these conditions, we show that the generalization error either does not depend on the ambient dimension $p$ or depends on $p$ via a poly-logarithmic factor. We also demonstrate that in several widely used statistical models, the ``low effective dimension'' arises naturally in overparameterized settings. The studied statistical applications include both convex models such as linear regression and logistic regression and non-convex models such as $M$-estimator and two-layer neural networks., Comment: 60 pages, 2 figures

CW photoinjectors operating at high accelerating gradients promise to revolutionize many areas of science and applications. They can establish the basis for a new generation of monochromatic X-ray free electron lasers, high brightness hadron beams, or a new generation of microchip production. In this letter we report on the record-performing superconducting RF electron gun with $\textrm{CsK}_{2}\textrm{Sb}$ photocathode. The gun is generating high charge electron bunches (up to 10 nC/bunch) and low transverse emittances, while operating for months with a single photocathode. This achievement opens a new era in generating high-power beams with a very high average brightness.

Gradient descent (GD) is known to converge quickly for convex objective functions, but it can be trapped at local minima. On the other hand, Langevin dynamics (LD) can explore the state space and find global minima, but in order to give accurate estimates, LD needs to run with a small discretization step size and weak stochastic force, which in general slow down its convergence. This paper shows that these two algorithms and their non-swapping variants. can ``collaborate" through a simple exchange mechanism, in which they swap their current positions if LD yields a lower objective function. This idea can be seen as the singular limit of the replica-exchange technique from the sampling literature. We show that this new algorithm converges to the global minimum linearly with high probability, assuming the objective function is strongly convex in a neighborhood of the unique global minimum. By replacing gradients with stochastic gradients, and adding a proper threshold to the exchange mechanism, our algorithm can also be used in online settings. We also study non-swapping variants of the algorithm, which achieve similar performance. We further verify our theoretical results through some numerical experiments and observe superior performance of the proposed algorithm over running GD or LD alone., Comment: 70 pages, 15 figures

Many data-science problems can be formulated as an inverse problem, where the parameters are estimated by minimizing a proper loss function. When complicated black-box models are involved, derivative-free optimization tools are often needed. The ensemble Kalman filter (EnKF) is a particle-based derivative-free Bayesian algorithm originally designed for data assimilation. Recently, it has been applied to inverse problems for computational efficiency. The resulting algorithm, known as ensemble Kalman inversion (EKI), involves running an ensemble of particles with EnKF update rules so they can converge to a minimizer. In this article, we investigate EKI convergence in general nonlinear settings. To improve convergence speed and stability, we consider applying EKI with non-constant step-sizes and covariance inflation. We prove that EKI can hit critical points with finite steps in non-convex settings. We further prove that EKI converges to the global minimizer polynomially fast if the loss function is strongly convex. We verify the analysis presented with numerical experiments on two inverse problems.

Concurrent observation technologies have made high-precision real-time data available in large quantities. Data assimilation (DA) is concerned with how to combine this data with physical models to produce accurate predictions. For spatial-temporal models, the Ensemble Kalman Filter with proper localization techniques is considered to be a state-of-the-art DA methodology. This article proposes and investigates a localized Ensemble Kalman Bucy Filter (l-EnKBF) for nonlinear models with short-range interactions. We derive dimension-independent and component-wise error bounds and show the long time path-wise error only has logarithmic dependence on the time range. The theoretical results are verified through some simple numerical tests.

Inverse problem is ubiquitous in science and engineering, and Bayesian methodologies are often used to infer the underlying parameters. For high dimensional temporal-spatial models, classical Markov chain Monte Carlo (MCMC) methods are often slow to converge, and it is necessary to apply Metropolis-within-Gibbs (MwG) sampling on parameter blocks. However, the computation cost of each MwG iteration is typically $O(n^2)$, where $n$ is the model dimension. This can be too expensive in practice. This paper introduces a new reduced computation method to bring down the computation cost to $O(n)$, for the inverse initial value problem of a stochastic differential equation (SDE) with local interactions. The key observation is that each MwG proposal is only different from the original iterate at one parameter block, and this difference will only propagate within a local domain in the SDE computations. Therefore we can approximate the global SDE computation with a surrogate updated only within the local domain for reduced computation cost. Both theoretically and numerically, we show that the approximation errors can be controlled by the local domain size. We discuss how to implement the local computation scheme using Euler--Maruyama and 4th order Runge--Kutta methods. We numerically demonstrate the performance of the proposed method with the Lorenz 96 model and a linear stochastic flow model.

Stochastic gradient Langevin dynamics (SGLD) is a fundamental algorithm in stochastic optimization. Recent work by Zhang et al. [2017] presents an analysis for the hitting time of SGLD for the first and second order stationary points. The proof in Zhang et al. [2017] is a two-stage procedure through bounding the Cheeger's constant, which is rather complicated and leads to loose bounds. In this paper, using intuitions from stochastic differential equations, we provide a direct analysis for the hitting times of SGLD to the first and second order stationary points. Our analysis is straightforward. It only relies on basic linear algebra and probability theory tools. Our direct analysis also leads to tighter bounds comparing to Zhang et al. [2017] and shows the explicit dependence of the hitting time on different factors, including dimensionality, smoothness, noise strength, and step size effects. Under suitable conditions, we show that the hitting time of SGLD to first-order stationary points can be dimension-independent. Moreover, we apply our analysis to study several important online estimation problems in machine learning, including linear regression, matrix factorization, and online PCA., Comment: 41 pages

To improve RHIC luminosity for heavy ion beam energies below 10 GeV/nucleon, the Low Energy RHIC electron Cooler (LEReC) is currently under commissioning at BNL. The Linac of LEReC is designed to deliver a 1.6 MeV to 2.6 MeV electron beam, with rms dp/p less than 5e-4. A 704 MHz superconducting radio frequency (SRF) booster cavity in this Linac provides up to 2.2 MeV accelerating voltage. With such a low energy and very demanding energy spread requirement, control of Higher Order Modes (HOMs) in the cavities becomes critical and needs to be carefully evaluated to ensure minimum impact on the beam. In this paper, we report the multiphysics design of the HOM damper for this cavity to meet the energy spread requirement, as well as experimental results of the cavity with and without the HOM damper., Comment: 9 pages, 7 figures

Ensemble Kalman inversion is a parallelizable methodology for solving inverse or parameter estimation problems. Although it is based on ideas from Kalman filtering, it may be viewed as a derivative-free optimization method. In its most basic form it regularizes ill-posed inverse problems through the subspace property: the solution found is in the linear span of the initial ensemble employed. In this work we demonstrate how further regularization can be imposed, incorporating prior information about the underlying unknown. In particular we study how to impose Tikhonov-like Sobolev penalties. As well as introducing this modified ensemble Kalman inversion methodology, we also study its continuous-time limit, proving ensemble collapse; in the language of multi-agent optimization this may be viewed as reaching consensus. We also conduct a suite of numerical experiments to highlight the benefits of Tikhonov regularization in the ensemble inversion context., Comment: 39 pages

Nonlinear dynamical stochastic models are ubiquitous in different areas. Excitable media models are typical examples with large state dimensions. Their statistical properties are often of great interest but are also very challenging to compute. In this article, a theoretical framework to understand the spatial localization for a large class of stochastically coupled nonlinear systems in high dimensions is developed. Rigorous mathematical theories show the covariance decay behavior due to both local and nonlocal effects, which result from the diffusion and the mean field interaction, respectively. The analysis is based on a comparison with an appropriate linear surrogate model, of which the covariance propagation can be computed explicitly. Two important applications of these theoretical results are discussed. They are the spatial averaging strategy for efficiently sampling the covariance matrix and the localization technique in data assimilation. Test examples of a surrogate linear model and a stochastically coupled FitzHugh-Nagumo model for excitable media are adopted to validate the theoretical results. The latter is also used for a systematical study of the spatial averaging strategy in efficiently sampling the covariance matrix in different dynamical regimes., Comment: 34 pages, 9 figures

Extreme events and the heavy tail distributions driven by them are ubiquitous in various scientific, engineering and financial research. They are typically associated with stochastic instability caused by hidden unresolved processes. Previous studies have shown that such instability can be modeled by a stochastic damping in conditional Gaussian models. However, these results are mostly obtained through numerical experiments, while a rigorous understanding of the underlying mechanism is sorely lacking. This paper contributes to this issue by establishing a theoretical framework, in which the tail density of conditional Gaussian models can be rigorously determined. In rough words, we show that if the stochastic damping takes negative values, the tail is polynomial; if the stochastic damping is nonnegative but takes value zero, the tail is between exponential and Gaussian. The proof is established by constructing a novel, product-type Lyapunov function, where a Feynman-Kac formula is applied. The same framework also leads to a non-asymptotic large deviation bound for long-time averaging processes., Comment: 34 pages, 4 figures

On the strength of a double Nobel prize winning experiment (Super)Kamiokande and an extremely successful long baseline neutrino programme, the third generation Water Cherenkov detector, Hyper-Kamiokande, is being developed by an international collaboration as a leading worldwide experiment based in Japan. The Hyper-Kamiokande detector will be hosted in the Tochibora mine, about 295 km away from the J-PARC proton accelerator research complex in Tokai, Japan. The currently existing accelerator will be steadily upgraded to reach a MW beam by the start of the experiment. A suite of near detectors will be vital to constrain the beam for neutrino oscillation measurements. A new cavern will be excavated at the Tochibora mine to host the detector. The experiment will be the largest underground water Cherenkov detector in the world and will be instrumented with new technology photosensors, faster and with higher quantum efficiency than the ones in Super-Kamiokande. The science that will be developed will be able to shape the future theoretical framework and generations of experiments. Hyper-Kamiokande will be able to measure with the highest precision the leptonic CP violation that could explain the baryon asymmetry in the Universe. The experiment also has a demonstrated excellent capability to search for proton decay, providing a significant improvement in discovery sensitivity over current searches for the proton lifetime. The atmospheric neutrinos will allow to determine the neutrino mass ordering and, together with the beam, able to precisely test the three-flavour neutrino oscillation paradigm and search for new phenomena. A strong astrophysical programme will be carried out at the experiment that will detect supernova neutrinos and will measure precisely solar neutrino oscillation., Comment: 325 pages

The Low Energy RHIC electron Cooler (LEReC) is currently under commissioning at BNL to improve RHIC luminosity for heavy ion beam energies below 10 GeV/nucleon. The linac of LEReC consists of a DC photoemission gun, one 704 MHz superconducting radio frequency (SRF) booster cavity, and three normal conducting cavities. It is designed to deliver a 1.6 MeV to 2.6 MeV electron beam, with peak-to-peak momentum spread dp/p of less than 7e4. Two of the three normal conducting cavities will be used in LEReC for energy spread correction. A single-cell 704 MHz cavity for energy de-chirping and a three-cell 2.1 GHz third harmonic cavity for RF curvature correction. In this paper, we present the designs and RF test results of these two cavities., Comment: 10 pages, 10 figures

The adiabatic evolution is the dynamics of an instantaneous eigenstate of a slowly varing Hamiltonian. Recently, an interesting phenomenon shows up that white noises can enhance and even induce adiabaticity, which is in contrast to previous perception that environmental noises always modify and even ruin a designed adiabatic passage. We experimentally realized a noise-induced adiabaticity in a nuclear magnetic resonance system. Adiabatic Hadamard gate and entangled state are demonstrated. The effect of noise on adiabaticity is experimentally exhibited and compared with the noise-free process. We utilized a noise-injected method, which can be applied to other quantum systems., Comment: 6 pages, 5 figures

We investigate how ideas from covariance localization in numerical weather prediction can be used in Markov chain Monte Carlo (MCMC) sampling of high-dimensional posterior distributions arising in Bayesian inverse problems. To localize an inverse problem is to enforce an anticipated "local" structure by (i) neglecting small off-diagonal elements of the prior precision and covariance matrices; and (ii) restricting the influence of observations to their neighborhood. For linear problems we can specify the conditions under which posterior moments of the localized problem are close to those of the original problem. We explain physical interpretations of our assumptions about local structure and discuss the notion of high dimensionality in local problems, which is different from the usual notion of high dimensionality in function space MCMC. The Gibbs sampler is a natural choice of MCMC algorithm for localized inverse problems and we demonstrate that its convergence rate is independent of dimension for localized linear problems. Nonlinear problems can also be tackled efficiently by localization and, as a simple illustration of these ideas, we present a localized Metropolis-within-Gibbs sampler. Several linear and nonlinear numerical examples illustrate localization in the context of MCMC samplers for inverse problems., Comment: 33 pages, 5 figures

This article presents a rigorous analysis for efficient statistically accurate algorithms for solving the Fokker-Planck equations associated with high-dimensional nonlinear turbulent dynamical systems with conditional Gaussian structures. Despite the conditional Gaussianity, these nonlinear systems contain many strong non-Gaussian features such as intermittency and fat-tailed probability density functions (PDFs). The algorithms involve a hybrid strategy that requires only a small number of samples $L$ to capture both the transient and the equilibrium non-Gaussian PDFs with high accuracy. Here, a conditional Gaussian mixture in a high-dimensional subspace via an extremely efficient parametric method is combined with a judicious Gaussian kernel density estimation in the remaining low-dimensional subspace. Rigorous analysis shows that the mean integrated squared error in the recovered PDFs in the high-dimensional subspace is bounded by the inverse square root of the determinant of the conditional covariance, where the conditional covariance is completely determined by the underlying dynamics and is independent of $L$. This is fundamentally different from a direct application of kernel methods to solve the full PDF, where $L$ needs to increase exponentially with the dimension of the system and the bandwidth shrinks. A detailed comparison between different methods justifies that the efficient statistically accurate algorithms are able to overcome the curse of dimensionality. It is also shown with mathematical rigour that these algorithms are robust in long time provided that the system is controllable and stochastically stable. Particularly, dynamical systems with energy-conserving quadratic nonlinearity as in many geophysical and engineering turbulence are proved to have these properties., Comment: 35 pages, 8 figures

Ensemble Kalman filter (EnKF) is an important data assimilation method for high dimensional geophysical systems. Efficient implementation of EnKF in practice often involves the localization technique, which updates each component using only information within a local radius. This paper rigorously analyzes the local EnKF (LEnKF) for linear systems, and shows that the filter error can be dominated by the ensemble covariance, as long as 1) the sample size exceeds the logarithmic of state dimension and a constant that depends only on the local radius; 2) the forecast covariance matrix admits a stable localized structure. In particular, this indicates that with small system and observation noises, the filter error will be accurate in long time even if the initialization is not. The analysis also reveals an intrinsic inconsistency caused by the localization technique, and a stable localized structure is necessary to control this inconsistency. While this structure is usually taken for granted for the operation of LEnKF, it can also be rigorously proved for linear systems with sparse local observations and weak local interactions. These theoretical results are also validated by numerical implementation of LEnKF on a simple stochastic turbulence in two dynamical regimes., Comment: 40 pages, 3 figures