Search

The well-known H ∞ control design gives robustness to a controller by rejecting perturbations from the external environment, which is difficult to do for completely unknown affine nonlinear systems. Accordingly, the immediate objective of this paper is to develop an on-line real-time synergetic learning algorithm, so that a data-driven H ∞ controller can be received. By converting the H ∞ control problem into a two-player zero-sum game, a model-free Hamilton–Jacobi–Isaacs equation (MF-HJIE) is first derived using off-policy reinforcement learning, followed by a proof of equivalence between the MF-HJIE and the conventional HJIE. Next, by applying the temporal difference to the MF-HJIE, a synergetic evolutionary rule with experience replay is designed to learn the optimal value function, the optimal control, and the worst perturbation, that can be performed on-line and in real-time along the system state trajectory. It is proven that the synergistic learning system constructed by the system plant and the evolutionary rule is uniformly ultimately bounded. Finally, simulation results on an F16 aircraft system and a nonlinear system back up the tractability of the proposed method. • A model-free Hamiltonian–Jacobi–Isaacs equation is derived for H ∞ control problems. • A real-time on-line evolutionary rule for tuning neural network weights is developed. • The system and neural network approximation errors achieve synergetic learning. [ABSTRACT FROM AUTHOR]

We employ properties of high-dimensional geometry to obtain some insights into capabilities of deep perceptron networks to classify large data sets. We derive conditions on network depths, types of activation functions, and numbers of parameters that imply that approximation errors behave almost deterministically. We illustrate general results by concrete cases of popular activation functions: Heaviside, ramp sigmoid, rectified linear, and rectified power. Our probabilistic bounds on approximation errors are derived using concentration of measure type inequalities (method of bounded differences) and concepts from statistical learning theory. [ABSTRACT FROM AUTHOR]

Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. However, most supervised SNN training methods, such as conversion from artificial neural networks or direct training with surrogate gradients, require complex computation rather than spike-based operations of spiking neurons during training. In this paper, we study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method, implicit differentiation on the equilibrium state (IDE), for supervised learning with purely spike-based computation, which demonstrates the potential for energy-efficient training of SNNs. Specifically, we introduce ternary spiking neuron couples and prove that implicit differentiation can be solved by spikes based on this design, so the whole training procedure, including both forward and backward passes, is made as event-driven spike computation, and weights are updated locally with two-stage average firing rates. Then we propose to modify the reset membrane potential to reduce the approximation error of spikes. With these key components, we can train SNNs with flexible structures in a small number of time steps and with firing sparsity during training, and the theoretical estimation of energy costs demonstrates the potential for high efficiency. Meanwhile, experiments show that even with these constraints, our trained models can still achieve competitive results on MNIST, CIFAR-10, CIFAR-100, and CIFAR10-DVS. • Novel method with purely spike-based computation to train spiking neural networks. • Analysis of the approximation error of spikes and the method to reduce the error. • Much lower energy costs by low latency and firing sparsity during training. • Competitive performance on static and neuromorphic datasets. [ABSTRACT FROM AUTHOR]

We study the expressive power of deep ReLU neural networks for approximating functions in dilated shift-invariant spaces, which are widely used in signal processing, image processing, communications and so on. Approximation error bounds are estimated with respect to the width and depth of neural networks. The network construction is based on the bit extraction and data-fitting capacity of deep neural networks. As applications of our main results, the approximation rates of classical function spaces such as Sobolev spaces and Besov spaces are obtained. We also give lower bounds of the L p (1 ≤ p ≤ ∞) approximation error for Sobolev spaces, which show that our construction of neural network is asymptotically optimal up to a logarithmic factor. [ABSTRACT FROM AUTHOR]

A recent paper (Mhaskar (2020)) introduces a straightforward and simple kernel based approximation for manifold learning that does not require the knowledge of anything about the manifold, except for its dimension. In this paper, we examine how the pointwise error in approximation using least squares optimization based on similarly localized kernels depends upon the data characteristics and deteriorates as one goes away from the training data. The theory is presented with an abstract localized kernel, which can utilize any prior knowledge about the data being located on an unknown sub-manifold of a known manifold. We demonstrate the performance of our approach using a publicly available micro-Doppler data set, and investigate the use of different preprocessing measures, kernels, and manifold dimensions. Specifically, it is shown that the localized kernel introduced in the above mentioned paper when used with PCA components leads to a near-competitive performance to deep neural networks, and offers significant improvements in training speed and memory requirements. To demonstrate the fact that our methods are agnostic to the domain knowledge, we examine the classification problem in a simple video data set. • We examine the pointwise error in approximation using least squares optimization based on the kernel presented in Mhaskar (2020). • The approximation error depends upon the data characteristics and deteriorates as one goes away from the training data. • The theory is presented with an abstract localized kernel, which can utilize any prior knowledge about the data being located on an unknown sub-manifold of a known manifold. • We demonstrate the performance of this kernel using a publicly available micro-Doppler data set, investigating the use of different preprocessing measures, kernels, and manifold dimension. • It is shown that the proposed method offers nearcompetitive performance to deep neural networks and significant improvements to speed and accuracy. [ABSTRACT FROM AUTHOR]

Approximation error is a key measure in the process of model validation and verification for neural networks. In this paper, the problems of guaranteed error estimation of neural networks and applications to assured system modeling and assured neural network compression are addressed. First, a concept called guaranteed error estimation of feedforward neural networks is proposed, which intends to provide the worst-case approximation error of a trained neural network with respect to a compact input set essentially containing an infinite number of values. Given different prior information about the original system, two approaches including Lipschitz constant analysis and set-valued reachability analysis methods are developed to efficiently compute upper-bounds of approximation errors. Based on the guaranteed approximation error estimation framework, an optimization for obtaining parameter values from data set is proposed. A robotic arm and neural network compression examples are presented to illustrate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

The expressive power of deep neural networks is manifested by their remarkable ability to approximate multivariate functions in a way that appears to overcome the curse of dimensionality. This ability is exemplified by their success in solving high-dimensional problems where traditional numerical solvers fail due to their limitations in accurately representing high-dimensional structures. To provide a theoretical framework for explaining this phenomenon, we analyze the approximation of Hölder functions defined on a d -dimensional smooth manifold M embedded in R D , with d ≪ D , using deep neural networks. We prove that the uniform convergence estimates of the approximation and generalization errors by deep neural networks with ReLU activation functions do not depend on the ambient dimension D of the function but only on its lower manifold dimension d , in a precise sense. Our result improves existing results from the literature where approximation and generalization errors were shown to depend weakly on D. • Deep ReLU neural networks exhibit remarkable approximation properties for multivariate functions. • We prove that the approximation error of manifold-embedded data depends only on the manifold dimension. • Argument relies on preservation of pairwise ambient distances on the manifold. • Our result improves existing results where the approximation error depends weakly on the ambient dimension. • Our result implies an improved estimate of the generalization error. [ABSTRACT FROM AUTHOR]

Invertible neural networks (INNs) represent an important class of deep neural network architectures that have been widely used in applications. The universal approximation properties of INNs have been established recently. However, the approximation rate of INNs is largely missing. In this work, we provide an analysis of the capacity of a class of coupling-based INNs to approximate bi-Lipschitz continuous mappings on a compact domain, and the result shows that it can well approximate both forward and inverse maps simultaneously. Furthermore, we develop an approach for approximating bi-Lipschitz maps on infinite-dimensional spaces that simultaneously approximate the forward and inverse maps, by combining model reduction with principal component analysis and INNs for approximating the reduced map, and we analyze the overall approximation error of the approach. Preliminary numerical results show the feasibility of the approach for approximating the solution operator for parameterized second-order elliptic problems. [ABSTRACT FROM AUTHOR]

Despite the widespread success of deep learning in various applications, neural network theory has been lagging behind. The choice of the activation function plays a critical role in the expressivity of a neural network but for reasons that are not yet fully understood. While the rectified linear unit (ReLU) is currently one of the most popular activation functions, ReLU squared has only recently been empirically shown to be pivotal in producing consistently superior results for state-of-the-art deep learning tasks (So et al., 2021). To analyze the expressivity of neural networks with ReLU powers, we employ the novel framework of Gribonval et al. (2022) based on the classical concept of approximation spaces. We consider the class of functions for which the approximation error decays at a sufficiently fast rate as network complexity, measured by the number of weights, increases. We show that when approximating sufficiently smooth functions that cannot be represented by sufficiently low-degree polynomials, networks with ReLU powers need less depth than those with ReLU. Moreover, if they have the same depth, networks with ReLU powers can have potentially faster approximation rates. Lastly, our computational experiments on approximating the Rastrigin and Ackley functions with deep neural networks showed that ReLU squared and ReLU cubed networks consistently outperform ReLU networks. • ReLU powers need less depth than ReLU in approximating sufficiently smooth functions • For the same depth, ReLU powers can potentially learn faster than those with ReLU • ReLU powers outperform ReLU in approximating Rastrigin and Ackley functions [ABSTRACT FROM AUTHOR]

We study the efficacy and efficiency of deep generative networks for approximating probability distributions. We prove that neural networks can transform a low-dimensional source distribution to a distribution that is arbitrarily close to a high-dimensional target distribution, when the closeness is measured by Wasserstein distances and maximum mean discrepancy. Upper bounds of the approximation error are obtained in terms of the width and depth of neural network. Furthermore, it is shown that the approximation error in Wasserstein distance grows at most linearly on the ambient dimension and that the approximation order only depends on the intrinsic dimension of the target distribution. On the contrary, when f -divergences are used as metrics of distributions, the approximation property is different. We show that in order to approximate the target distribution in f -divergences, the dimension of the source distribution cannot be smaller than the intrinsic dimension of the target distribution. [ABSTRACT FROM AUTHOR]

The Delta method is a classical procedure for quantifying epistemic uncertainty in statistical models, but its direct application to deep neural networks is prevented by the large number of parameters P. We propose a low cost approximation of the Delta method applicable to L 2 -regularized deep neural networks based on the top K eigenpairs of the Fisher information matrix. We address efficient computation of full-rank approximate eigendecompositions in terms of the exact inverse Hessian, the inverse outer-products of gradients approximation and the so-called Sandwich estimator. Moreover, we provide bounds on the approximation error for the uncertainty of the predictive class probabilities. We show that when the smallest computed eigenvalue of the Fisher information matrix is near the L 2 -regularization rate, the approximation error will be close to zero even when K ≪ P. A demonstration of the methodology is presented using a TensorFlow implementation, and we show that meaningful rankings of images based on predictive uncertainty can be obtained for two LeNet and ResNet-based neural networks using the MNIST and CIFAR-10 datasets. Further, we observe that false positives have on average a higher predictive epistemic uncertainty than true positives. This suggests that there is supplementing information in the uncertainty measure not captured by the classification alone. [ABSTRACT FROM AUTHOR]

We derive bounds on the error, in high-order Sobolev norms, incurred in the approximation of Sobolev-regular as well as analytic functions by neural networks with the hyperbolic tangent activation function. These bounds provide explicit estimates on the approximation error with respect to the size of the neural networks. We show that tanh neural networks with only two hidden layers suffice to approximate functions at comparable or better rates than much deeper ReLU neural networks. • Explicit bounds for function approximation in Sobolev norms by tanh neural networks. • Tanh networks with 2 hidden layers are at least as expressive as deeper ReLU networks. • Improved convergence rate for neural network approximation of analytic functions. [ABSTRACT FROM AUTHOR]

In this paper, we aim to solve the optimal tracking control problem for a class of nonaffine discrete-time systems with actuator saturation. First, a data-based neural identifier is constructed to learn the unknown system dynamics. Then, according to the expression of the trained neural identifier, we can obtain the steady control corresponding to the reference trajectory. Next, by involving the iterative dual heuristic dynamic programming algorithm, the new costate function and the tracking control law are developed. Two other neural networks are used to estimate the costate function and approximate the tracking control law. Considering approximation errors of neural networks, the stability analysis of the proposed algorithm for the specific systems is provided by introducing the Lyapunov approach. Finally, via conducting simulation and comparison, the superiority of the developed optimal tracking method is confirmed. Moreover, the trajectory tracking performance of the wastewater treatment application is also involved for further verifying the proposed approach. [ABSTRACT FROM AUTHOR]

We study the computation complexity of deep ReLU (Rectified Linear Unit) neural networks for the approximation of functions from the Hölder–Zygmund space of mixed smoothness defined on the d -dimensional unit cube when the dimension d may be very large. The approximation error is measured in the norm of isotropic Sobolev space. For every function f from the Hölder–Zygmund space of mixed smoothness, we explicitly construct a deep ReLU neural network having an output that approximates f with a prescribed accuracy ɛ , and prove tight dimension-dependent upper and lower bounds of the computation complexity of the approximation, characterized as the size and depth of this deep ReLU neural network, explicitly in d and ɛ. The proof of these results in particular, relies on the approximation by sparse-grid sampling recovery based on the Faber series. [ABSTRACT FROM AUTHOR]

Numerical simulation of wildland fire spread is useful to predict the locations that are likely to burn and to support decision in an operational context, notably for crisis situations and long-term planning. For short-term, the computational time of traditional simulators is too high to be tractable over large zones like a country or part of a country, especially for fire danger mapping. This issue is tackled by emulating the area of the burned surface returned after simulation of a fire igniting anywhere in Corsica island and spreading freely during one hour, with a wide range of possible environmental input conditions. A deep neural network with a hybrid architecture is used to account for two types of inputs: the spatial fields describing the surrounding landscape and the remaining scalar inputs. After training on a large simulation dataset, the network shows a satisfactory approximation error on a complementary test dataset with a MAPE of 32.8%. The convolutional part is pre-computed and the emulator is defined as the remaining part of the network, saving significant computational time. On a 32-core machine, the emulator has a speed-up factor of several thousands compared to the simulator and the overall relationship between its inputs and output is consistent with the expected physical behavior of fire spread. This reduction in computational time allows the computation of one-hour burned area map for the whole island of Corsica in less than a minute, opening new application in short-term fire danger mapping. • Wildfires lasting one hour are simulated in a wide range of environmental conditions. • A large simulation database is used to train a hybrid deep neural network emulator. • The emulator estimates the area burned with a speed-up factor of several thousands • The emulator results are consistent with typical behavior of wildland fire spread. • Emulation opens perspectives for fast short-term fire danger high-resolution mapping. [ABSTRACT FROM AUTHOR]

We examine the closedness of sets of realized neural networks of a fixed architecture in Sobolev spaces. For an exactly m -times differentiable activation function ρ , we construct a sequence of neural networks (Φ n) n ∈ N whose realizations converge in order- (m − 1) Sobolev norm to a function that cannot be realized exactly by a neural network. Thus, sets of realized neural networks are not closed in order- (m − 1) Sobolev spaces W m − 1 , p for p ∈ [ 1 , ∞). We further show that these sets are not closed in W m , p under slightly stronger conditions on the m th derivative of ρ. For a real analytic activation function, we show that sets of realized neural networks are not closed in W k , p for any k ∈ N. The nonclosedness allows for approximation of non-network target functions with unbounded parameter growth. We partially characterize the rate of parameter growth for most activation functions by showing that a specific sequence of realized neural networks can approximate the activation function's derivative with weights increasing inversely proportional to the L p approximation error. Finally, we present experimental results showing that networks are capable of closely approximating non-network target functions with increasing parameters via training. [ABSTRACT FROM AUTHOR]

The accuracy of deep learning, i.e., deep neural networks, can be characterized by dividing the total error into three main types: approximation error, optimization error, and generalization error. Whereas there are some satisfactory answers to the problems of approximation and optimization, much less is known about the theory of generalization. Most existing theoretical works for generalization fail to explain the performance of neural networks in practice. To derive a meaningful bound, we study the generalization error of neural networks for classification problems in terms of data distribution and neural network smoothness. We introduce the cover complexity (CC) to measure the difficulty of learning a data set and the inverse of the modulus of continuity to quantify neural network smoothness. A quantitative bound for expected accuracy/error is derived by considering both the CC and neural network smoothness. Although most of the analysis is general and not specific to neural networks, we validate our theoretical assumptions and results numerically for neural networks by several data sets of images. The numerical results confirm that the expected error of trained networks scaled with the square root of the number of classes has a linear relationship with respect to the CC. We also observe a clear consistency between test loss and neural network smoothness during the training process. In addition, we demonstrate empirically that the neural network smoothness decreases when the network size increases whereas the smoothness is insensitive to training dataset size. [ABSTRACT FROM AUTHOR]

Non-negative matrix factorization (NMF) is a knowledge discovery method that is used in many fields. Variational inference and Gibbs sampling methods for it are also well-known. However, the variational approximation error has not been clarified yet, because NMF is not statistically regular and the prior distribution used in variational Bayesian NMF (VBNMF) has zero or divergence points. In this paper, using algebraic geometrical methods, we theoretically analyze the difference in negative log evidence (a.k.a. free energy) between VBNMF and Bayesian NMF, i.e., the Kullback–Leibler divergence between the variational posterior and the true posterior. We derive an upper bound for the learning coefficient (a.k.a. the real log canonical threshold) in Bayesian NMF. By using the upper bound, we find a lower bound for the approximation error, asymptotically. The result quantitatively shows how well the VBNMF algorithm can approximate Bayesian NMF; the lower bound depends on the hyperparameters and the true non-negative rank. A numerical experiment demonstrates the theoretical result. • Variational inference approximates the posterior using mean-field approximation. • We give a lower bound for the approximation error by using algebraic geometry. • The lower bound depends on the true non-negative rank and the hyperparameters. [ABSTRACT FROM AUTHOR]

The time-delay-based reservoir computing setup has seen tremendous success in both experiment and simulation. It allows for the construction of large neuromorphic computing systems with only few components. However, until now the interplay of the different timescales has not been investigated thoroughly. In this manuscript, we investigate the effects of a mismatch between the time-delay and the clock cycle for a general model. Typically, these two time scales are considered to be equal. Here we show that the case of equal or resonant time-delay and clock cycle could be actively detrimental and leads to an increase of the approximation error of the reservoir. In particular, we can show that non-resonant ratios of these time scales have maximal memory capacities. We achieve this by translating the periodically driven delay-dynamical system into an equivalent network. Networks that originate from a system with resonant delay-times and clock cycles fail to utilize all of their degrees of freedom, which causes the degradation of their performance. [ABSTRACT FROM AUTHOR]

This paper is motivated by an open problem around deep networks, namely, the apparent absence of over-fitting despite large over-parametrization which allows perfect fitting of the training data. In this paper, we analyze this phenomenon in the case of regression problems when each unit evaluates a periodic activation function. We argue that the minimal expected value of the square loss is inappropriate to measure the generalization error in approximation of compositional functions in order to take full advantage of the compositional structure. Instead, we measure the generalization error in the sense of maximum loss, and sometimes, as a pointwise error. We give estimates on exactly how many parameters ensure both zero training error as well as a good generalization error. We prove that a solution of a regularization problem is guaranteed to yield a good training error as well as a good generalization error and estimate how much error to expect at which test data. [ABSTRACT FROM AUTHOR]

Given a function dictionary D and an approximation budget N ∈ N , nonlinear approximation seeks the linear combination of the best N terms { T n } 1 ≤ n ≤ N ⊆ D to approximate a given function f with the minimum approximation error ε L , f ≔ min { g n } ⊆ R , { T n } ⊆ D ‖ f (x) − ∑ n = 1 N g n T n (x) ‖. Motivated by recent success of deep learning, we propose dictionaries with functions in a form of compositions, i.e., T (x) = T (L) ∘ T (L − 1) ∘ ⋯ ∘ T (1) (x) for all T ∈ D , and implement T using ReLU feed-forward neural networks (FNNs) with L hidden layers. We further quantify the improvement of the best N -term approximation rate in terms of N when L is increased from 1 to 2 or 3 to show the power of compositions. In the case when L > 3 , our analysis shows that increasing L cannot improve the approximation rate in terms of N. In particular, for any function f on [ 0 , 1 ] , regardless of its smoothness and even the continuity, if f can be approximated using a dictionary when L = 1 with the best N -term approximation rate ε L , f = O (N − η) , we show that dictionaries with L = 2 can improve the best N -term approximation rate to ε L , f = O (N − 2 η). We also show that for Hölder continuous functions of order α on [ 0 , 1 ] d , the application of a dictionary with L = 3 in nonlinear approximation can achieve an essentially tight best N -term approximation rate ε L , f = O (N − 2 α ∕ d). Finally, we show that dictionaries consisting of wide FNNs with a few hidden layers are more attractive in terms of computational efficiency than dictionaries with narrow and very deep FNNs for approximating Hölder continuous functions if the number of computer cores is larger than N in parallel computing. [ABSTRACT FROM AUTHOR]

Single hidden layer feedforward neural networks (SLFNs) with fixed weights possess the universal approximation property provided that approximated functions are univariate. But this phenomenon does not lay any restrictions on the number of neurons in the hidden layer. The more this number, the more the probability of the considered network to give precise results. In this note, we constructively prove that SLFNs with the fixed weight 1 and two neurons in the hidden layer can approximate any continuous function on a compact subset of the real line. The proof is implemented by a step by step construction of a universal sigmoidal activation function. This function has nice properties such as computability, smoothness and weak monotonicity. The applicability of the obtained result is demonstrated in various numerical examples. Finally, we show that SLFNs with fixed weights cannot approximate all continuous multivariate functions. [ABSTRACT FROM AUTHOR]

Limitations of approximation capabilities of shallow perceptron networks are investigated. Lower bounds on approximation errors are derived for binary-valued functions on finite domains. It is proven that unless the number of network units is sufficiently large (larger than any polynomial of the logarithm of the size of the domain) a good approximation cannot be achieved for almost any uniformly randomly chosen function on a given domain. The results are obtained by combining probabilistic Chernoff–Hoeffding bounds with estimates of the sizes of sets of functions exactly computable by shallow networks with increasing numbers of units. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, the problem of learning the functional dependency between input and output variables from scattered data using fractional polynomial models (FPM) is investigated. The estimation error bounds are obtained by calculating the pseudo-dimension of FPM, which is shown to be equal to that of sparse polynomial models (SPM). A linear decay of the approximation error is obtained for a class of target functions which are dense in the space of continuous functions. We derive a structural risk analogous to the Schwartz Criterion and demonstrate theoretically that the model minimizing this structural risk can achieve a favorable balance between estimation and approximation errors. An empirical model selection comparison is also performed to justify the usage of this structural risk in selecting the optimal complexity index from the data. We show that the construction of FPM can be efficiently addressed by the variable projection method. Furthermore, our empirical study implies that FPM could attain better generalization performance when compared with SPM and cubic splines. [Copyright &y& Elsevier]

The paper "New study on neural networks: the essential order of approximation" by Jianjun Wang and Zongben Xu, which appeared in Neural Networks 23 (2010), deals with upper and lower estimates for the error of best approximation with sums of nearly exponential type activation functions in terms of moduli of smoothness. In particular, the presented lower bound is astonishingly good. However, the proof is incorrect and the bound is wrong. [ABSTRACT FROM AUTHOR]