Search

Estimates of PM2.5 levels are crucial for monitoring air quality and studying the epidemiological impact of air quality on the population. Currently, the most precise measurements of PM2.5 are obtained from ground stations, resulting in limited spatial coverage. In this study, we consider satellite-based PM2.5 retrieval, which involves conversion of high-resolution satellite retrieval of Aerosol Optical Depth (AOD) into high-resolution PM2.5 retrieval. To improve the accuracy of the AOD to PM2.5 conversion, we employ the machine learning based post-process correction to correct the AOD-to-PM conversion ratio derived from Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2) reanalysis model data. The post-process correction approach utilizes a fusion and downscaling of satellite observation and retrieval data, MERRA-2 reanalysis data, various high resolution geographical indicators, meteorological data and ground station observations for learning a predictor for the approximation error in the AOD to PM2.5 conversion ratio. The corrected conversion ratio is then applied to estimate PM2.5 levels given the high-resolution satellite AOD retrieval data derived from Sentinel-3 observations. Our model produces PM2.5 estimates with a spatial resolution of 100 meters at satellite overpass times. Additionally, we have incorporated an ensemble of neural networks to provide error envelopes for machine learning related uncertainty in the PM2.5 estimates. [ABSTRACT FROM AUTHOR]

The choice of the factorization rank of a matrix is critical, e.g., in dimensionality reduction, filtering, clustering, deconvolution, etc., because selecting a rank that is too high amounts to adjusting the noise, while selecting a rank that is too low results in the oversimplification of the signal. Numerous methods for selecting the factorization rank of a non-negative matrix have been proposed. One of them is the cophenetic correlation coefficient ( c c c ), widely used in data science to evaluate the number of clusters in a hierarchical clustering. In previous work, it was shown that c c c performs better than other methods for rank selection in non-negative matrix factorization (NMF) when the underlying structure of the matrix consists of orthogonal clusters. In this article, we show that using the ratio of c c c to the approximation error significantly improves the accuracy of the rank selection. We also propose a new criterion, c o n c o r d a n c e , which, like c c c , benefits from the stochastic nature of NMF; its accuracy is also improved by using its ratio-to-error form. Using real and simulated data, we show that c o n c o r d a n c e , with a CUSUM-based automatic detection algorithm for its original or ratio-to-error forms, significantly outperforms c c c . It is important to note that the new criterion works for a broader class of matrices, where the underlying clusters are not assumed to be orthogonal. [ABSTRACT FROM AUTHOR]

As we know, feedforward neural networks (FNNs) with sigmoidal activation function are universal approximators. Theoretically, for any continuous function defined on a compact set, there exists an FNN such that the FNN can approximate the function with arbitrary accuracy, which constitutes a theoretical guarantee that FNN can be used as an efficient learning machine. This paper addresses the construction and approximation for FNNs. We construct an FNN with sigmoidal activation function and estimate its approximation error. In particular, an inverse theorem of the approximation is established, which implies the equivalence characterization theorem of the approximation and reveals the relationship between the topological structure of the FNN and its approximation ability. As keys in this study, the concepts of modulus of continuity of function, K-functional, and their relationship are utilized, and two Bernstein-type inequalities are established. [ABSTRACT FROM AUTHOR]

Homomorphic encryption (HE) is the ultimate tool for performing secure computations even in untrusted environments. Application of HE for deep learning (DL) inference is an active area of research, given the fact that DL models are often deployed in untrusted environments (e.g., third-party servers) yet inferring on private data. However, existing HE libraries [somewhat (SWHE), leveled (LHE) or fully homomorphic (FHE)] suffer from extensive computational and memory overhead. Few performance optimized high-speed homomorphic libraries are either suffering from certain approximation issues leading to decryption errors or proven to be insecure according to recent published attacks. In this article, we propose architectural tricks to achieve performance speedup for encrypted DL inference developed with exact HE schemes without any approximation or decryption error in homomorphic computations. The main idea is to apply quantization and suitable data packing in the form of bitslicing to reduce the costly noise handling operation, Bootstrapping while achieving a functionally correct and highly parallel DL pipeline with a moderate memory footprint. Experimental evaluation on the MNIST dataset shows a significant ($37\times$) speedup over the nonbitsliced versions of the same architecture. Low memory bandwidths (700 MB) of our design pipelines further highlight their promise toward scaling over larger gamut of Edge-AI analytics use cases. [ABSTRACT FROM AUTHOR]

The chaotic systems have extensive applications in various branches of engineering problems such as financial problems, image processing, secure communications, and medical problems, among many others. In most applications, a synchronization needs to be made with another favorite chaotic system, or output trajectories track the desired signal. The dynamics of these systems are complicated, they are very sensitive to the initial conditions, and they exhibit a stochastic unpredictable behavior. In this study, a new robust type-3 fuzzy logic control (T3-FLC) is designed that can be applied for a large case of chaotic systems under faulty actuators and unknown perturbed dynamics. The dynamic uncertainties are estimated by the online learned type-3 fuzzy logic systems (T3-FLSs). The rules of T3-FLS are optimized by the Lyapunov theorem. The actuator nonlinearities are identified by a new method. The effects of approximation error (AE), dynamic perturbations and unknown time-varying control gains are tackled by the designed adaptive compensator. The designed compensator is constructed by online estimation of the upper bound of AE. By several simulations and comparison with the new FLS-based controllers, the better performance of the designed T3-FLC is shown. In addition, the performance of the designed controller is examined in a secure communication system. [ABSTRACT FROM AUTHOR]

This paper develops fundamental limits of deep neural network learning by characterizing what is possible if no constraints are imposed on the learning algorithm and on the amount of training data. Concretely, we consider Kolmogorov-optimal approximation through deep neural networks with the guiding theme being a relation between the complexity of the function (class) to be approximated and the complexity of the approximating network in terms of connectivity and memory requirements for storing the network topology and the associated quantized weights. The theory we develop establishes that deep networks are Kolmogorov-optimal approximants for markedly different function classes, such as unit balls in Besov spaces and modulation spaces. In addition, deep networks provide exponential approximation accuracy—i.e., the approximation error decays exponentially in the number of nonzero weights in the network—of the multiplication operation, polynomials, sinusoidal functions, and certain smooth functions. Moreover, this holds true even for one-dimensional oscillatory textures and the Weierstrass function—a fractal function, neither of which has previously known methods achieving exponential approximation accuracy. We also show that in the approximation of sufficiently smooth functions finite-width deep networks require strictly smaller connectivity than finite-depth wide networks. [ABSTRACT FROM AUTHOR]

This paper proposes a dynamic sliding mode control (SMC) approach to the robust voltage regulation of dc–dc boost converters by using interval type-2 fuzzy neural networks (IT2FNNs). First, uncertainties caused by the perturbation of the input inductor and the output capacitor are represented with some bounded approximation errors, by the utilization of a Takagi–Sugeno (T–S) fuzzy modeling approach. Based on the represented model of the boost converter, a new type of sliding surface is designed depending on the duty cycle and reference inputs of the converter. Then, a dynamic SMC law is designed, by considering that the perturbation of the uncertain parameters, including input inductor, output capacitor, load resistor, and input voltage, is bounded. Meanwhile, we adopt an exponential plus power approaching law in the sliding mode controller for fast reachability of the sliding surface and a small chattering in the duty cycle input. Moreover, in terms of the considered uncertainties, a novel IT2FNN-based dynamic SMC law is derived, by applying simplified ellipsoidal-type membership functions in the type-2 fuzzy neural network. To improve the capacity to manage the uncertainties, some online learning algorithms for the updating of the IT2FNN are designed by a gradient descent method (GDM), without the requirement of the boundedness of the uncertainties. The resulting tracking error system is synthesized to be bounded stable based on the designed IT2FNN-based dynamic SMC. Finally, the effectiveness of the proposed adaptive IT2FNN-based dynamic SMC method is verified by some comparative simulation results. [ABSTRACT FROM AUTHOR]

Learning of the cell-load in radio access networks (RANs) has to be performed within a short time period. Therefore, we propose a learning framework that is robust against uncertainties resulting from the need for learning based on a relatively small training set. To this end, we incorporate prior knowledge about the cell-load in the learning framework. For example, an inherent property of the cell-load is that it is monotonic in downlink (data) rates. To obtain additional prior knowledge we first study the feasible rate region, i.e., the set of all vectors of user rates that can be supported by the network. We prove that the feasible rate region is compact. Moreover, we show the existence of a Lipschitz function that maps feasible rate vectors to cell-load vectors. With these results in hand, we present a learning technique that guarantees a minimum approximation error in the worst-case scenario by using prior knowledge and a small training sample set. Simulations in the network simulator NS3 demonstrate that the proposed method exhibits better robustness and accuracy than standard learning techniques, especially for small training sample sets. [ABSTRACT FROM AUTHOR]

The frequent directions (FD) technique is a deterministic approach for online sketching that has many applications in machine learning. The conventional FD is a heuristic procedure that often outputs rank deficient matrices. To overcome the rank deficiency problem, we propose a new sketching strategy called robust frequent directions (RFD) by introducing a regularization term. RFD can be derived from an optimization problem. It updates the sketch matrix and the regularization term adaptively and jointly. RFD reduces the approximation error of FD without increasing the computational cost. We also apply RFD to online learning and propose an effective hyperparameterfree online Newton algorithm. We derive a regret bound for our online Newton algorithm based on RFD, which guarantees the robustness of the algorithm. The experimental studies demonstrate that the proposed method outperforms state-of-the-art second order online learning algorithms. [ABSTRACT FROM AUTHOR]

Domain transfer learning generalizes a learning model across training data and testing data with different distributions. A general principle to tackle this problem is reducing the distribution difference between training data and testing data such that the generalization error can be bounded. Current methods typically model the sample distributions in input feature space, which depends on nonlinear feature mapping to embody the distribution discrepancy. However, this nonlinear feature space may not be optimal for the kernel-based learning machines. To this end, we propose a transfer kernel learning (TKL) approach to learn a domain-invariant kernel by directly matching source and target distributions in the reproducing kernel Hilbert space (RKHS). Specifically, we design a family of spectral kernels by extrapolating target eigensystem on source samples with Mercer’s theorem. The spectral kernel minimizing the approximation error to the ground truth kernel is selected to construct domain-invariant kernel machines. Comprehensive experimental evidence on a large number of text categorization, image classification, and video event recognition datasets verifies the effectiveness and efficiency of the proposed TKL approach over several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The linear parameter-varying (LPV) model is widely used in aero engine control system design. The conventional local modeling method is inaccurate and inefficient in the full flying envelope. Hence, a novel online data-driven LPV modeling method based on the online sequential extreme learning machine (OS-ELM) with an additional multiplying layer (MLOS-ELM) was proposed. An extra multiplying layer was inserted between the hidden layer and the output layer, where the hidden layer outputs were multiplied by the input variables and state variables of the LPV model. Additionally, the input layer was set to the LPV model's scheduling parameter. With the multiplying layer added, the state space equation matrices of the LPV model could be easily calculated using online gathered data. Simulation results showed that the outputs of the MLOS-ELM matched that of the component level model of a turbo-shaft engine precisely. The maximum approximation error was less than 0.18%. The predictive outputs of the proposed online data-driven LPV model after five samples also matched that of the component level model well, and the maximum predictive error within a large flight envelope was less than 1.1% with measurement noise considered. Thus, the efficiency and accuracy of the proposed method were validated. [ABSTRACT FROM AUTHOR]

Abstract: Dropout is a recently introduced algorithm for training neural networks by randomly dropping units during training to prevent their co-adaptation. A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates. The framework allows a complete analysis of the ensemble averaging properties of dropout in linear networks, which is useful to understand the non-linear case. The ensemble averaging properties of dropout in non-linear logistic networks result from three fundamental equations: (1) the approximation of the expectations of logistic functions by normalized geometric means, for which bounds and estimates are derived; (2) the algebraic equality between normalized geometric means of logistic functions with the logistic of the means, which mathematically characterizes logistic functions; and (3) the linearity of the means with respect to sums, as well as products of independent variables. The results are also extended to other classes of transfer functions, including rectified linear functions. Approximation errors tend to cancel each other and do not accumulate. Dropout can also be connected to stochastic neurons and used to predict firing rates, and to backpropagation by viewing the backward propagation as ensemble averaging in a dropout linear network. Moreover, the convergence properties of dropout can be understood in terms of stochastic gradient descent. Finally, for the regularization properties of dropout, the expectation of the dropout gradient is the gradient of the corresponding approximation ensemble, regularized by an adaptive weight decay term with a propensity for self-consistent variance minimization and sparse representations. [Copyright &y& Elsevier]

Expectation Propagation (EP) provides a framework for approximate inference. When the model under consideration is over a latent Gaussian field, with the approximation being Gaussian, we show how these approximations can systematically be corrected. A perturbative expansion is made of the exact but intractable correction, and can be applied to the model's partition function and other moments of interest. The correction is expressed over the higher-order cumulants which are neglected by EP's local matching of moments. Through the expansion, we see that EP is correct to first order. By considering higher orders, corrections of increasing polynomial complexity can be applied to the approximation. The second order provides a correction in quadratic time, which we apply to an array of Gaussian process and Ising models. The corrections generalize to arbitrarily complex approximating families, which we illustrate on tree-structured Ising model approximations. Furthermore, they provide a polynomial-time assessment of the approximation error. We also provide both theoretical and practical insights on the exactness of the EP solution. [ABSTRACT FROM AUTHOR]

Economic efficient Autonomous Road Vehicles (ARVs) are invariably subjected to uncertainties and perturbations. Therefore, control of vehicle systems requires stability to withstand the effect of variations in the nominal performance. Lateral path-tracking is a substantial task of ARVs, especially in critical maneuvering and cornering with variable speed. In this study, a new controller on the basis of interval type-3 (T3) fuzzy logic system (FLSs) is designed. The main novelties and advantages are as follows. (1) The uncertainty is a main challenge in the path-following problem of ARVs. However, in the fuzzy-based approaches, the bounds of uncertainty are assumed to be known. However, in the our suggested approach, the bounds of uncertainties are also fuzzy sets and type-3 FLSs with online adaptation rules are suggested to handle the uncertainties. (2) The approximation errors (AEs) and perturbations are investigated and tackled by the compensators. (3) The bounds of estimation errors are also uncertain and are estimated by the suggested adaptation laws. (4) The stability is ensured under unknown dynamics, perturbations and critical maneuvers. (5) Comparison with the benchmarking techniques and conventional fuzzy approaches verifies that the suggested path-following scheme results in better maneuver performance. [ABSTRACT FROM AUTHOR]

We consider the minimum error entropy (MEE) criterion and an empirical risk minimization learning algorithm when an approximation of Rényi's entropy (of order 2) by Parzen windowing is minimized. This learning algorithm involves a Parzen windowing scaling parameter. We present a learning theory approach for this MEE algorithm in a regression setting when the scaling parameter is large. Consistency and explicit convergence rates are provided in terms of the approximation ability and capacity of the involved hypothesis space. Novel analysis is carried out for the generalization error associated with Rényi's entropy and a Parzen windowing function, to overcome technical difficulties arising from the essential differences between the classical least squares problems and the MEE setting. An involved symmetrized least squares error is introduced and analyzed, which is related to some ranking algorithms. [ABSTRACT FROM AUTHOR]

We study the density problem and approximation error of reproducing kernel Hilbert spaces for the purpose of learning theory. For a Mercer kernel K on a compact metric space (X, d), a characterization for the generated reproducing kernel Hilbert space (RKHS) HK to be dense in C(X) is given. As a corollary, we show that the density is always true for convolution type kernels. Some estimates for the rate of convergence of interpolation schemes are presented for general Mercer kernels. These are then used to establish for convolution type kernels quantitative analysis for the approximation error in learning theory. Finally, we show by the example of Gaussian kernels with varying variances that the approximation error can be improved when we adaptively change the value of the parameter for the used kernel. This confirms the method of choosing varying parameters which is used often in many applications of learning theory. [ABSTRACT FROM AUTHOR]

We consider the minimum error entropy (MEE) criterion and an empirical risk minimization learning algorithm when an approximation of Rényi's entropy (of order 2) by Parzen windowing is minimized. This learning algorithm involves a Parzen windowing scaling parameter. We present a learning theory approach for this MEE algorithm in a regression setting when the scaling parameter is large. Consistency and explicit convergence rates are provided in terms of the approximation ability and capacity of the involved hypothesis space. Novel analysis is carried out for the generalization error associated with Rényi's entropy and a Parzen windowing function, to overcome technical difficulties arising from the essential differences between the classical least squares problems and the MEE setting. An involved symmetrized least squares error is introduced and analyzed, which is related to some ranking algorithms. [ABSTRACT FROM AUTHOR]

This paper aims to consider approximation-estimation tests for decision-making by machine-learning methods, and integral-estimation tests are defined, which is a generalization for the continuous case. Approximation-estimation tests are measurable sampling functions (statistics) that estimate the approximation error of monotonically increasing number sequences in different classes of functions. These tests make it possible to determine the Markov moments of a qualitative change in the increase in such sequences, from linear to nonlinear type. If these sequences are trajectories of discrete quasi-deterministic random processes, then moments of change in the nature of their growth and qualitative change in the process match up. For example, in cluster analysis, approximation-estimation tests are a formal generalization of the "elbow method" heuristic. In solid mechanics, they can be used to determine the proportionality limit for the stress strain curve (boundaries of application of Hooke's law). In molecular biology methods, approximation-estimation tests make it possible to determine the beginning of the exponential phase and the transition to the plateau phase for the curves of fluorescence accumulation of the real-time polymerase chain reaction, etc. [ABSTRACT FROM AUTHOR]

Deep neural networks have successfully been trained in various application areas with stochastic gradient descent. However, there exists no rigorous mathematical explanation why this works so well. The training of neural networks with stochastic gradient descent has four different discretization parameters: (i) the network architecture; (ii) the amount of training data; (iii) the number of gradient steps; and (iv) the number of randomly initialized gradient trajectories. While it can be shown that the approximation error converges to zero if all four parameters are sent to infinity in the right order, we demonstrate in this paper that stochastic gradient descent fails to converge for ReLU networks if their depth is much larger than their width and the number of random initializations does not increase to infinity fast enough. [ABSTRACT FROM AUTHOR]

We propose a conservative physics-informed neural network (cPINN) on discrete domains for nonlinear conservation laws. Here, the term discrete domain represents the discrete sub-domains obtained after division of the computational domain, where PINN is applied and the conservation property of cPINN is obtained by enforcing the flux continuity in the strong form along the sub-domain interfaces. In case of hyperbolic conservation laws, the convective flux contributes at the interfaces, whereas in case of viscous conservation laws, both convective and diffusive fluxes contribute. Apart from the flux continuity condition, an average solution (given by two different neural networks) is also enforced at the common interface between two sub-domains. One can also employ a deep neural network in the domain, where the solution may have complex structure, whereas a shallow neural network can be used in the sub-domains with relatively simple and smooth solutions. Another advantage of the proposed method is the additional freedom it gives in terms of the choice of optimization algorithm and the various training parameters like residual points, activation function, width and depth of the network etc. Various forms of errors involved in cPINN such as optimization, generalization and approximation errors and their sources are discussed briefly. In cPINN, locally adaptive activation functions are used, hence training the model faster compared to its fixed counterparts. Both, forward and inverse problems are solved using the proposed method. Various test cases ranging from scalar nonlinear conservation laws like Burgers, Korteweg–de Vries (KdV) equations to systems of conservation laws, like compressible Euler equations are solved. The lid-driven cavity test case governed by incompressible Navier–Stokes equation is also solved and the results are compared against a benchmark solution. The proposed method enjoys the property of domain decomposition with separate neural networks in each sub-domain, and it efficiently lends itself to parallelized computation, where each sub-domain can be assigned to a different computational node. • Domain decomposition technique is proposed for PINNs with tailored neural network in each sub-domain for solving conservation laws. • Due to presence of multiple neural networks, the representation capacity of the proposed cPINN method increases. • Based on prior knowledge of the solution regularity in each sub-domain, the hyper-parameter set of corresponding PINN can be properly adjusted. • The partial independence of individual PINNs in decomposed domains can be further employed to implement cPINN in a parallelized algorithm. [ABSTRACT FROM AUTHOR]

This paper deals with a control design problem for a diesel engine air path system that has strong nonlinearity and requires multi-input and multi-output control to satisfy requirements and constraints. We focus on a neural network based approximation of nonlinear model predictive control (NMPC) for high-speed computation. Most neural approximation methods are verified only through simulation; further, the influence of approximation on the closed-loop performance has been not sufficiently discussed. In this study, we discuss this influence, and propose a new method to improve stability against degradation due to an approximation error. The control system is assembled using a neural network based controller, obtained by the proposed method, and an unscented Kalman filter. This system is verified both numerically and experimentally; the results demonstrate the capability of the proposed method to track the boost pressure, EGR rate, and pumping loss according to the reference values, and satisfy the constraints of compressor surge and choke. The high computation speed that can be achieved using a standard on-board ECU is also demonstrated using the approximated controller. • The proposed method realizes fast computation of nonlinear MPC. • It approximates optimal control inputs with high accuracy using a neural network. • It provides a new method to cope with stability degradation due to approximation. • Its effectiveness is validated through an experiment using a real diesel engine. • It can be applied to any other continuous variable system. [ABSTRACT FROM AUTHOR]