Search

In wind speed prediction technologies, deep learning-based methods have achieved promising advantages. However, most existing methods focus on learning implicit knowledge in a data-driven manner but neglect some explicit knowledge from the physical theory of meteorological dynamics, failing to make stable and long-term predictions. In this paper, we explore introducing explicit physical knowledge into neural networks and propose Physical Equations Predictive Network (PEPNet) for multi-step wind speed predictions. In PEPNet, a new neural block called the Augmented Neural Barotropic Equations (ANBE) block is designed as its key component, which aims to capture the wind dynamics by combining barotropic primitive equations and deep neural networks. Specifically, the ANBE block adopts a two-branch structure to model wind dynamics, where one branch is physic-based and the other is data-driven-based. The physic-based branch constructs temporal partial derivatives of meteorological elements (including u-component wind, v-component wind, and geopotential height) in a new Neural Barotropic Equations Unit (NBEU). The NBEU is developed based on the barotropic primitive equations mode in numerical weather prediction (NWP). Besides, considering that the barotropic primitive mode is a crude assumption of atmospheric motion, another data-driven-based branch is developed in the ANBE block, which aims at capturing meteorological dynamics beyond barotropic primitive equations. Finally, the PEPNet follows a time-variant structure to enhance the model's capability to capture wind dynamics over time. To evaluate the predictive performance of PEPNet, we have conducted several experiments on two real-world datasets. Experimental results show that the proposed method outperforms the state-of-the-art techniques and achieve optimal performance. [ABSTRACT FROM AUTHOR]

The problem of vanishing and exploding gradients has been a long-standing obstacle that hinders the effective training of neural networks. Despite various tricks and techniques that have been employed to alleviate the problem in practice, there still lacks satisfactory theories or provable solutions. In this paper, we address the problem from the perspective of high-dimensional probability theory. We provide a rigorous result that shows, under mild conditions, how the vanishing/exploding gradients problem disappears with high probability if the neural networks have sufficient width. Our main idea is to constrain both forward and backward signal propagation in a nonlinear neural network through a new class of activation functions, namely Gaussian–Poincaré normalized functions, and orthogonal weight matrices. Experiments on both synthetic and real-world data validate our theory and confirm its effectiveness on very deep neural networks when applied in practice. • The vanishing/exploding gradient problem in training deep neural networks is addressed. • The problem is provably solved under mild conditions using high-dimensional probability theory. • Experiments show neural network of 200 layers can be trained without linearizing the networks. [ABSTRACT FROM AUTHOR]

Weight-tied models have attracted attention in the modern development of neural networks. The deep equilibrium model (DEQ) represents infinitely deep neural networks with weight-tying, and recent studies have shown the potential of this type of approach. DEQs are needed to iteratively solve root-finding problems in training and are built on the assumption that the underlying dynamics determined by the models converge to a fixed point. In this paper, we present the stable invariant model (SIM) , a new class of deep models that in principle approximates DEQs under stability and extends the dynamics to more general ones converging to an invariant set (not restricted in a fixed point). The key ingredient in deriving SIMs is a representation of the dynamics with the spectra of the Koopman and Perron–Frobenius operators. This perspective approximately reveals stable dynamics with DEQs and then derives two variants of SIMs. We also propose an implementation of SIMs that can be learned in the same way as feedforward models. We illustrate the empirical performance of SIMs with experiments and demonstrate that SIMs achieve comparative or superior performance against DEQs in several learning tasks. • A Koopman spectral analysis is performed for deep equilibrium models. • The analysis identifies stable dynamics determined by deep equilibrium models. • We develop a class of deep models to approximate the stable dynamics. • Models in the class can represent dynamics converging to an invariant set. [ABSTRACT FROM AUTHOR]

Despite the popularism of Bayesian neural networks (BNNs) in recent years, its use is somewhat limited in complex and big data situations due to the computational cost associated with full posterior evaluations. Variational Bayes (VB) provides a useful alternative to circumvent the computational cost and time complexity associated with the generation of samples from the true posterior using Markov Chain Monte Carlo (MCMC) techniques. The efficacy of the VB methods is well established in machine learning literature. However, its potential broader impact is hindered due to a lack of theoretical validity from a statistical perspective. In this paper, we establish the fundamental result of posterior consistency for the mean-field variational posterior (VP) for a feed-forward artificial neural network model. The paper underlines the conditions needed to guarantee that the VP concentrates around Hellinger neighborhoods of the true density function. Additionally, the role of the scale parameter and its influence on the convergence rates has also been discussed. The paper mainly relies on two results (1) the rate at which the true posterior grows (2) the rate at which the Kullback–Leibler (KL) distance between the posterior and variational posterior grows. The theory provides a guideline for building prior distributions for BNNs along with an assessment of accuracy of the corresponding VB implementation. [ABSTRACT FROM AUTHOR]

The application of neural network models to solve combinatorial optimization has recently drawn much attention and shown promising results in dealing with similar problems, like Travelling Salesman Problem. The neural network allows to learn solutions based on given problem instances, using reinforcement learning or supervised learning. In this paper, we present a novel end-to-end method to solve routing problems. In specific, we propose a gated cosine-based attention model (GCAM) to train policies, which accelerates the training process and the convergence of policy. Extensive experiments on different scale of routing problems show that the proposed method can achieve faster convergence of the training process than the state-of-the-art deep learning models while achieving solutions of the same quality. • We proposed a cosine-based neural network model based on the transformer architecture to improve node embeddings. • We use cosine decomposability to accelerate training speed of the proposed model. • We use a gating connection to stabilize training and get faster convergence, which leads to better overall performance. • We apply the proposed gated cosine-based attention model to solve several Routing Problems. [ABSTRACT FROM AUTHOR]

Deep Neural Networks are widely used for solving complex problems in several scientific areas, such as speech recognition, machine translation, image analysis. The strategies employed to investigate their theoretical properties mainly rely on Euclidean geometry, but in the last years new approaches based on Riemannian geometry have been developed. Motivated by some open problems, we study a particular sequence of maps between manifolds, with the last manifold of the sequence equipped with a Riemannian metric. We investigate the structures induced through pullbacks on the other manifolds of the sequence and on some related quotients. In particular, we show that the pullbacks of the final Riemannian metric to any manifolds of the sequence is a degenerate Riemannian metric inducing a structure of pseudometric space. We prove that the Kolmogorov quotient of this pseudometric space yields a smooth manifold, which is the base space of a particular vertical bundle. We investigate the theoretical properties of the maps of such sequence, eventually we focus on the case of maps between manifolds implementing neural networks of practical interest and we present some applications of the geometric framework we introduced in the first part of the paper. • We consider Neural Networks as sequences of smooth maps between manifolds. • We investigate the structures induced through pull-backs on the manifolds. • The pull-backs of the final Riemannian metric to any manifolds is a degenerate metric. • The theoretical study leads to construct equivalence classes in the input manifold. [ABSTRACT FROM AUTHOR]

Abstract This paper carries out a theoretical investigation of the class of neutral-type delayed Cohen–Grossberg neural networks by using the Lyapunov stability theory. By employing a suitable Lyapunov functional candidate, we derive some new delay independent sufficient conditions for the global asymptotic stability of the equilibrium point for the neutral-type Cohen–Grossberg neural networks with time delays. The obtained stability conditions can be completely characterized by the networks parameters of the neutral systems under consideration. Therefore, it is easy to verify the applicability of our results by simply using some algebraic manipulations of the conditions. Some numerical examples are also given to show the effectiveness of the derived analytical results. A detailed comparison between our proposed results and recently reported corresponding stability results is also made, revealing that the conditions given in this paper establish a new set of stability criteria for Neutral-Type Cohen–Grossberg Neural Networks. [ABSTRACT FROM AUTHOR]

This paper mainly deals with the problem of designing delayed state-feedback controller for neural networks with leakage delay. By constructing an appropriate Lyapunov–Krasovskii functional including double integral terms having two different exponential decay rates and utilizing linear matrix inequality (LMI) technique with the help of slack variables, some sufficient conditions for globally exponential stabilization results are obtained by designing of delayed state-feedback controller. The novelty of this paper includes: (i) although many papers dealt with dynamics of neural networks with leakage delay, there is little work on design of feedback controller. Even there is almost no result that deals with the delay-feedback controller for such delay systems, which is the main motivation of this paper; (ii) the derived delay-dependent stability criteria establish the relationship between leakage delay and time delay in the feedback term, which can be easily checked via the MATLAB LMI toolbox; (iii) we consider the general case that the control term is given in the form of B u , where B is an n × m real matrix. Such case is more difficult to handle than the special case that B is unit matrix. It indicates that the controller can be used in all states or in some states by proper selection of the matrix B . The effectiveness of the proposed method in this paper is illustrated via numerical examples. [ABSTRACT FROM AUTHOR]

The paper considers a general neural networks model with variable-time impulses. It is shown that each solution of the system intersects with every discontinuous surface exactly once via several new well-proposed assumptions. Moreover, based on the comparison principle, this paper shows that neural networks with variable-time impulse can be reduced to the corresponding neural network with fixed-time impulses under well-selected conditions. Meanwhile, the fixed-time impulsive systems can be regarded as the comparison system of the variable-time impulsive neural networks. Furthermore, a series of sufficient criteria are derived to ensure the existence and global exponential stability of periodic solution of variable-time impulsive neural networks, and to illustrate the same stability properties between variable-time impulsive neural networks and the fixed-time ones. The new criteria are established by applying Schaefer’s fixed point theorem combined with the use of inequality technique. Finally, a numerical example is presented to show the effectiveness of the proposed results. [ABSTRACT FROM AUTHOR]

In this paper a new design of neural networks is introduced, which is able to generate oscillatory patterns. The fundamental building block of the neural network is O-neurons that can generate an oscillation in its transfer functions. Since the natural policy gradient learning has been used in training a central pattern generator paradigm, it is called Natural Learner CPG Neural Networks (NLCPGNN). O-neurons are connected and coupled to each other in order to shape a network and their unknown parameters are found by a natural policy gradient learning algorithm. The main contribution of this paper is design of this learning algorithm which is able to simultaneously search for the weights and topology of the network. This system is capable to obtain any complex motion and rhythmic trajectory via first layer and learn rhythmic trajectories in the second layer and converge towards all these movements. Moreover this two layers system is able to provide various features of a learner model for instance resistance against perturbations, modulation of trajectories amplitude and frequency. Simulation of the learning system in the robot simulator (WEBOTS) that is linked with MATLAB software has been done. Implementation on a real NAO robot demonstrates that the robot has learned desired motion with high accuracy. These results show proposed system produces high convergence rate and low test errors. [ABSTRACT FROM AUTHOR]

In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest in the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as trainable , learnable or adaptable activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constrains the corresponding weight layers. [ABSTRACT FROM AUTHOR]

In this paper, the synchronization problem for neural networks with stochastic perturbation is studied with intermittent control via adaptive aperiodicity. Under the framework of stochastic theory and Lyapunov stability method, we develop some techniques of intermittent control with adaptive aperiodicity to achieve the synchronization of a class of neural networks, modeled by stochastic systems. Some effective sufficient conditions are established for the realization of synchronization of the underlying network. Numerical simulations of two examples are provided to illustrate the theoretical results obtained in the paper. [ABSTRACT FROM AUTHOR]

In this paper, a novel integral reinforcement learning (IRL)-based event-triggered adaptive dynamic programming scheme is developed for input-saturated continuous-time nonlinear systems. By using the IRL technique, the learning system does not require the knowledge of the drift dynamics. Then, a single critic neural network is designed to approximate the unknown value function and its learning is not subjected to the requirement of an initial admissible control. In order to reduce computational and communication costs, the event-triggered control law is designed. The triggering threshold is given to guarantee the asymptotic stability of the control system. Two examples are employed in the simulation studies, and the results verify the effectiveness of the developed IRL-based event-triggered control method. [ABSTRACT FROM AUTHOR]

This paper is concerned with the periodic synchronization problem for a general class of delayed neural networks (DNNs) with discontinuous neuron activation. One of the purposes is to analyze the problem of periodic orbits. To do so, we introduce new tools including inequality techniques and Kakutani’s fixed point theorem of set-valued maps to derive the existence of periodic solution. Another purpose is to design a switching state-feedback control for realizing global exponential synchronization of the drive–response network system with periodic coefficients. Unlike the previous works on periodic synchronization of neural network, both the neuron activations and controllers in this paper are allowed to be discontinuous. Moreover, owing to the occurrence of delays in neuron signal, the neural network model is described by the functional differential equation. So we introduce extended Filippov-framework to deal with the basic issues of solutions for discontinuous DNNs. Finally, two examples and simulation experiments are given to illustrate the proposed method and main results which have an important instructional significance in the design of periodic synchronized DNNs circuits involving discontinuous or switching factors. [ABSTRACT FROM AUTHOR]

This paper studies the problem of robust stability of dynamical neural networks with discrete time delays under the assumptions that the network parameters of the neural system are uncertain and norm-bounded, and the activation functions are slope-bounded. By employing the results of Lyapunov stability theory and matrix theory, new sufficient conditions for the existence, uniqueness and global asymptotic stability of the equilibrium point for delayed neural networks are presented. The results reported in this paper can be easily tested by checking some special properties of symmetric matrices associated with the parameter uncertainties of neural networks. We also present a numerical example to show the effectiveness of the proposed theoretical results. [ABSTRACT FROM AUTHOR]

This paper is concerned with the problem of coexistence and dynamical behaviors of multiple equilibrium points for neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays. The fixed point theorem and other analytical tools are used to develop certain sufficient conditions that ensure that the n -dimensional discontinuous neural networks with time-varying delays can have at least 5 n equilibrium points, 3 n of which are locally stable and the others are unstable. The importance of the derived results is that it reveals that the discontinuous neural networks can have greater storage capacity than the continuous ones. Moreover, different from the existing results on multistability of neural networks with discontinuous activation functions, the 3 n locally stable equilibrium points obtained in this paper are located in not only saturated regions, but also unsaturated regions, due to the non-monotonic structure of discontinuous activation functions. A numerical simulation study is conducted to illustrate and support the derived theoretical results. [ABSTRACT FROM AUTHOR]

In this paper, based on the knowledge of memristor and recurrent neural networks (RNNs), the model of the memristor-based RNNs with discrete and distributed delays is established. By constructing proper Lyapunov functionals and using inequality technique, several sufficient conditions are given to ensure the passivity of the memristor-based RNNs with discrete and distributed delays in the sense of Filippov solutions. The passivity conditions here are presented in terms of linear matrix inequalities, which can be easily solved by using Matlab Tools. In addition, the results of this paper complement and extend the earlier publications. Finally, numerical simulations are employed to illustrate the effectiveness of the obtained results. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have achieved outstanding performance in a wide range of applications, e.g., image classification, natural language processing, etc. Despite the good performance, the huge number of parameters in DNNs brings challenges to efficient training of DNNs and also their deployment in low-end devices with limited computing resources. In this paper, we explore the correlations in the weight matrices, and approximate the weight matrices with the low-rank block-term tensors. We name the new corresponding structure as block-term tensor layers (BT-layers), which can be easily adapted to neural network models, such as CNNs and RNNs. In particular, the inputs and the outputs in BT-layers are reshaped into low-dimensional high-order tensors with a similar or improved representation power. Sufficient experiments have demonstrated that BT-layers in CNNs and RNNs can achieve a very large compression ratio on the number of parameters while preserving or improving the representation power of the original DNNs. [ABSTRACT FROM AUTHOR]

A neural network model upon differential inclusion is designed for solving the complex-variables convex programming, and the chain rule for real-valued function with the complex-variables is established in this paper. The model does not need to choose penalty parameters when applied to practical problems, which makes it easier to design. The result is obtained that its state reaches the feasible region in finite time. Furthermore, the convergence for its state to an optimal solution is proved. Some typical examples are shown for the effectiveness of the designed model. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a neural network framework for semi-supervised clustering with pairwise (must-link or cannot-link) constraints. In contrast to existing approaches, we decompose semi-supervised clustering into two simpler classification tasks: the first stage uses a pair of Siamese neural networks to label the unlabeled pairs of points as must-link or cannot-link; the second stage uses the fully pairwise-labeled dataset produced by the first stage in a supervised neural-network-based clustering method. The proposed approach is motivated by the observation that binary classification (such as assigning pairwise relations) is usually easier than multi-class clustering with partial supervision. On the other hand, being classification-based, our method solves only well-defined classification problems, rather than less well specified clustering tasks. Extensive experiments on various datasets demonstrate the high performance of the proposed method. [ABSTRACT FROM AUTHOR]

Abstract: The paper considers nonsmooth neural networks described by a class of differential inclusions termed differential variational inequalities (DVIs). The DVIs include the relevant class of neural networks, introduced by Li, Michel and Porod, described by linear systems evolving in a closed hypercube of . The main result in the paper is a necessary and sufficient condition for multistability of DVIs with nonsymmetric and cooperative (nonnegative) interconnections between neurons. The condition is easily checkable and provides a sharp bound between DVIs that can store multiple patterns, as asymptotically stable equilibria, and those for which this is not possible. Numerical examples and simulations are presented to confirm and illustrate the theoretic findings. [Copyright &y& Elsevier]

Abstract: This paper addresses the design of a discontinuous finite time convergent learning law for neural networks with continuous dynamics. The neural network was used here to obtain a non-parametric model for uncertain systems described by a set of ordinary differential equations. The source of uncertainties was the presence of some external perturbations and poor knowledge of the nonlinear function describing the system dynamics. A new adaptive algorithm based on discontinuous algorithms was used to adjust the weights of the neural network. The adaptive algorithm was derived by means of a non-standard Lyapunov function that is lower semi-continuous and differentiable in almost the whole space. A compensator term was included in the identifier to reject some specific perturbations using a nonlinear robust algorithm. Two numerical examples demonstrated the improvements achieved by the learning algorithm introduced in this paper compared to classical schemes with continuous learning methods. The first one dealt with a benchmark problem used in the paper to explain how the discontinuous learning law works. The second one used the methane production model to show the benefits in engineering applications of the learning law proposed in this paper. [Copyright &y& Elsevier]

Abstract: The Pareto-optimality concept is used in this paper in order to represent a constrained set of solutions that are able to trade-off the two main objective functions involved in neural networks supervised learning: data-set error and network complexity. The neural network is described as a dynamic system having error and complexity as its state variables and learning is presented as a process of controlling a learning trajectory in the resulting state space. In order to control the trajectories, sliding mode dynamics is imposed to the network. It is shown that arbitrary learning trajectories can be achieved by maintaining the sliding mode gains within their convergence intervals. Formal proofs of convergence conditions are therefore presented. The concept of trajectory learning presented in this paper goes further beyond the selection of a final state in the Pareto set, since it can be reached through different trajectories and states in the trajectory can be assessed individually against an additional objective function. [Copyright &y& Elsevier]

Abstract: This paper studies the existence, uniqueness and global asymptotic stability of the equilibrium point for the class of neural networks with discrete constant time delays under parameter uncertainties. The class of the neural network considered in this paper employs the activation functions which are assumed to be continuous and slope-bounded but not required to be bounded or differentiable. We conduct a stability analysis by exploiting the stability theory of Lyapunov functionals and the theory of Homomorphic mapping to derive some easily verifiable sufficient conditions for existence, uniqueness and global asymptotic stability of the equilibrium point. The conditions obtained mainly establish some time-independent relationships between the network parameters of the neural network. We make a detailed comparison between our results and the previously published corresponding results. This comparison proves that our results are new and improve and generalize the results derived in the past literature. We also give some illustrative numerical examples to show the effectiveness and applicability of our proposed stability results. [Copyright &y& Elsevier]

This paper deals with the fixed-time synchronization problem of coupled delayed neural networks with discontinuous activations. Based on pinning control, a discontinuous controller is firstly proposed to guarantee that coupled neural networks achieve synchronization with a desired trajectory in finite time. Then, a discontinuous fixed-time controller is designed. With the fixed-time controller, the settling time can be estimated regardless of initial conditions. By providing a topology-dependent Lyapunov function, some criteria of finite-/fixed-time synchronization are derived. Finally, two numerical examples are given to show the effectiveness of the proposed controllers. [ABSTRACT FROM AUTHOR]

This paper introduces novel deep architectures using the hybrid neural-kernel core model as the first building block. The proposed models follow a combination of a neural networks based architecture and a kernel based model enriched with pooling layers. In particular, in this context three kernel blocks with average, maxout and convolutional pooling layers are introduced and examined. We start with a simple merging layer which averages the output of the previous representation layers. The maxout layer on the other hand triggers competition among different representations of the input. Thanks to this pooling layer, not only the dimensionality of the output of multi-scale representations is reduced but also multiple sub-networks are formed within the same model. In the same context, the pointwise convolutional layer is also employed with the aim of projecting the multi-scale representations onto a new space. Experimental results show an improvement over the core deep hybrid model as well as kernel based models on several real-life datasets. • Three novel average, maxout and convolutional kernel blocks are introduced. • Their corresponding deep architectures are developed using a stacking scheme. • Training and regularizations techniques for the developed models are discussed. • The numerical experiments on small to large scale real-life datasets show the effectiveness of the proposed architectures. [ABSTRACT FROM AUTHOR]

Abstract The essential purpose of this work is to conduct an investigation into stability problem for the class of neutral-type Cohen–Grossberg neural networks including multiple time delays in states and multiple neutral delays in time derivative of states. By proposing an appropriate Lyapunov functional, a new sufficient criterion is derived for global asymptotic stability of neutral-type neural networks. The obtained stability criterion is independent of time delay and neutral delay parameters, and it is completely stated in terms of the elements of interconnection matrices and other network parameters. Thus, this newly presented stability condition can be validated by simply examining some algebraic equations establishing some relationships among the system parameters and matrices of the neutral-type neural system. A constructive example is presented to indicate applicability of the obtained sufficient stability condition. Since stability analysis of the class of neutral-type neural networks considered in this work has not been given much attention due to the difficulty of developing efficient mathematical techniques and methods enabling to conduct a stability analysis of such neutral-type neural systems, the criterion proposed in this paper can be considered as a leading stability result for neutral-type Cohen–Grossberg neural systems including multiple time and multiple neutral delays. [ABSTRACT FROM AUTHOR]

Abstract This paper is concerned with the fixed-time synchronization control of inertial memristor-based neural networks with discrete delay. We design four different kinds of feedback controllers, under which the considered inertial memristor-based neural networks can realize fixed-time synchronization perfectly. Moreover, the obtained fixed-time synchronization criteria can be verified by algebraic operations. For any initial synchronization error, the settling time of fixed-time synchronization is bounded by a fixed constant, which can be calculated beforehand based on system parameters and controller parameters. Numerical simulations are given to illustrate the effectiveness of our theoretical results. [ABSTRACT FROM AUTHOR]

Abstract Information needs to be appropriately encoded to be reliably transmitted over physical media. Similarly, neurons have their own codes to convey information in the brain. Even though it is well-known that neurons exchange information using a pool of several protocols of spatio-temporal encodings, the suitability of each code and their performance as a function of network parameters and external stimuli is still one of the great mysteries in neuroscience. This paper sheds light on this by modeling small-size networks of chemically and electrically coupled Hindmarsh–Rose spiking neurons. We focus on a class of temporal and firing-rate codes that result from neurons' membrane-potentials and phases, and quantify numerically their performance estimating the Mutual Information Rate, aka the rate of information exchange. Our results suggest that the firing-rate and interspike-intervals codes are more robust to additive Gaussian white noise. In a network of four interconnected neurons and in the absence of such noise, pairs of neurons that have the largest rate of information exchange using the interspike-intervals and firing-rate codes are not adjacent in the network, whereas spike-timings and phase codes (temporal) promote large rate of information exchange for adjacent neurons. If that result would have been possible to extend to larger neural networks, it would suggest that small microcircuits would preferably exchange information using temporal codes (spike-timings and phase codes), whereas on the macroscopic scale, where there would be typically pairs of neurons not directly connected due to the brain's sparsity, firing-rate and interspike-intervals codes would be the most efficient codes. [ABSTRACT FROM AUTHOR]

Abstract: This paper considers a class of online gradient learning methods for backpropagation (BP) neural networks with a single hidden layer. We assume that in each training cycle, each sample in the training set is supplied in a stochastic order to the network exactly once. It is interesting that these stochastic learning methods can be shown to be deterministically convergent. This paper presents some weak and strong convergence results for the learning methods, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. The conditions on the activation function and the learning rate to guarantee the convergence are relaxed compared with the existing results. Our convergence results are valid for not only S–S type neural networks (both the output and hidden neurons are Sigmoid functions), but also for P–P, P–S and S–P type neural networks, where S and P represent Sigmoid and polynomial functions, respectively. [ABSTRACT FROM AUTHOR]

Abstract Deep-Learning has become a leading strategy for artificial intelligence and is being applied in many fields due to its excellent performance that has surpassed human cognitive abilities in a number of classification and control problems (Ciregan, Meier, & Schmidhuber, 2012; Mnih et al., 2015). However, the training process of Deep-Learning is usually slow and requires high-performance computing, capable of handling large datasets. The optimization of the training method can improve the learning rate of the Deep-Learning networks and result in a higher performance while using the same number of training epochs (cycles). This paper considers the use of estimation theory for training of large neural networks and in particular Deep-Learning networks. Two estimation strategies namely the Extended Kalman Filter (EKF) and the Smooth Variable Structure Filter (SVSF) have been revised (subsequently referred to as RSVSF and REKF) and used for network training. They are applied to several benchmark datasets and comparatively evaluated. Highlights • Two Deep-Learning training algorithms are introduced, called REKF and RSVSF. • REKF and RSVSF allow Deep-Learning networks to learn fast. • REKF and RSVSF use less memory than current adaptive methods. [ABSTRACT FROM AUTHOR]

Abstract: This paper studies the general neural networks dynamical systems with variable coefficients and time-varying delays. By applying the Young inequality technique, Dini derivative and introducing many real parameters, and estimating the upper bound of solutions of the system, a series of new and useful criteria on the boundedness, global exponential stability and the existence and global exponential stability of the periodic solutions are established. Particularly, when the system degenerates into the autonomous case, the new criteria on the existence, uniqueness and global exponential stability of the equilibrium points are obtained. The results obtained in this paper extend and generalize the corresponding results existing in previous literature. [Copyright &y& Elsevier]

This paper presents a methodology that reflected functions by reflecting the weight matrices of an artificial neural network. One of the major problems with the connectionist approach is that trained neural networks can only associate fixed sets of input–output mappings. We provide a methodology which allows the post-trained net to associate different input–output mappings. The different mappings are reflected in a horizontal axis, reflected in a vertical axis and scaling of the initial mapping. The methodology does not train the net on the different mappings but it transforms the weight matrix of the neural network. This paper describes a novel way of utilising sigma–pi neural networks. Our new methodology manipulates sigma–pi unit''s weight matrices which transform the unit''s output. The weights are cast in a matrix formulation, and then transformations can be performed on the weight matrix of the sigma–pi net. To test the new methodology, the following three steps were carried out on a neural network: (1) the network was trained to perform a mapping function, f; (2) the weights of the network were transformed; and (3) the network was tested to evaluate whether it performs the reflection in the vertical axis, fref−vert(x)=a−f(x). This reflects the function in one dimension. A reflection transformation was used to manipulate the network''s weight matrices to obtain a reflection in the vertical axis. Note that the network was not trained to perform the reflection in the vertical axis. The transformation of the weight matrix transformed the function the output performs. This article explains the theory which enables us to perform transformations of sigma–pi networks and obtain reflections of the output by reflecting the weight matrices. These transforms empower the network to perform related mapping tasks once one mapping task has been learnt. This article explains how each transformation is performed and it considers whether a set of ‘standard’ transformations can indeed be derived. [Copyright &y& Elsevier]

Abstract This paper is concerned with the problem of passivity for uncertain neural networks with time-varying delays. First, the recently developed integral inequality called generalized free-matrix-based integral inequality is extended to estimate further tight lower bound of integral terms. By constructing a suitable augmented LKF, an enhanced passivity condition for the concerned network is derived in terms of linear matrix inequalities (LMIs). Here, the integral terms having three states in its quadratic form is estimated by the proposed Lemma. As special cases of main results, for neural networks without uncertainties, passivity and stability conditions are derived. Through three numerical examples, it will be shown that the developed conditions can promote the level of passivity and stability criteria. [ABSTRACT FROM AUTHOR]

This paper is concerned with leaderless synchronization of coupled delayed neural networks. A distributed event-triggered control strategy under the periodic sampling scheme is introduced to reduce control updates. By introducing a weighted average state as a virtual leader, the leaderless synchronization problem can be transformed to the stability problem of the error system, which is defined as the distance between each node and the virtual leader. A leaderless synchronization criterion under the periodic event-triggered scheme in strongly connected networks is first derived based on Finsler’s lemma. The results are then extended to the network containing a directed spanning tree. A corollary with the lower dimension in the form of LMIs is further presented. Two examples are provided to validate the effectiveness of the theoretical results. [ABSTRACT FROM AUTHOR]

This paper is concerned with the problem of decentralized adaptive backstepping state-feedback control for uncertain high-order large-scale stochastic nonlinear time-delay systems. For the control design of high-order large-scale nonlinear systems, only one adaptive parameter is constructed to overcome the over-parameterization, and neural networks are employed to cope with the difficulties raised by completely unknown system dynamics and stochastic disturbances. And then, the appropriate Lyapunov–Krasovskii functional and the property of hyperbolic tangent functions are used to deal with the unknown unmatched time-delay interactions of high-order large-scale systems for the first time. At last, on the basis of Lyapunov stability theory, the decentralized adaptive neural controller was developed, and it decreases the number of learning parameters. The actual controller can be designed so as to ensure that all the signals in the closed-loop system are semi-globally uniformly ultimately bounded (SGUUB) and the tracking error converges in the small neighborhood of zero. The simulation example is used to further show the validity of the design method. [ABSTRACT FROM AUTHOR]

In this paper, we aim to tackle the neural robust tracking control problem for a class of nonlinear systems using the adaptive critic technique. The main contribution is that a neural-network-based robust tracking control scheme is established for nonlinear systems involving matched uncertainties. The augmented system considering the tracking error and the reference trajectory is formulated and then addressed under adaptive critic optimal control formulation, where the initial stabilizing controller is not needed. The approximate control law is derived via solving the Hamilton–Jacobi–Bellman equation related to the nominal augmented system, followed by closed-loop stability analysis. The robust tracking control performance is guaranteed theoretically via Lyapunov approach and also verified through simulation illustration. [ABSTRACT FROM AUTHOR]

In this paper, we propose a fractional-order neural network system with time-varying delays and reaction–diffusion terms. We first develop a new Mittag-Leffler synchronization strategy for the controlled nodes via impulsive controllers. Using the fractional Lyapunov method sufficient conditions are given. We also study the global Mittag-Leffler synchronization of two identical fractional impulsive reaction–diffusion neural networks using linear controllers, which was an open problem even for integer-order models. Since the Mittag-Leffler stability notion is a generalization of the exponential stability concept for fractional-order systems, our results extend and improve the exponential impulsive control theory of neural network system with time-varying delays and reaction–diffusion terms to the fractional-order case. The fractional-order derivatives allow us to model the long-term memory in the neural networks, and thus the present research provides with a conceptually straightforward mathematical representation of rather complex processes. Illustrative examples are presented to show the validity of the obtained results. We show that by means of appropriate impulsive controllers we can realize the stability goal and to control the qualitative behavior of the states. An image encryption scheme is extended using fractional derivatives. [ABSTRACT FROM AUTHOR]

In this paper, an identifier–critic structure is introduced to find an online near-optimal controller for continuous-time nonaffine nonlinear systems having saturated control signal. By employing two Neural Networks (NNs), the solution of Hamilton–Jacobi–Bellman (HJB) equation associated with the cost function is derived without requiring a priori knowledge about system dynamics. Weights of the identifier and critic NNs are tuned online and simultaneously such that unknown terms are approximated accurately and the control signal is kept between the saturation bounds. The convergence of NNs’ weights, identification error, and system states is guaranteed using Lyapunov’s direct method. Finally, simulation results are performed on two nonlinear systems to confirm the effectiveness of the proposed control strategy. [ABSTRACT FROM AUTHOR]

Stochastic Gradient Descent (SGD) updates Convolutional Neural Network (CNN) with a noisy gradient computed from a random batch, and each batch evenly updates the network once in an epoch. This model applies the same training effort to each batch, but it overlooks the fact that the gradient variance, induced by Sampling Bias and Intrinsic Image Difference, renders different training dynamics on batches. In this paper, we develop a new training strategy for SGD, referred to as Inconsistent Stochastic Gradient Descent (ISGD) to address this problem. The core concept of ISGD is the inconsistent training, which dynamically adjusts the training effort w.r.t the loss. ISGD models the training as a stochastic process that gradually reduces down the mean of batch’s loss, and it utilizes a dynamic upper control limit to identify a large loss batch on the fly. ISGD stays on the identified batch to accelerate the training with additional gradient updates, and it also has a constraint to penalize drastic parameter changes. ISGD is straightforward, computationally efficient and without requiring auxiliary memories. A series of empirical evaluations on real world datasets and networks demonstrate the promising performance of inconsistent training. [ABSTRACT FROM AUTHOR]

This paper is concerned with event-triggered H ∞ filtering for delayed neural networks via sampled data. A novel event-triggered scheme is proposed, which can lead to a significant reduction of the information communication burden in the network; the feature of this scheme is that whether or not the sampled data should be transmitted is determined by the current sampled data and the error between the current sampled data and the latest transmitted data. By constructing a proper Lyapunov–Krasovskii functional, utilizing the reciprocally convex combination technique and Jensen’s inequality sufficient conditions are derived to ensure that the resultant filtering error system is asymptotically stable. Based on the derived H ∞ performance analysis results, the H ∞ filter design is formulated in terms of Linear Matrix Inequalities (LMIs). Finally, the proposed stability conditions are demonstrated with numerical example. [ABSTRACT FROM AUTHOR]

This paper focuses on stability analysis for neural networks systems with time-varying delays. A more general auxiliary function-based integral inequality is established and some improved delay-dependent stability conditions formulated in terms of linear matrix inequalities (LMIs) are derived by employing a suitable Lyapunov–Krasovskii functional (LKF) and the novel integral inequality. Three well-known application examples are provided to demonstrate the effectiveness and improvements of the proposed method. [ABSTRACT FROM AUTHOR]

This paper investigates pinning synchronization of coupled neural networks with both current-state coupling and distributed-delay coupling via impulsive control. A novel impulse pinning strategy involving pinning ratio is proposed and a general criterion is derived to ensure an array of neural networks with two different topologies synchronizes with the desired trajectory. In order to handle the difficulties of high-dimension criteria, some inequality techniques and matrix decomposition methods through simultaneous diagonalization of two matrices are introduced and low-dimensional criteria are obtained. Finally, an illustrative example is given to show the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

This paper is concerned with H ∞ filter design for a class of neural network systems with event-triggered communication scheme and quantization. Firstly, a new event-triggered communication scheme is introduced to determine whether or not the current sampled sensor data should be broadcasted and transmitted to quantizer, which can save the limited communication resource. Secondly, a logarithmic quantizer is used to quantify the sampled data, which can reduce the data transmission rate in the network. Thirdly, considering the influence of the constrained network resource, we investigate the problem of H ∞ filter design for a class of event-triggered neural network systems with quantization. By using Lyapunov functional and linear matrix inequality (LMI) techniques, some delay-dependent stability conditions for the existence of the desired filter are obtained. Furthermore, the explicit expression is given for the designed filter parameters in terms of LMIs. Finally, a numerical example is given to show the usefulness of the obtained theoretical results. [ABSTRACT FROM AUTHOR]

Weight elimination offers a simple and efficient improvement of training algorithm of feedforward neural networks. It is a general regularization technique in terms of the flexible scaling parameters. Actually, the weight elimination technique also contains the weight decay regularization for a large scaling parameter. Many applications of this technique and its improvements have been reported. However, there is little research concentrated on its convergence behavior. In this paper, we theoretically analyze the weight elimination for cyclic learning method and determine the conditions for the uniform boundedness of weight sequence, and weak and strong convergence. Based on the assumed network parameters, the optimal choice for the scaling parameter can also be determined. Moreover, two illustrative simulations have been done to support the theoretical explorations as well. [ABSTRACT FROM AUTHOR]

This paper proposes a new multi-input multi-output (MIMO) transmit scheme aided by artificial neural network (ANN). The morphological perceptron with competitive learning (MP/CL) concept is deployed as a decision rule in the MIMO detection stage. The proposed MIMO transmission scheme is able to achieve double spectral efficiency; hence, in each time-slot the receiver decodes two symbols at a time instead one as Alamouti scheme. Other advantage of the proposed transmit scheme with MP/CL-aided detector is its polynomial complexity according to modulation order, while it becomes linear when the data stream length is greater than modulation order. The performance of the proposed scheme is compared to the traditional MIMO schemes, namely Alamouti scheme and maximum-likelihood MIMO (ML-MIMO) detector. Also, the proposed scheme is evaluated in a scenario with variable channel information along the frame. Numerical results have shown that the diversity gain under space-time coding Alamouti scheme is partially lost, which slightly reduces the bit-error rate (BER) performance of the proposed MP/CL-NN MIMO scheme. [ABSTRACT FROM AUTHOR]

In this paper, the recursive least square algorithm is designed for the big data learning of a feedforward neural network. The proposed method as the combination of the recursive least square and feedforward neural network obtains four advantages over the alone algorithms: it requires less number of regressors, it is fast, it has the learning ability, and it is more compact. Stability, convergence, boundedness of parameters, and local minimum avoidance of the proposed technique are guaranteed. The introduced strategy is applied for the modeling of the crude oil blending process. [ABSTRACT FROM AUTHOR]

This paper is concerned with global exponential stability problem for a class of neural networks with time-varying delays. Using a new proposed inequality called free-matrix-based integral inequality, a less conservative criterion is proposed, which is expressed by linear matrix inequalities. Two numerical examples are given to show the effectiveness and superiority of the obtained criterion. [ABSTRACT FROM AUTHOR]

Cluster synchronization is a typical collective behavior in coupled dynamical systems, where the synchronization occurs within one group, while there is no synchronization among different groups. In this paper, under event-based mechanism, pinning cluster synchronization in an array of coupled neural networks is studied. A new event-triggered sampled-data transmission strategy, where only local and event-triggering states are utilized to update the broadcasting state of each agent, is proposed to realize cluster synchronization of the coupled neural networks. Furthermore, a self-triggered pinning cluster synchronization algorithm is proposed, and a set of iterative procedures is given to compute the event-triggered time instants. Hence, this will reduce the computational load significantly. Finally, an example is given to demonstrate the effectiveness of the theoretical results. [ABSTRACT FROM AUTHOR]

In this paper, a learning algorithm is developed for Dynamic Plastic Continuous Neural Networks (DPCNNs) to improve their learning of highly nonlinear time dependent problems. A DPCNN is comprised of a base medium, which is nonlinear and plastic, and a number of neurons that are attached to the base by wire-like connections similar to perceptrons. The information is distributed within DPCNNs gradually and through wave propagation mechanism. While a DPCNN is adaptive due to its connection weights, the material properties of its base medium can also be adjusted to improve its learning. The material of the medium is plastic and can contribute to memorizing the history of input-response similar to neuroplasticity in natural brain. The results obtained from numerical simulation of DPCNNs have been encouraging. Nonlinear plastic finite element modeling has been used for numerical simulation of dynamic behavior and wave propagation in the medium. Two significant differences of DPCNNs with other types of neural networks are that: (1) there is a medium to which the neurons are attached where the medium can contribute to the learning, (2) the input layer is not made of nodes but it is an edge terminal which is capable of receiving a continuous function over the input edge, though it is discretized in the finite element model. A DPCNN is reduced to a perceptron if the medium is removed and the neurons are connected to each other only by wires. Continuity of the input lets the discretization of data take place intrinsically within the DPCNN instead of being applied by the user. [ABSTRACT FROM AUTHOR]