Search

Appropriate weight initialization settings, along with the ReLU activation function, have become cornerstones of modern deep learning, enabling the training and deployment of highly effective and efficient neural network models across diverse areas of artificial intelligence. The problem of "dying ReLU," where ReLU neurons become inactive and yield zero output, presents a significant challenge in the training of deep neural networks with ReLU activation function. Theoretical research and various methods have been introduced to address the problem. However, even with these methods and research, training remains challenging for extremely deep and narrow feedforward networks with ReLU activation function. In this paper, we propose a novel weight initialization method to address this issue. We establish several properties of our initial weight matrix and demonstrate how these properties enable the effective propagation of signal vectors. Through a series of experiments and comparisons with existing methods, we demonstrate the effectiveness of the novel initialization method. [ABSTRACT FROM AUTHOR]

How can deep neural networks encode information that corresponds to words in human speech into raw acoustic data? This paper proposes two neural network architectures for modeling unsupervised lexical learning from raw acoustic inputs: ciwGAN (Categorical InfoWaveGAN) and fiwGAN (Featural InfoWaveGAN). These combine Deep Convolutional GAN architecture for audio data (WaveGAN; Donahue et al., 2019) with the information theoretic extension of GAN – InfoGAN (Chen et al., 2016) – and propose a new latent space structure that can model featural learning simultaneously with a higher level classification and allows for a very low-dimension vector representation of lexical items. In addition to the Generator and Discriminator networks, the architectures introduce a network that learns to retrieve latent codes from generated audio outputs. Lexical learning is thus modeled as emergent from an architecture that forces a deep neural network to output data such that unique information is retrievable from its acoustic outputs. The networks trained on lexical items from the TIMIT corpus learn to encode unique information corresponding to lexical items in the form of categorical variables in their latent space. By manipulating these variables, the network outputs specific lexical items. The network occasionally outputs innovative lexical items that violate training data, but are linguistically interpretable and highly informative for cognitive modeling and neural network interpretability. Innovative outputs suggest that phonetic and phonological representations learned by the network can be productively recombined and directly paralleled to productivity in human speech: a fiwGAN network trained on suit and dark outputs innovative start , even though it never saw start or even a [st] sequence in the training data. We also argue that setting latent featural codes to values well beyond training range results in almost categorical generation of prototypical lexical items and reveals underlying values of each latent code. Probing deep neural networks trained on well understood dependencies in speech bears implications for latent space interpretability and understanding how deep neural networks learn meaningful representations, as well as potential for unsupervised text-to-speech generation in the GAN framework. [ABSTRACT FROM AUTHOR]

This paper mainly deals with the problem of designing delayed state-feedback controller for neural networks with leakage delay. By constructing an appropriate Lyapunov–Krasovskii functional including double integral terms having two different exponential decay rates and utilizing linear matrix inequality (LMI) technique with the help of slack variables, some sufficient conditions for globally exponential stabilization results are obtained by designing of delayed state-feedback controller. The novelty of this paper includes: (i) although many papers dealt with dynamics of neural networks with leakage delay, there is little work on design of feedback controller. Even there is almost no result that deals with the delay-feedback controller for such delay systems, which is the main motivation of this paper; (ii) the derived delay-dependent stability criteria establish the relationship between leakage delay and time delay in the feedback term, which can be easily checked via the MATLAB LMI toolbox; (iii) we consider the general case that the control term is given in the form of B u , where B is an n × m real matrix. Such case is more difficult to handle than the special case that B is unit matrix. It indicates that the controller can be used in all states or in some states by proper selection of the matrix B . The effectiveness of the proposed method in this paper is illustrated via numerical examples. [ABSTRACT FROM AUTHOR]

This paper investigates the anti-synchronization of complex-valued memristor-based neural networks with time delays via designed external controllers. By constructing appropriate Lyapunov functions and using inequality technique, two different types of controllers are derived to guarantee the exponential anti-synchronization of complex-valued memristor-based delayed neural networks. Compared with existing relevant results, the proposed results of this paper are more general and less conservative. In addition, the presented theoretical results are easy to be checked with the parameters of systems themselves. Finally, an example with numerical simulations illustrates the effectiveness of the obtained results. [ABSTRACT FROM AUTHOR]

In this paper, without separating the complex-valued neural networks into two real-valued systems, the quasi-projective synchronization of fractional-order complex-valued neural networks is investigated. First, two new fractional-order inequalities are established by using the theory of complex functions, Laplace transform and Mittag-Leffler functions, which generalize traditional inequalities with the first-order derivative in the real domain. Additionally, different from hybrid control schemes given in the previous work concerning the projective synchronization, a simple and linear control strategy is designed in this paper and several criteria are derived to ensure quasi-projective synchronization of the complex-valued neural networks with fractional-order based on the established fractional-order inequalities and the theory of complex functions. Moreover, the error bounds of quasi-projective synchronization are estimated. Especially, some conditions are also presented for the Mittag-Leffler synchronization of the addressed neural networks. Finally, some numerical examples with simulations are provided to show the effectiveness of the derived theoretical results. [ABSTRACT FROM AUTHOR]

The effects of leakage delay on the dynamics of neural networks with integer-order have lately been received considerable attention. It has been confirmed that fractional neural networks more appropriately uncover the dynamical properties of neural networks, but the results of fractional neural networks with leakage delay are relatively few. This paper primarily concentrates on the issue of bifurcation for high-order fractional bidirectional associative memory(BAM) neural networks involving leakage delay. The first attempt is made to tackle the stability and bifurcation of high-order fractional BAM neural networks with time delay in leakage terms in this paper. The conditions for the appearance of bifurcation for the proposed systems with leakage delay are firstly established by adopting time delay as a bifurcation parameter. Then, the bifurcation criteria of such system without leakage delay are successfully acquired. Comparative analysis wondrously detects that the stability performance of the proposed high-order fractional neural networks is critically weakened by leakage delay, they cannot be overlooked. Numerical examples are ultimately exhibited to attest the efficiency of the theoretical results. [ABSTRACT FROM AUTHOR]

Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics. In this paper, we take the visual recognition task as a case study to answer the questions of " what workloads are ideal for SNNs and how to evaluate SNNs makes sense ". We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics. [ABSTRACT FROM AUTHOR]

Abstract Digital images of rock samples have been using extensively in Digital Rock Physics (DRP) to evaluate physical parameters of rock such as permeability, P- and S-wave velocities and formation factor. The parameters are numerically computed by simulation of the corresponding physical processes through segmented image of rock, which provide a direct and accurate evaluation of rock properties. However, recent advances in machine learning and Convolutional Neural Networks (CNN) allow using images as input. Such networks, however, require a considerable number of images as input. In this paper, CNNs are used to estimate the P- and S-wave velocities from images of rock medium. To deal with lack of input data, a hybrid pattern- and pixel-based simulation (HYPPS) is used as an efficient data augmentation method to increase the training data set. For each input image, 10 stochastic realizations are produced. Compare to the case wherein the stochastic models are not used, the new results from the enhanced network indicate a sharp improvement in the estimations such that R 2 is increased to 0.94. Furthermore, the newly developed CNN network, unlike the one with the small data set (R 2 = 0. 75), manifests no over/underestimation. The estimated properties, in comparison with the computational results, indicate that CNNs perform outstandingly in predicting the physical parameters of rock without conducting any time-demanding forward modeling if enough input data are provided. [ABSTRACT FROM AUTHOR]

Abstract In this paper, we explore the coexistence and dynamical behaviors of multiple equilibrium points for fractional-order competitive neural networks with Gaussian activation functions. By virtue of the geometrical properties of activation functions, the fixed point theorem and the theory of fractional-order differential equation, some sufficient conditions are established to guarantee that such n -neuron neural networks have exactly 3 k equilibrium points with 0 ≤ k ≤ n , among which 2 k equilibrium points are locally Mittag-Leffler stable. The obtained results cover both multistability and mono-stability of fractional-order neural networks and integer-order neural networks. Two illustrative examples with their computer simulations are presented to verify the theoretical analysis. Highlights • FOCNNs with Gaussian activation functions are introduced. • Coexistence and local stability of multiple equilibrium points are investigated. • n -neuron networks have exactly 3 k equilibrium points with 0 ≤ k ≤ n. • 2 k equilibrium points are locally Mittag-Leffler stable. • The obtained results cover both multistability and mono-stability. [ABSTRACT FROM AUTHOR]

Abstract Plasticity is one of the most important properties of the nervous system, which enables animals to adjust their behavior to the ever-changing external environment. Changes in synaptic efficacy between neurons constitute one of the major mechanisms of plasticity. Therefore, estimation of neural connections is crucial for investigating information processing in the brain. Although many analysis methods have been proposed for this purpose, most of them suffer from one or all the following mathematical difficulties: (1) only partially observed neural activity is available; (2) correlations can include both direct and indirect pseudo-interactions; and (3) biological evidence that a neuron typically has only one type of connection (excitatory or inhibitory) should be considered. To overcome these difficulties, a novel probabilistic framework for estimating neural connections from partially observed spikes is proposed in this paper. First, based on the property of a sum of random variables, the proposed method estimates the influence of unobserved neurons on observed neurons and extracts only the correlations among observed neurons. Second, the relationship between pseudo-correlations and target connections is modeled by neural propagation in a multiplicative manner. Third, a novel information-theoretic framework is proposed for estimating neuron types. The proposed method was validated using spike data generated by artificial neural networks. In addition, it was applied to multi-unit data recorded from the CA1 area of a rat's hippocampus. The results confirmed that our estimates are consistent with previous reports. These findings indicate that the proposed method is useful for extracting crucial interactions in neural signals as well as in other multi-probed point process data. [ABSTRACT FROM AUTHOR]

Abstract Biological neural networks are systems of extraordinary computational capabilities shaped by evolution, development, and lifelong learning. The interplay of these elements leads to the emergence of biological intelligence. Inspired by such intricate natural phenomena, Evolved Plastic Artificial Neural Networks (EPANNs) employ simulated evolution in-silico to breed plastic neural networks with the aim to autonomously design and create learning systems. EPANN experiments evolve networks that include both innate properties and the ability to change and learn in response to experiences in different environments and problem domains. EPANNs' aims include autonomously creating learning systems, bootstrapping learning from scratch, recovering performance in unseen conditions, testing the computational advantages of particular neural components, and deriving hypotheses on the emergence of biological learning. Thus, EPANNs may include a large variety of different neuron types and dynamics, network architectures, plasticity rules, and other factors. While EPANNs have seen considerable progress over the last two decades, current scientific and technological advances in artificial neural networks are setting the conditions for radically new approaches and results. Exploiting the increased availability of computational resources and of simulation environments, the often challenging task of hand-designing learning neural networks could be replaced by more autonomous and creative processes. This paper brings together a variety of inspiring ideas that define the field of EPANNs. The main methods and results are reviewed. Finally, new opportunities and possible developments are presented. [ABSTRACT FROM AUTHOR]

This paper discusses octonion-valued neural networks (OVNNs) with leakage delay, time-varying delays, and distributed delays, for which the states, weights, and activation functions belong to the normed division algebra of octonions. The octonion algebra is a nonassociative and noncommutative generalization of the complex and quaternion algebras, but does not belong to the category of Clifford algebras, which are associative. In order to avoid the nonassociativity of the octonion algebra and also the noncommutativity of the quaternion algebra, the Cayley–Dickson construction is used to decompose the OVNNs into 4 complex-valued systems. By using appropriate Lyapunov–Krasovskii functionals, with double and triple integral terms, the free weighting matrix method, and simple and double integral Jensen inequalities, delay-dependent criteria are established for the exponential stability of the considered OVNNs. The criteria are given in terms of complex-valued linear matrix inequalities, for two types of Lipschitz conditions which are assumed to be satisfied by the octonion-valued activation functions. Finally, two numerical examples illustrate the feasibility, effectiveness, and correctness of the theoretical results. [ABSTRACT FROM AUTHOR]

To solve dynamic Sylvester equation in the presence of additive noises, a novel recurrent neural network (NRNN) with finite-time convergence and excellent robustness is proposed and analyzed in this paper. As compared with the design process of Zhang neural network (ZNN), the proposed NRNN is based on an ingenious integral design formula activated by nonlinear functions, which are able to expedite the convergence speed and suppress unknown additive noises during the solving process of dynamic Sylvester equation. In addition, the global stability, finite-time convergence and denoising property of the NRNN model are theoretically proved. The upper bound of the finite convergence time for the NRNN model is also estimated in theory. Simulative results further verify the efficiency of the NRNN model, as well as its superior robust and finite-time performance to the conventional ZNN model for dynamic Sylvester equation in front of additive noises. At last, the proposed design method for establishing the NRNN model is successfully applied to kinematical control of robotic manipulator in front of additive noises. [ABSTRACT FROM AUTHOR]

Recently, the hybrid neuron models which combine the basic neuron models with impulsive effect(the state reset process) had been proposed, however, the preset value and the reset value of membrane potential were both fixed constants in the known models. In this paper, the Hindmarsh–Rose neuron model with nonlinear reset process is presented where the preset value and the reset value of membrane potential are variable constants. We conduct a qualitative analysis in the vicinity of the equilibrium point or the limit cycle of the proposed system by using the theories of impulsive semi-dynamical systems. Firstly, the more detailed impulsive set and phase set are given, then using the fixed point of Poincaré map, the existences of order-1and order- k ( k > = 2 ) period solutions are investigated subsequently. Furthermore, numerical investigations including period-adding bifurcation, multiple attractors coexistence, switch-like behavior are presented to further describe the bifurcation and chaos phenomena. Finally, the obtained results and possible applications of the proposed model are elaborated. [ABSTRACT FROM AUTHOR]

This paper investigates H ∞ state estimation problem for a class of semi-Markovian jumping discrete-time neural networks model with event-triggered scheme and quantization. First, a new event-triggered communication scheme is introduced to determine whether or not the current sampled sensor data should be broad-casted and transmitted to the quantizer, which can save the limited communication resource. Second, a novel communication framework is employed by the logarithmic quantizer that quantifies and reduces the data transmission rate in the network, which apparently improves the communication efficiency of networks. Third, a stabilization criterion is derived based on the sufficient condition which guarantees a prescribed H ∞ performance level in the estimation error system in terms of the linear matrix inequalities. Finally, numerical simulations are given to illustrate the correctness of the proposed scheme. [ABSTRACT FROM AUTHOR]

This paper deals with the stabilization problem of memristive recurrent neural networks with inertial items, discrete delays, bounded and unbounded distributed delays. First, for inertial memristive recurrent neural networks (IMRNNs) with second-order derivatives of states, an appropriate variable substitution method is invoked to transfer IMRNNs into a first-order differential form. Then, based on nonsmooth analysis theory, several algebraic criteria are established for the global stabilizability of IMRNNs under proposed feedback control, where the cases with both bounded and unbounded distributed delays are successfully addressed. Finally, the theoretical results are illustrated via the numerical simulations. [ABSTRACT FROM AUTHOR]

Nonlinear components extracted from deep structures of bottleneck neural networks exhibit a great ability to express input space in a low-dimensional manifold. Sharing and combining the components boost the capability of the neural networks to synthesize and interpolate new and imaginary data. This synthesis is possibly a simple model of imaginations in human brain where the components are expressed in a nonlinear low dimensional manifold. The current paper introduces a novel Dynamic Deep Bottleneck Neural Network to analyze and extract three main features of videos regarding the expression of emotions on the face. These main features are identity, emotion and expression intensity that are laid in three different sub-manifolds of one nonlinear general manifold. The proposed model enjoying the advantages of recurrent networks was used to analyze the sequence and dynamics of information in videos. It is noteworthy to mention that this model also has also the potential to synthesize new videos showing variations of one specific emotion on the face of unknown subjects. Experiments on discrimination and recognition ability of extracted components showed that the proposed model has an average of 97.77% accuracy in recognition of six prominent emotions (Fear, Surprise, Sadness, Anger, Disgust, and Happiness), and 78.17% accuracy in the recognition of intensity. The produced videos revealed variations from neutral to the apex of an emotion on the face of the unfamiliar test subject which is on average 0.8 similar to reference videos in the scale of the SSIM method. [ABSTRACT FROM AUTHOR]

This paper is concerned with leaderless synchronization of coupled delayed neural networks. A distributed event-triggered control strategy under the periodic sampling scheme is introduced to reduce control updates. By introducing a weighted average state as a virtual leader, the leaderless synchronization problem can be transformed to the stability problem of the error system, which is defined as the distance between each node and the virtual leader. A leaderless synchronization criterion under the periodic event-triggered scheme in strongly connected networks is first derived based on Finsler’s lemma. The results are then extended to the network containing a directed spanning tree. A corollary with the lower dimension in the form of LMIs is further presented. Two examples are provided to validate the effectiveness of the theoretical results. [ABSTRACT FROM AUTHOR]

Video image recognition has been extensively studied with rapid progress recently. However, most methods focus on short-term rather than long-term (contextual) video recognition. Convolutional recurrent neural networks (ConvRNNs) provide robust spatio-temporal information processing capabilities for contextual video recognition, but require extensive computation that slows down training. Inspired by normalization and detrending methods, in this paper we propose “adaptive detrending” (AD) for temporal normalization in order to accelerate the training of ConvRNNs, especially of convolutional gated recurrent unit (ConvGRU). For each neuron in a recurrent neural network (RNN), AD identifies the trending change within a sequence and subtracts it, removing the internal covariate shift. In experiments testing for contextual video recognition with ConvGRU, results show that (1) ConvGRU clearly outperforms feed-forward neural networks, (2) AD consistently and significantly accelerates training and improves generalization, (3) performance is further improved when AD is coupled with other normalization methods, and most importantly, (4) the more long-term contextual information is required, the more AD outperforms existing methods. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of finite-time synchronization of stochastic coupled neural networks (SCNNs) subject to Markovian switching, mixed time delay, and actuator saturation. In addition, coupling strengths of the SCNNs are characterized by mutually independent random variables. By utilizing a simple linear transformation, the problem of stochastic finite-time synchronization of SCNNs is converted into a mean-square finite-time stabilization problem of an error system. By choosing a suitable mode dependent switched Lyapunov–Krasovskii functional, a new set of sufficient conditions is derived to guarantee the finite-time stability of the error system. Subsequently, with the help of anti-windup control scheme, the actuator saturation risks could be mitigated. Moreover, the derived conditions help to optimize estimation of the domain of attraction by enlarging the contractively invariant set. Furthermore, simulations are conducted to exhibit the efficiency of proposed control scheme. [ABSTRACT FROM AUTHOR]

Although representation learning methods developed within the framework of traditional neural networks are relatively mature, developing a spiking representation model remains a challenging problem. This paper proposes an event-based method to train a feedforward spiking neural network (SNN) layer for extracting visual features. The method introduces a novel spike-timing-dependent plasticity (STDP) learning rule and a threshold adjustment rule both derived from a vector quantization-like objective function subject to a sparsity constraint. The STDP rule is obtained by the gradient of a vector quantization criterion that is converted to spike-based, spatio-temporally local update rules in a spiking network of leaky, integrate-and-fire (LIF) neurons. Independence and sparsity of the model are achieved by the threshold adjustment rule and by a softmax function implementing inhibition in the representation layer consisting of WTA-thresholded spiking neurons. Together, these mechanisms implement a form of spike-based, competitive learning. Two sets of experiments are performed on the MNIST and natural image datasets. The results demonstrate a sparse spiking visual representation model with low reconstruction loss comparable with state-of-the-art visual coding approaches, yet our rule is local in both time and space, thus biologically plausible and hardware friendly. [ABSTRACT FROM AUTHOR]

The ability to train a network without memorizing the input/output data, thereby allowing a good predictive performance when applied to unseen data, is paramount in ANN applications. In this paper, we propose a frequency-domain approach to evaluate the network initialization in terms of quality of training, i.e., generalization capabilities. As an alternative to the conventional time-domain methods, the proposed approach eliminates the approximate nature of network validation using an excess of unseen data. The benefits of the proposed approach are demonstrated using two numerical examples, where two trained networks performed similarly on the training and the validation data sets, yet they revealed a significant difference in prediction accuracy when tested using a different data set. This observation is of utmost importance in modeling applications requiring a high degree of accuracy. The efficiency of the proposed approach is further demonstrated on a real-world problem, where unlike other initialization methods, a more conclusive assessment of generalization is achieved. On the practical front, subtle methodological and implementational facets are addressed to ensure reproducibility and pinpoint the limitations of the proposed approach. [ABSTRACT FROM AUTHOR]

Memristor describes the relationship between charge and flux. Although several window functions for memristors based on the HP linear and nonlinear dopant drift models have been studied, most of them are inadequate to capture the full characteristics of memristors. To address this issue, this paper proposes a unified window function to describe a general memristor with restrictions of its parameters given. Compared with other window functions, the proposed function demonstrates high validity and accuracy. In order to make the simulation results have high consistency with the results of actual circuit, we apply the new window function to the simulation of a memristor-based multilayer neural network (MNN) circuit. The overall accuracy will vary with the change of control parameters in the window function. It implies that the proposed model can guide the design of actual memristor-based circuits. [ABSTRACT FROM AUTHOR]

Fractional order system is playing an increasingly important role in terms of both theory and applications. In this paper we investigate the global existence of Filippov solutions and the robust generalized Mittag-Leffler synchronization of fractional order neural networks with discontinuous activation and impulses. By means of growth conditions, differential inclusions and generalized Gronwall inequality, a sufficient condition for the existence of Filippov solution is obtained. Then, sufficient criteria are given for the robust generalized Mittag-Leffler synchronization between discontinuous activation function of impulsive fractional order neural network systems with (or without) parameter uncertainties, via a delayed feedback controller and M-Matrix theory. Finally, four numerical simulations demonstrate the effectiveness of our main results. [ABSTRACT FROM AUTHOR]

Unlike the first and the second generation artificial neural networks, spiking neural networks (SNNs) model the human brain by incorporating not only synaptic state but also a temporal component into their operating model. However, their intrinsic properties require expensive computation during training. This paper presents a novel algorithm to SpikeProp for SNN by introducing smoothing L 1 ∕ 2 regularization term into the error function. This algorithm makes the network structure sparse, with some smaller weights that can be eventually removed. Meanwhile, the convergence of this algorithm is proved under some reasonable conditions. The proposed algorithms have been tested for the convergence speed, the convergence rate and the generalization on the classical XOR-problem, Iris problem and Wisconsin Breast Cancer classification. [ABSTRACT FROM AUTHOR]

This paper is devoted to studying the fixed-time synchronization of memristor-based BAM neural networks (MBAMNNs) with discrete delay. Fixed-time synchronization means that synchronization can be achieved in a fixed time for any initial values of the considered systems. In the light of the double-layer structure of MBAMNNs, we design two similar feedback controllers. Based on Lyapunov stability theories, several criteria are established to guarantee that the drive and response MBAMNNs can realize synchronization in a fixed time. In particular, by changing the parameters of controllers, this fixed time can be adjusted to some desired value in advance, irrespective of the initial values of MBAMNNs. Numerical simulations are included to validate the derived results. [ABSTRACT FROM AUTHOR]

In this paper, a novel imbalance learning method for binary classes is proposed, named as Post-Boosting of classification boundary for Imbalanced data (PBI), which can significantly improve the performance of any trained neural networks (NN) classification boundary. The procedure of PBI simply consists of two steps: an (imbalanced) NN learning method is first applied to produce a classification boundary, which is then adjusted by PBI under the geometric mean (G-mean). For data imbalance, the geometric mean of the accuracies of both minority and majority classes is considered, that is statistically more suitable than the common metric accuracy . PBI also has the following advantages over traditional imbalance methods: (i) PBI can significantly improve the classification accuracy on minority class while improving or keeping that on majority class as well; (ii) PBI is suitable for large data even with high imbalance ratio (up to 0.001). For evaluation of (i), a new metric called Majority loss/Minority advance ratio (MMR) is proposed that evaluates the loss ratio of majority class to minority class. Experiments have been conducted for PBI and several imbalance learning methods over benchmark datasets of different sizes, different imbalance ratios, and different dimensionalities. By analyzing the experimental results, PBI is shown to outperform other imbalance learning methods on almost all datasets. [ABSTRACT FROM AUTHOR]

To prevent large interconnected power system from a cascading failure, brownout or even blackout, grid operators require access to faster than real-time information to make appropriate just-in-time control decisions. However, the communication and computational system limitations of currently used supervisory control and data acquisition (SCADA) system can only deliver delayed information. However, the deployment of synchrophasor measurement devices makes it possible to capture and visualize, in near-real-time, grid operational data with extra granularity. In this paper, a cellular computational network (CCN) approach for frequency situational intelligence (FSI) in a power system is presented. The distributed and scalable computing unit of the CCN framework makes it particularly flexible for customization for a particular set of prediction requirements. Two soft-computing algorithms have been implemented in the CCN framework: a cellular generalized neuron network (CCGNN) and a cellular multi-layer perceptron network (CCMLPN), for purposes of providing multi-timescale frequency predictions, ranging from 16.67 ms to 2 s. These two developed CCGNN and CCMLPN systems were then implemented on two different scales of power systems, one of which installed a large photovoltaic plant. A real-time power system simulator at weather station within the Real-Time Power and Intelligent Systems (RTPIS) laboratory at Clemson, SC, was then used to derive typical FSI results. [ABSTRACT FROM AUTHOR]

This paper deals with the finite-time synchronization problem for a class of uncertain coupled switched neural networks under asynchronous switching. By constructing appropriate Lyapunov-like functionals and using the average dwell time technique, some sufficient criteria are derived to guarantee the finite-time synchronization of considered uncertain coupled switched neural networks. Meanwhile, the asynchronous switching feedback controller is designed to finite-time synchronize the concerned networks. Finally, two numerical examples are introduced to show the validity of the main results. [ABSTRACT FROM AUTHOR]

Abstract: This paper proposes a technique, called Evolving Probabilistic Neural Network (), that presents many interesting features, including incremental learning, evolving architecture, the capacity to learn continually throughout its existence and requiring that each training sample be used only once in the training phase without reprocessing. A series of experiments was performed on data sets in the public domain; the results indicate that is superior or equal to the other incremental neural networks evaluated in this paper. These results also demonstrate the advantage of the small architecture and show that its architecture is more stable than the other incremental neural networks evaluated. thus appears to be a promising alternative for a quick learning system and a fast classifier with a low computational cost. [Copyright &y& Elsevier]

Abstract: In this paper, we first investigate the existence of a periodic solution to interval general bidirectional associative memory (BAM) neural networks with multiple delays on time scales by the continuation theorem of coincidence degree theory. Then, by constructing a Lyapunov functional, we discuss the global exponential stability of the periodic solution for such neural networks on time scales. The paper unifies periodic discrete-time and continuous-time BAM neural networks under the same framework. [Copyright &y& Elsevier]

Abstract: The neocognitron, which was proposed by , is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It acquires the ability to recognize patterns through learning. This paper proposes a new rule for competitive learning, named winner-kill-loser, and apply it to the neocognitron. The winner-kill-loser rule resembles the winner-take-all rule. Every time when a training stimulus is presented, non-silent cells compete with each other. The winner, however, not only takes all, but also kills losers. In other words, the winner learns the training stimulus, and losers are removed from the network. If all cells are silent, a new cell is generated and it learns the training stimulus. Thus feature-extracting cells gradually come to distribute uniformly in the feature space. The use of winner-kill-loser rule is not limited to the neocognitron. It is useful for various types of competitive learning, in general. This paper also proposes several improvements made on the neocognitron: such as, disinhibition to the inhibitory surround in the connections to C-cells (or complex cells) from S-cells (or simple cells); and square root shaped saturation in the input-to-output characteristics of C-cells. As a result of these improvements, the recognition rate of the neocognitron has been largely increased. [Copyright &y& Elsevier]

Abstract: In this paper we present a sound-source model for localising and tracking an acoustic source of interest along the azimuth plane in acoustically cluttered environments, for a mobile service robot. The model we present is a hybrid architecture using cross-correlation and recurrent neural networks to develop a robotic model accurate and robust enough to perform within an acoustically cluttered environment. This model has been developed with considerations of both processing power and physical robot size, allowing for this model to be deployed on to a wide variety of robotic systems where power consumption and size is a limitation. The development of the system we present has its inspiration taken from the central auditory system (CAS) of the mammalian brain. In this paper we describe experimental results of the proposed model including the experimental methodology for testing sound-source localisation systems. The results of the system are shown in both restricted test environments and in real-world conditions. This paper shows how a hybrid architecture using band pass filtering, cross-correlation and recurrent neural networks can be used to develop a robust, accurate and fast sound-source localisation model for a mobile robot. [Copyright &y& Elsevier]

Abstract: This paper presents new experimental results on the variadic neural network (VNN) [McGregor, S. (2007). Neural network processing for multiset data. In Proceedings: Vol. 4668. Artificial neural networks — ICANN 2007, 17th international conference (pp. 460–470). Springer]. The inputs to a variadic network are an arbitrary-length list of -tuples of real numbers, where is fixed, and the function computed by the network is unaffected by permutation of the inputs. This paper describes improvements in the training algorithm for the variadic perceptron, based on a constructive cascade topology, and performance of the improved networks on geometric problems inspired by vector graphics. Further development may allow practical application of these or similar networks to vector graphics processing and statistical analysis. [Copyright &y& Elsevier]

Abstract: In an earlier paper [Fukushima, K., & Tohyama, K. (2005). Analysis of optic flow: a neural network model. In ICONIP 2005, International Conference on Neural Information Processing (pp. 312–317); Tohyama, K., & Fukushima, K. (2005). Neural network model for extracting 98 optic flow. Neural Networks, 18(5–6), 549–556], we proposed a neural network extracting optic flow based on vector field hypothesis. We started discussion there, however, from the stage where local velocities of visual objects have already been extracted. This paper proposes a new mechanism of extracting local velocity from retinal images, and adds it to the previous network. The network has a hierarchical multilayered architecture. X- and Y-cells of the retina extract spatial and temporal contrast of brightness. The network contains several types of V1 cells, namely, S- C- and V-cells. S- and C-cells extract orientated edges. V-cells extract local velocities, based on the signals from Y- and S-cells. MT cells extract relative velocities between adjoining small visual fields. MST cells add the responses of MT cells to extract a specific optic flow, such as rotation and expansion/contraction. The difference in type of optic flow extracted by MST cells can be created simply by the difference in relative locations of inhibitory to excitatory areas in the receptive fields of MT cells. [Copyright &y& Elsevier]

Abstract: The paper deals with investigating approximation abilities of a special class of discrete-time dynamic neural networks. The networks considered are called locally recurrent globally feed-forward, because they are designed with dynamic neuron models which contain inner feedbacks, but interconnections between neurons are strict feed-forward ones like in the well-known multi-layer perceptron. The paper presents analytical results showing that a locally recurrent network with two hidden layers is able to approximate a state-space trajectory produced by any Lipschitz continuous function with arbitrary accuracy. Moreover, based on these results, the network can be simplified and transformed into a more practical structure needed in real world applications. [Copyright &y& Elsevier]

Abstract: Object recognition is one of the most important functions of the human visual system, yet one of the least understood, this despite the fact that vision is certainly the most studied function of the brain. We understand relatively well how several processes in the cortical visual areas that support recognition capabilities take place, such as orientation discrimination and color constancy. This paper proposes a model of the development of object recognition capability, based on two main theoretical principles. The first is that recognition does not imply any sort of geometrical reconstruction, it is instead fully driven by the two dimensional view captured by the retina. The second assumption is that all the processing functions involved in recognition are not genetically determined or hardwired in neural circuits, but are the result of interactions between epigenetic influences and basic neural plasticity mechanisms. The model is organized in modules roughly related to the main visual biological areas, and is implemented mainly using the LISSOM architecture, a recent neural self-organizing map model that simulates the effects of intercortical lateral connections. This paper shows how recognition capabilities, similar to those found in brain ventral visual areas, can develop spontaneously by exposure to natural images in an artificial cortical model. [Copyright &y& Elsevier]

The paper presents a methodology for the analysis of functional changes in brain activity across different conditions and different groups of subjects. This analysis is based on the recently proposed NeuCube spiking neural network (SNN) framework and more specifically on the analysis of the connectivity of a NeuCube model trained with electroencephalography (EEG) data. The case study data used to illustrate this method is EEG data collected from three groups—subjects with opiate addiction, patients undertaking methadone maintenance treatment, and non-drug users/healthy control group. The proposed method classifies more accurately the EEG data than traditional statistical and artificial intelligence (AI) methods and can be used to predict response to treatment and dose-related drug effect. But more importantly, the method can be used to compare functional brain activities of different subjects and the changes of these activities as a result of treatment, which is a step towards a better understanding of both the EEG data and the brain processes that generated it. The method can also be used for a wide range of applications, such as a better understanding of disease progression or aging. [ABSTRACT FROM AUTHOR]

Abstract: This paper proposes the combination of two state-of-the-art algorithms for processing graph input data, viz., the probabilistic mapping graph self organizing map, an unsupervised learning approach, and the graph neural network, a supervised learning approach. We organize these two algorithms in a cascade architecture containing a probabilistic mapping graph self organizing map, and a graph neural network. We show that this combined approach helps us to limit the long-term dependency problem that exists when training the graph neural network resulting in an overall improvement in performance. This is demonstrated in an application to a benchmark problem requiring the detection of spam in a relatively large set of web sites. It is found that the proposed method produces results which reach the state of the art when compared with some of the best results obtained by others using quite different approaches. A particular strength of our method is its applicability towards any input domain which can be represented as a graph. [Copyright &y& Elsevier]

Abstract: Selection of the optimal neural architecture to solve a pattern classification problem entails to choose the relevant input units, the number of hidden neurons and its corresponding interconnection weights. This problem has been widely studied in many research works but their solutions usually involve excessive computational cost in most of the problems and they do not provide a unique solution. This paper proposes a new technique to efficiently design the MultiLayer Perceptron (MLP) architecture for classification using the Extreme Learning Machine (ELM) algorithm. The proposed method provides a high generalization capability and a unique solution for the architecture design. Moreover, the selected final network only retains those input connections that are relevant for the classification task. Experimental results show these advantages. [Copyright &y& Elsevier]

Abstract: Robust synchronized activity is ubiquitous in real neural networks. Diverse synaptic properties contribute to such robust synchronization. Synaptic current induced by an input spike is dependent on the voltage of post-synaptic neuron in real neural systems. Current-based synapse models, which neglect the voltage-dependent properties of synaptic currents, are widely used for analysis of robust synchronization in neural networks due to their simplicity. However, it was found in this paper that the voltage-dependent properties of synaptic currents play a key role in robust network synchronization in most cases, implying that current-based synapses are oversimplified for analysis of robust network synchronization. [Copyright &y& Elsevier]

Abstract: In this paper, we propose a novel framework based on a collective network of evolutionary binary classifiers (CNBC) to address the problems of feature and class scalability. The main goal of the proposed framework is to achieve a high classification performance over dynamic audio and video repositories. The proposed framework adopts a “Divide and Conquer” approach in which an individual network of binary classifiers (NBC) is allocated to discriminate each audio class. An evolutionary search is applied to find the best binary classifier in each NBC with respect to a given criterion. Through the incremental evolution sessions, the CNBC framework can dynamically adapt to each new incoming class or feature set without resorting to a full-scale re-training or re-configuration. Therefore, the CNBC framework is particularly designed for dynamically varying databases where no conventional static classifiers can adapt to such changes. In short, it is entirely a novel topology, an unprecedented approach for dynamic, content/data adaptive and scalable audio classification. A large set of audio features can be effectively used in the framework, where the CNBCs make appropriate selections and combinations so as to achieve the highest discrimination among individual audio classes. Experiments demonstrate a high classification accuracy (above 90%) and efficiency of the proposed framework over large and dynamic audio databases. [Copyright &y& Elsevier]

Abstract: This paper reports two-dimensional parameter-space plots for both, the hyperbolic tangent and the piecewise-linear neuron activation functions of a three-dimensional Hopfield neural network. The plots obtained using both neuron activation functions are compared, and we show that similar features are present on them. The occurrence of self-organized periodic structures embedded in chaotic regions is verified for the two cases. [Copyright &y& Elsevier]

Abstract: In the paper, effective and simple features for image recognition (named LiRA-features) are investigated in the task of handwritten digit recognition. Two neural network classifiers are considered—a modified 3-layer perceptron LiRA and a modular assembly neural network. A method of feature selection is proposed that analyses connection weights formed in the preliminary learning process of a neural network classifier. In the experiments using the MNIST database of handwritten digits, the feature selection procedure allows reduction of feature number (from 60 000 to 7000) preserving comparable recognition capability while accelerating computations. Experimental comparison between the LiRA perceptron and the modular assembly neural network is accomplished, which shows that recognition capability of the modular assembly neural network is somewhat better. [Copyright &y& Elsevier]

Abstract: In this paper, we first discuss the existence and uniqueness of the equilibrium point of interval general BAM neural networks with reaction–diffusion terms and multiple time-varying delays by means of using degree theory. Then by applying the existence result of an equilibrium point and constructing a Lyapunov functional, we discuss global exponential stability for above neural networks. In the last section, we also give an example to demonstrate the validity of our global exponential stability result for above neural network. [Copyright &y& Elsevier]

Abstract: Echo State Networks (ESNs) constitute an emerging approach for efficiently modeling Recurrent Neural Networks (RNNs). In this paper we investigate some of the main aspects that can be accounted for the success and limitations of this class of models. In particular, we propose complementary classes of factors related to contractivity and architecture of reservoirs and we study their relative relevance. First, we show the existence of a class of tasks for which ESN performance is independent of the architectural design. The effect of the Markovian factor, characterizing a significant class within these cases, is shown by introducing instances of easy/hard tasks for ESNs featured by contractivity of reservoir dynamics. In the complementary cases, for which architectural design is effective, we investigate and decompose the aspects of network design that allow a larger reservoir to progressively improve the predictive performance. In particular, we introduce four key architectural factors: input variability, multiple time-scales dynamics, non-linear interactions among units and regression in an augmented feature space. To investigate the quantitative effects of the different architectural factors within this class of tasks successfully approached by ESNs, variants of the basic ESN model are proposed and tested on instances of datasets of different nature and difficulty. Experimental evidences confirm the role of the Markovian factor and show that all the identified key architectural factors have a major role in determining ESN performances. [Copyright &y& Elsevier]

Abstract: This paper is concerned with multi-sensor optimal fusion filtering for a class of nonlinear intelligent systems with time delays. A unified model consisting of a linear dynamic system and a bounded static nonlinear operator is employed to describe these systems, such as neural networks and Takagi and Sugeno (T–S) fuzzy models. Based on the performance analysis of this unified model using the linear matrix inequality (LMI) approach, centralized and distributed fusion filters are designed for multi-sensor time-delayed systems to guarantee the asymptotic stability of the fusion error systems and to reduce the influence of noise on the filtering error. The parameters of these filters are obtained by solving the eigenvalue problem (EVP). As most artificial neural networks or fuzzy systems with or without time delays can be described with this unified model, fusion filter design for these systems can be done in a unified way. Simulation examples are provided to illustrate the design procedure and effectiveness of the proposed approach. [Copyright &y& Elsevier]

Abstract: A Recurrent Neural Network (RNN) is a powerful connectionist model that can be applied to many challenging sequential problems, including problems that naturally arise in language and speech. However, RNNs are extremely hard to train on problems that have long-term dependencies, where it is necessary to remember events for many timesteps before using them to make a prediction. In this paper we consider the problem of training RNNs to predict sequences that exhibit significant long-term dependencies, focusing on a serial recall task where the RNN needs to remember a sequence of characters for a large number of steps before reconstructing it. We introduce the Temporal-Kernel Recurrent Neural Network (TKRNN), which is a variant of the RNN that can cope with long-term dependencies much more easily than a standard RNN, and show that the TKRNN develops short-term memory that successfully solves the serial recall task by representing the input string with a stable state of its hidden units. [Copyright &y& Elsevier]

Abstract: In this paper, the local coupled feedforward neural network is presented. Its connection structure is same as that of Multilayer Perceptron with one hidden layer. In the local coupled feedforward neural network, each hidden node is assigned an address in an input space, and each input activates only the hidden nodes near it. For each input, only the activated hidden nodes take part in forward and backward propagation processes. Theoretical analysis and simulation results show that this neural network owns the “universal approximation” property and can solve the learning problem of feedforward neural networks. In addition, its characteristic of local coupling makes knowledge accumulation possible. [Copyright &y& Elsevier]

Abstract: In this paper, improved criteria for the passivity of neural networks with time delay are proposed. The improvement is achieved by exploiting the finer relationships between the time-varying delay and its size bound which allow the use of more slack variables. It is illustrated via numerical examples that the proposed results are less conservative than those available in the literature. [Copyright &y& Elsevier]