Search

Negative correlation learning (NCL) is a neural network ensemble learning algorithm that introduces a correlation penalty term to the cost function of each individual network so that each neural network minimizes its mean square error/MSEI together with the correlation of the ensemble. This paper analyzes NCL and reveals that the training of NCL (when λ = 1) corresponds to training the entire ensemble as a single learning machine that only minimizes the MSE without regularization. This analysis explains the reason why NCL is prone to overtitting the noise in the training set. This paper also demonstrates that tuning the correlation parameter λ in NCL by cross validation cannot overcome the overfitting problem. The paper analyzes this problem and proposes the regularized negative correlation learning (RNCL) algorithm which incorporates an additional regularization term for the whole ensemble. RNCL decomposes the ensemble's training objectives, including MSE and regularization, into a set of sub-objectives, and each sub-objective is implemented by an individual neural network. In this paper, we also provide a Bayesian interpretation for RNCL and provide an automatic algorithm to optimize regularization parameters based on Bayesian inference. The RNCL formulation is applicable to any nonlinear estimator minimizing the MSE. The experiments on synthetic as well as real-world data sets demonstrate that RNCL achieves better performance than NCL, especially when the noise level is nontrivial in the data set. [ABSTRACT FROM AUTHOR]

Original Hebbian encoding scheme of bidirectional associative memory (BAM) provides a poor pattern capacity and recall performance. Based on Rosenblatt's perceptron learning algorithm, the pattern capacity of BAM is enlarged, and perfect recall of all training pattern pairs is guaranteed. However, these methods put their emphases on pattern capacity, rather than error correction capability which is another critical point of BAM. This paper analyzes the sensitivity to noise in RAM and obtains an interesting idea to improve noise immunity of BAM. Some researchers have found that the noise sensitivity of BAM relates to the minimum absolute value of net inputs (MAV). However, in this paper, the analysis on failure association shows that it is related not only to MAV but also to the variance of weights associated with synapse connections. In fact, it is a positive monotone increasing function of the quotient of MAV divided by the variance of weights. This idea provides an useful principle of improving error correction capability of RAM. Some revised encoding schemes, such as small variance learning for RAM (SVBAM), evolutionary pseudorelaxation learning for BAM (EPRLAB) and evolutionary bidirectional learning (EBL), have been introduced to illustrate the performance of this principle. All these methods perform better than their original versions in noise immunity. Moreover, these methods have no negative effect on the pattern capacity of BAM. The convergence of these methods is also discussed in this paper. If there exist solutions, EPRLAB and EBL always converge to a global optimal solution in the senses of both, pattern capacity and noise immunity. However, the convergence of SVBAM may be affected by a preset function. [ABSTRACT FROM AUTHOR]

Under-complete models, which derive lower dimensional representations of input data, are valuable in domains in which the number of input dimensions is very large, such as data consisting of a temporal sequence of images. This paper presents the under-complete product of experts (UPoE), where each expert models a one-dimensional projection of the data. Maximum-likelihood learning rules for this model constitute a tractable and exact algorithm for learning under-complete independent components. The learning rules for this model coincide with approximate learning rules proposed earlier for under-complete independent component analysis (UICA) models. This paper also derives an efficient sequential learning algorithm from this model and discusses its relationship to sequential independent component analysis (ICA), projection pursuit density estimation, and feature induction algorithms for additive random field models. This paper demonstrates the efficacy of these novel algorithms on high-dimensional continuous datasets. [ABSTRACT FROM AUTHOR]

This paper introduces a general framework for describing dynamic neural networks—the layered digital dynamic network (LDDN). This framework allows the development of two general algorithms for computing the gradients and Jacobians for these dynamic networks: backpropagation-through-time (BPTT) and real-time recurrent learning (RTRL). The structure of the LDDN framework enables an efficient implementation of both algorithms for arbitrary dynamic networks. This paper demonstrates that the BPTT algorithm is more efficient for gradient calculations, but the RTRL algorithm is more efficient for Jacobian calculations. [ABSTRACT FROM AUTHOR]

This paper presents a new sequential learning algorithm for radial basis function (RBF) networks referred to as generalized growing and pruning algorithm for RBF (GGAP-RBF). The paper first introduces the concept of significance for the hidden neurons and then uses it in the learning algorithm to realize parsimonious networks. The growing and pruning strategy of GGAP-RBF is based on linking the required learning accuracy with the significance of the nearest or intentionally added new neuron. Significance of a neuron is a measure of the average information content of that neuron. The GGAP-RBF algorithm can be used for any arbitrary sampling density for training samples and is derived from a rigorous statistical point of view. Simulation re- suits for bench mark problems in the function approximation area show that the GGAP-RBF outperforms several other sequential learning algorithms in terms of learning speed, network size and generalization performance regardless of the sampling density function of the training data. [ABSTRACT FROM AUTHOR]

In this paper, a novel exemplar-based constructive approach using kernels is proposed for simultaneous pattern classification and multidomain pattern association tasks. The kernel networks are constructed on a modular basis by a simple one-shot self-structuring algorithm motivated from the traditional Hebbian principle and then, they act as the flexible memory capable of generalization for the respective classes. In the self-structuring kernel memory (SSKM), any arduous and iterative network parameter tuning is not involved for establishing the weight connections during the construction, unlike conventional approaches, and thereby, it is considered that the networks do not inherently suffer from the associated numerical instability. Then, the approach is extended for multidomain pattern association, in which a particular domain input cannot only activate some kernel units (KUs) but also the kernels in other domain(s) via the cross-domain connection(s) in between. Thereby, the SSKM can be regarded as a simultaneous pattern classifier and associator. In the simulation study for pattern classification, it is justified that an SSKM consisting of distinct kernel networks can yield relatively compact-sized pattern classifiers, while preserving a reasonably high generalization capability, in comparison with the approach using support vector machines (SVMs). [ABSTRACT FROM AUTHOR]

This paper proposes noisy speech recognition using hierarchical singleton-type recurrent neural fuzzy networks (HSRNFNs). The proposed HSRNFN is a hierarchical connection of two singleton-type recurrent neural fuzzy networks (SRNFNs), where one is used for noise filtering and the other for recognition. The SRNFN is constructed by recurrent fuzzy if-then rules with fuzzy singletons in the consequences, and their recurrent properties make them suitable for processing speech patterns with temporal characteristics. In n words recognition, n SRNFNs are created for modeling n words, where each SRNFN receives the current frame feature and predicts the next one of its modeling word. The prediction error of each SRNFN is used as recognition criterion. In filtering, one SRNFN is created, and each SRNFN recognizer is connected to the same SRNFN filter, which filters noisy speech patterns in the feature domain before feeding them to the SRNFN recognizer. Experiments with Mandarin word recognition under different types of noise are performed. Other recognizers, including multilayer perceptron (MLP), time-delay neural networks (TDNNs), and hidden Markov models (HMMs), are also tested and compared. These experiments and comparisons demonstrate good results with HSRNFN for noisy speech recognition tasks. [ABSTRACT FROM AUTHOR]

In this paper, we present two fast sparse approximation schemes for least squares support vector machine (LS-SVM), named FSALS-SVM and PFSALS-SVM, to overcome the limitation of LS-SVM that it is not applicable to large data sets and to improve test speed. FSALS-SVM iteratively builds the decision function by adding one basis function from a kernel-based dictionary at one time. The process is terminated by using a flexible and stable epsilon insensitive stopping criterion. A probabilistic speedup scheme is employed to further improve the speed of FSALS-SVM and the resulting classifier is named PFSALS-SVM. Our algorithms are of two compelling features: low complexity and sparse solution. Experiments on benchmark data sets show that our algorithms obtain sparse classifiers at a rather low cost without sacrificing the generalization performance. [ABSTRACT FROM AUTHOR]

Real-time recurrent learning (RTRL), commonly employed for training a fully connected recurrent neural network (RNN), has a drawback of slow convergence rate. In the light of this deficiency, a decision feedback recurrent neural equalizer (DFRNE) using the RTRL requires long training sequences to achieve good performance. In this paper, extended Kalman filter (EKF) algorithms based on the RTRL for the DFRNE are presented in state-space formulation of the system, in particular for complex-valued signal processing. The main features of global EKF and decoupled EKF algorithms are fast convergence and good tracking performance. Through nonlinear channel equalization, performance of the DFRNE with the EKF algorithms is evaluated and compared with that of the DFRNE with the RTRL. [ABSTRACT FROM AUTHOR]

This paper proposes new modified constrained learning neural root finders (NRFs) of polynomial constructed by backpropagation network (BPN). The technique is based on the relationships between the roots and the coefficients of polynomial as well as between the root moments and the coefficients of the polynomial. We investigated different resulting constrained learning algorithms (CLAs) based on the variants of the error cost functions (ECFs) in the constrained BPN and derived a new modified CLA (MCLA), and found that the computational complexities of the CLA and the MCLA based on the root-moment method (RMM) are the order of polynomial, and that the MCLA is simpler than the CLA. Further, we also discussed the effects of the different parameters with the CLA and the MCLA on the NRFs. In particular, considering the coefficients of the polynomials involved in practice to possibly be perturbed by noisy sources, thus, we also evaluated and discussed the effects of noises on the two NRFs. Finally, to demonstrate the advantage of our neural approaches over the nonneural ones, a series of simulating experiments are conducted. [ABSTRACT FROM AUTHOR]

A novel approach toward neural networks modeling is presented in the paper. It is unique in the fact that allows nets' weights to change according to changes of some environmental factors even after completing the learning process. The models of context-dependent (cd) neuron, one- and multilayer feedforward net are presented, with basic learning algorithms and examples of functioning. The Vapnik-Chervonenkis (VC) dimension of a cd neuron is derived, as well as VC dimension of multilayer feedforward nets. Cd nets' properties are discussed and compared with the properties of traditional nets. Possibilities of applications to classification and control problems are also outlined and an example presented. [ABSTRACT FROM AUTHOR]

With the aim of getting understandable symbolic rules to explain a given phenomenon, we split the task of learning these rules from sensory data in two phases: a multilayer perceptron maps features into propositional variables and a set of subsequent layers operated by a PAC-like algorithm learns Boolean expressions on these variables. The special features of this procedure are that: i) the neural network is trained to produce a Boolean output having the principal task of discriminating between classes of inputs; ii) the symbolic part is directed to compute rules within a family that is not known a priori; iii) the welding point between the two learning systems is represented by a feedback based on a suitability evaluation of the computed rules. The procedure we propose is based on a computational learning paradigm set up recently in some papers in the fields of theoretical computer science, artificial intelligence and cognitive systems. The present article focuses on information management aspects of the procedure. We deal with the lack of prior information about the rules through learning strategies that affect both the meaning of the variables and the description length of the rules into which they combine. The paper uses the task of learning to formally discriminate among several emotional states as both a working example and a test bench for a comparison with previous symbolic and subsymbolic methods in the field. [ABSTRACT FROM AUTHOR]

This paper presents an implementation-independent measure of the amount of information processing performed by (part of) an adaptive system which depends on the goal to be performed by the overall system. This new measure gives rise to a theoretical framework under which several classical supervised and unsupervised learning algorithms fall and, additionally, new efficient learning algorithms can be derived. In the context of neural networks, the framework of information theory strives to design neurally inspired structures from which complex functionality should emerge. Yet, classical measures of information have not taken an explicit account of some of the fundamental concepts in brain theory and neural computation, namely that optimal coding depends on the specific task(s) to be solved by the system and that goal orientedness also depends on extracting relevant information from the environment to be able to affect it in the desired way. We present a new information processing measure that takes into account both the extraction of relevant information and the reduction of spurious information for the task to be solved by the system. This measure is implementation-independent and therefore can be used to analyze and design different adaptive systems. Specifically, we show its application for learning perceptrons, decision trees and linear autoencoders. [ABSTRACT FROM AUTHOR]

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perception (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets. [ABSTRACT FROM AUTHOR]

This paper proposes a constructive approach for finding arbitrary (real or complex) roots of arbitrary (real or complex) polynomials by multilayer perceptron network (MLPN) using constrained learning algorithm (CLA), which encodes the a priori information of constraint relations between root moments and coefficients of a polynomial into the usual BP algorithm (BPA). Moreover, the root moment method (RMM) is also simplified into a recursive version so that the computational complexity can be further decreased, which leads the roots of those higher order polynomials to be readily found. In addition, an adaptive learning parameter with the CLA is also proposed in this paper; an initial weight selection method is also given. Finally, several experimental results show that our proposed neural connectionism approaches, with respect to the nonneural ones, are more efficient and feasible in finding the arbitrary roots of arbitrary polynomials. [ABSTRACT FROM AUTHOR]

Decision theory shows that the optimal decision is a function of the posterior class probabilities. More specifically, in binary classification, the optimal decision is based on the comparison of the posterior probabilities with some threshold. Therefore, the most accurate estimates of the posterior probabilities are required near these decision thresholds. This paper discusses the design of objective functions that provide more accurate estimates of the probability values, taking into account the characteristics of each decision problem. We propose learning algorithms based on the stochastic gradient minimization of these loss functions. We show that the performance of the classifier is improved when these algorithms behave like sample selectors: samples near the decision boundary are the most relevant during learning. [ABSTRACT FROM AUTHOR]

Independent component analysis is to extract independent signals from their linear mixtures without assuming prior knowledge of their mixing coefficients. As we know, a number of factors are likely to affect separation results in practical applications, such as the number of active sources, the distribution of source signals, and noise. The purpose of this paper to develop a general framework of blind separation from a practical point of view with special emphasis on the activation function adaptation. First, we propose the exponential generative model for probability density functions. A method of constructing an exponential generative model from the activation functions is discussed. Then, a Learning algorithm is derived to update the parameters in the exponential generative model. The learning algorithm for the activation function adaptation is consistent with the one for training the denting model. Stability analysis of the learning algorithm for the activation function is also discussed. Both theoretical analysis and simulations show that the proposed approach is universally convergent regardless of the distributions of sources. Finally, computer simulations are given to demonstrate the effectiveness and validity of the approach. [ABSTRACT FROM AUTHOR]

The improved computation presented in this paper is aimed to optimize the neural networks learning process using Levenberg-Marquardt (LM) algorithm. Quasi-Hessian matrix and gradient vector are computed directly, without Jacobian matrix multiplication and storage. The memory limitation problem for LM training is solved. Considering the symmetry of quasi-Hessian matrix, only elements in its upper/lower triangular array need to be calculated. Therefore, training speed is improved significantly, not only because of the smaller array stored in memory, but also the reduced operations in quasi-Hessian matrix calculation. The improved memory and time efficiencies are especially true for large sized patterns training. [ABSTRACT FROM AUTHOR]

The hierarchical graph neuron (HGN) implements a single cycle memorization and recall operation through a novel algorithmic design. The HGN is an improvement on the already published original graph neuron (GN) algorithm. In this improved approach, it recognizes incomplete/noisy patterns. It also resolves the crosstalk problem, which is identified in the previous publications, within closely matched patterns. To accomplish this, the HGN links multiple GN networks for filtering noise and crosstalk out of pattern data inputs. Intrinsically, the HGN is a lightweight in-network processing algorithm which does not require expensive floating point computations; hence, it is very suitable for real-time applications and tiny devices such as the wireless sensor networks. This paper describes that the HGN's pattern matching capability and the small response time remain insensitive to the increases in the number of stored patterns. Moreover, the HGN does not require definition of rules or setting of thresholds by the operator to achieve the desired results nor does it require heuristics entailing iterative operations for memorization and recall of patterns. [ABSTRACT FROM AUTHOR]

In this paper, we analyze a model of recurrent kernel associative memory (RKAM) recently proposed by Garcia and Moreno. We show that this model consists in a kernelization of the recurrent correlation associative memory (RCAM) of Chiueh and Goodman. In particular, using an exponential kernel, we obtain a generalization of the well-known exponential correlation associative memory (ECAM), while using a polynomial kernel, we obtain a generalization of higher order Hopfield networks with Hebbian weights. We show that the RKAM can outperform the aforementioned associative memory models, becoming equivalent to them when a dominance condition is fulfilled by the kernel matrix. To ascertain the dominance condition, we propose a statistical measure which can be easily computed from the probability distribution of the interpattern Hamming distance or directly estimated from the memory vectors. The RKAM can be used below saturation to realize associative memories with reduced dynamic range with respect to the ECAM and with reduced number of synaptic coefficients with respect to higher order Hopfield networks. [ABSTRACT FROM AUTHOR]

In this paper, we present a recursive algorithm for extracting classification rules from feedforward neural networks (NNs) that have been trained on data sets having both discrete and continuous attributes. The novelty of this algorithm lies in the conditions of the extracted rules: the rule conditions involving discrete attributes are disjoint from those involving continuous attributes. The algorithm starts by first generating rules with discrete attributes only to explain the classification process of the NN. If the accuracy of a rule with only discrete attributes is not satisfactory, the algorithm refines this rule by recursively generating more rules with discrete attributes not already present in the rule condition, or by generating a hyperplane involving only the continuous attributes. We show that for three real-life credit scoring data sets, the algorithm generates rules that are not only more accurate but also more comprehensible than those generated by other NN rule extraction methods. [ABSTRACT FROM AUTHOR]

This paper proposes unconstrained functional networks as a new classifier to deal with the pattern recognition problems. Both methodology and learning algorithm for this kind of computational intelligence classifier using the iterative least squares optimization criterion are derived. The performance of this new intelligent systems scheme is demonstrated and examined using real-world applications. A comparative study with the most common classification algorithms in both machine learning and statistics communities is carried out. The study was achieved with only sets of second-order linearly independent polynomial functions to approximate the neuron functions. The results show that this new framework classifier is reliable, flexible, stable, and achieves a high-quality performance. [ABSTRACT FROM AUTHOR]

This paper presents an analysis of a random access memory (RAM)-based associative memory which uses a weighted voting scheme for information retrieval. This weighted voting memory can operate in heteroassociative or autoassociative mode, can store both real-valued and binary-valued patterns and, unlike memory models, is equipped with a rejection mechanism. A theoretical analysis of the performance of the weighted voting memory is given for the case of binary and random memory sets. Performance measures are derived as a function of the model parameters: pattern size, window size, and number of patterns in the memory set. It is shown that the weighted voting model has large capacity and error correction. The results show that the weighted voting model can successfully achieve high-detection and -identification rates and, simultaneously, low-false-acceptance rates. [ABSTRACT FROM AUTHOR]

In this paper, we present an incremental method for model selection and learning of Gaussian mixtures based on the recently proposed variational Bayes approach. The method adds components to the mixture using a Bayesian splitting test procedure: a component is split into two components and then variational update equations are applied only to the parameters of the two components. As a result, either both components are retained in the model or one of them is found to be redundant and is eliminated from the model. In our approach, the model selection problem is treated locally, in a region of the data space, so we can set more informative priors based on the local data distribution. A modified Bayesian mixture model is presented to implement this approach, along with a learning algorithm that iteratively applies a splitting test on each mixture component. Experimental results and comparisons with two other techniques testify for the adequacy of the proposed approach. [ABSTRACT FROM AUTHOR]

A stable neural network (NN)-based observer for general multivariable nonlinear systems is presented in this paper. Unlike most previous neural network observers, the proposed observer uses a nonlinear-in-parameters neural network (NLPNN). Therefore, it can be applied to systems with higher degrees of non- linearity without any a priori knowledge about system dynamics. The learning rule for the neural network is a novel approach based on the modified backpropagatfon (BP) algorithm. An e-modification term is added to guarantee robustness of the observer. No strictly positive real (SPR) or any other strong assumption is imposed on the proposed approach. The stability of the recurrent neural network observer is shown by Lyapunov's direct method. Simulation results for a flexible-joint manipulator are presented to demonstrate the enhanced performance achieved by utilizing the proposed neural network observer. [ABSTRACT FROM AUTHOR]

In this paper, an efficient method for high-speed face recognition based on the discrete cosine transform (DCT), the Fisher's linear discriminant (FLD) and radial basis function (RBF) neural networks is presented. First, the dimensionality of the original face image is reduced by using the DCT and the large area illumination variations are alleviated by discarding the first few low-frequency DCT coefficients. Next, the truncated DCT coefficient vectors are clustered using the proposed clustering algorithm. This process makes the subsequent FLD snore efficient. After implementing the FLD, the most discriminating and invariant facial features are maintained and the training samples are clustered well. As a consequence, further parameter estimation for the RBF neural networks is fulfilled easily which facilitates fast training in the RBF neural networks. Simulation results show that the proposed system achieves excellent performance with high training and recognition speed, high recognition rate as well as very good illumination robustness. [ABSTRACT FROM AUTHOR]

This paper proposes a novel cross-correlation neural network (CNN) model for finding the principal singular subspace of a cross-correlation matrix between two high-dimensional data streams. We introduce a novel nonquadratic criterion (NQC) for searching the optimum weights of two linear neural networks (LNN). The NQC exhibits a single global minimum attained if and only if the weight matrices of the left and right neural networks span the left and right principal singular subspace of a cross-correlation matrix, respectively. The other stationary points of the NQC are (unstable) saddle points. We develop an adaptive algorithm based on the NQC for tracking the principal singular subspace of a cross-correlation matrix between two high-dimensional vector sequences. The NQC algorithm provides a fast online learning of the optimum weights for two LNN. The global asymptotic stability of the NQC algorithm is analyzed. The NQC algorithm has several key advantages such as faster convergence, which is illustrated through simulations. [ABSTRACT FROM AUTHOR]

Fuzzy neural systems have been a subject of great interest in the last few years, due to their abilities to facilitate the exchange of information between symbolic and subsymbolic domains. However, the models in the literature are not able to deal with structured organization of information, that is typically required by symbolic processing. In many application domains, the patterns are not only structured, but a fuzziness degree is attached to each subsymbolic pattern primitive. The purpose of this paper is to show how recursive neural networks, properly conceived for dealing with structured information, can represent nondeterministic fuzzy frontier-to-root tree automata. Whereas available prior knowledge expressed in terms of fuzzy state transition rules are injected into a recursive network, unknown rules are supposed to be filled in by data-driven learning. We also prove the stability of the encoding algorithm, extending previous results on the injection of fuzzy finite-state dynamics in high-order recurrent networks. [ABSTRACT FROM AUTHOR]

In this paper, we present a gradient ascent learning method of the Hopfield neural network for bipartite subgraph problem. The method is intended to provide a near-optimum parallel algorithm for solving the bipartite subgraph problem. To do this we use the Hopfield neural network to get a near-maximum bipartite subgraph, and increase the energy by modifying weights in a gradient ascent direction of the energy to help the network escape from the state of the near-maximum bipartite subgraph to the state of the maximum bipartite subgraph or better one. A large number of instances are simulated to verify the proposed method with the simulation results showing that the solution quality is superior to that of best existing parallel algorithm. We also test the learning method on total coloring problem. The simulation results show that our method finds op- timal solution in every test graph. [ABSTRACT FROM AUTHOR]

This paper presents two novel approaches to deter- mine optimum growing multi-experts network (GMN) structure. The first method called direct method deals with expertise domain and levels in connection with local experts. The growing neural gas (GNG) algorithm is used to cluster the local experts. The concept of error distribution is used to apportion error among the local experts. After reaching the specified size of the network, redundant experts removal algorithm is invoked to prune the size of the network based on the ranking of the experts. However, GMN is not ergonomic due to too many network control parameters. Therefore, a self-regulating GMN (SGMN) algorithm is proposed. SGMN adopts self-adaptive learning rates for gradient-descent learning rules. In addition, SGMN adopts a more rigorous clustering method called fully self-organized simplified adaptive resonance theory in a modified form. Experimental results show SGMN obtains comparative or even better performance than GMN in four benchmark examples, with reduced sensitivity to learning parameters setting. Moreover, both GMN and SGMN outperform the other neural networks and statistical models. The efficacy of SGMN is further justified in three industrial applications and a control problem. It provides consistent results besides holding out a profound potential and promise for building a novel type of nonlinear model consisting of several local linear models. [ABSTRACT FROM AUTHOR]

This paper presents feature selection algorithms for multilayer perceptrons (MLPs) and multiclass support vector machines (SVMs), using mutual information between class labels and classifier outputs, as an objective function. This objective function involves inexpensive computation of information measures only on discrete variables; provides immunity to prior class probabilities; and brackets the probability of error of the classifier. The maximum output information (MOl) algorithms employ this function for feature subset selection by greedy elimination and directed search. The output of the MOI algorithms is a feature subset of user-defined size and an associated trained classifier (MLP/SVM). These algorithms compare favorably with a number of other methods in terms of performance on various artificial and real-world data sets. [ABSTRACT FROM AUTHOR]

A key step of using gradient descend methods to develop learning algorithms of a regular feedforward fuzzy neural network (FNN) is to differentiate max -- min functions, which con- thin max (v) and mm (∇) operations. The paper aims at several objectives. First, investigate further the differentiation of V - ∇ functions. Second, employ general fuzzy numbers, which include triangular and trapezoidal fuzzy numbers as special cases to define a three-layer regular FNN. The general fuzzy numbers related can he approximately determined by their corresponding finite level sets. So, we can approximately represent the input-output (I/O) relationship of the regular FNN as functions of the endpoints of all finite level sets. Third, a fuzzy back-propagation algorithm is presented. And to speed up the convergence of the learning algorithm, a fuzzy conjugate gradient algorithm for fuzzy weights and biases is developed, furthermore, the convergence of the algorithm is analyzed, systematically. Finally, some real simulations demonstrate the efficiency of our learning algorithms. The regular FNN is applied to the approximate realization of fuzzy inference rules and fuzzy functions defined on given compact sets. [ABSTRACT FROM AUTHOR]

Projective adaptive resonance theory (PART) neural network developed by Cao and Wu recently has been shown to be very effective in clustering data sets in high dimensional spaces. The PART algorithm is based on the assumptions that the model equations of PART (a large scale and singularly perturbed system of differential equations coupled with a reset mechanism) have quite regular computational performance. This paper provides a rigorous proof of these regular dynamics of the PART model when the signal functions are special step functions, and provides additional simulation results to illustrate the computational performance of PART. [ABSTRACT FROM AUTHOR]

This paper proposes an approach of cooperative control that is based on the concept of combining neural networks and the methodology of sliding mode control (SMC). The main purpose is to eliminate the chattering phenomenon. Next, the system performance can be improved by using the method of SMC. In the present approach, two parallel Neural Networks are utilized to realize a neuro-sliding mode control (NSMC), where the equivalent control and the corrective control are the outputs of neural network 1 and neural network 2, respectively. Based on expressions of the SMC, the weight adaptations of neural network can be determined. Furthermore, the gradient descent method is used to minimize the control force so that the chattering phenomenon can be eliminated. Finally, experimental results are given to show the effectiveness and feasibility of the approach. [ABSTRACT FROM AUTHOR]

In this paper, we elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training [1], [2]. By concentrating on activation clusters in RNNs, while not throwing away the continuous state space network dynamics, we extract predictive models that we call neural prediction machines (NPMs). When RNNs with sigmoid activation functions are initialized with small weights (a common technique in the RNN community), the clusters of recurrent activations emerging prior to training are indeed meaningful and correspond to Markov prediction contexts. In this case, the extracted NPMs correspond to a class of Markov models, called variable memory length Markov models (VLMMs). In order to appreciate how much information has really been induced during the training, the RNN performance should always be compared with that of VLMMs and NPMs extracted before training as the "null" base models. Our arguments are supported by experiments on a chaotic symbolic sequence and a context-free language with a deep recursive structure. [ABSTRACT FROM AUTHOR]

Presents a study that discussed a simple ∞neuron model as a biologically plausible principle component analyzer. Analysis of the proposed algorithm; Relationship to some known learning algorithms; Conclusion.

Presents a systematic approach for construction reformulated radial basis function (RBF) neural networks, which was developed to facilitate their training by supervised learning algorithms based on gradient descent. Admissibility conditions for RBF; Estimation of the free parameters of RBF; Experimental results.

Proposes an unsupervised video object segmentation and tracking algorithm based on an adaptable neural-network architecture. Neural-network retraining strategy; Unsupervised retraining set estimation in videoconference sequences; Specifications of the decision mechanism; Experimental results; Conclusion.

In the above-mentioned paper, Wei and Fahn proposed a neural architecture, the multisynapse neural network, to solve constrained optimization problems including high-order, logarithmic, and sinusoidal forms, etc. As one of its main applications, a fuzzy bidirectional associative clustering network (FBACN) was proposed for fuzzy-partition clustering according to the objective-functional method. The connection between the objective-functional-based fuzzy c-partition algorithms and FBACN is the Lagrange multiplier approach. Unfortunately, the Lagrange multiplier approach was incorrectly applied so that FBACN does not equivalently minimize its corresponding constrained objective-function. Additionally, Wei and Fahn adopted traditional definition of fuzzy c-partition, which is not satisfied by FBACN. Therefore, FBACN can not solve constrained optimization problems, either. [ABSTRACT FROM AUTHOR]

Distributed synchronization is known to occur at several scales in the brain, and has been suggested as playing a key functional role in perceptual grouping. State-of-the-art visual grouping algorithms, however, seem to give comparatively little attention to neural synchronization analogies. Based on the framework of concurrent synchronization of dynamical systems. simple networks of neural oscillators coupled with diffusive connections are proposed to solve visual grouping problems. The key idea is to embed the desired grouping properties in the choice of the diffusive couplings, so that synchronization of oscillators within each group indicates perceptual grouping of the underlying stimulative atoms, while desynchronization between groups corresponds to group segregation. Compared with state-of-the-art approaches, the same algorithm is shown to achieve promising results on several classical visual grouping problems, including point clustering, contour integration, and image segmentation. [ABSTRACT FROM AUTHOR]

We present a hybrid radial basis function (RBF) sigmoid neural network with a three-step training algorithm that utilizes both global search and gradient descent training. The algorithm used is intended to identify global features of an input-output relationship before adding local detail to the approximating function. It aims to achieve efficient function approximation through the separate identification of aspects of a relationship that are expressed universally from those that vary only within particular regions of the input space. We test the effectiveness of our method using five regression tasks; four use synthetic datasets while the last problem uses real-world data on the wave overtopping of seawalls. It is shown that the hybrid architecture is often superior to architectures containing neurons of a single type in several ways: lower mean square errors are often achievable using fewer hidden neurons and with less need for regularization. Our global-local artificial neural network (GL-ANN) is also seen to compare favorably with both perceptron radial basis net and regression tree derived RBFs. A number of issues concerning the training of GL-ANNs are discussed: the use of regularization, the inclusion of a gradient descent optimization step, the choice of RBF spreads, model selection, and the development of appropriate stopping criteria. [ABSTRACT FROM AUTHOR]

Evolving gradient-learning artificial neural networks (ANNs) using an evolutionary algorithm (EA) is a popular approach to address the local optima and design problems of ANN. The typical approach is to combine the strength of back-propagation (BP) in weight learning and EA's capability of searching the architecture space. However, the UP's "gradient descent" approach requires a highly computer-intensive operation that relatively restricts the search coverage of EA by compelling it to use a small population size. To address this problem, we utilized mutation-based genetic neural network (MGNN) to replace HP by using the mutation strategy of local adaptation of evolutionary programming (EP) to effect weight learning. The MGNN's muta. lion enables the network to dynamically evolve its structure and adapt its weights at the same time. Moreover, MGNN's EP-based encoding scheme allows for a flexible and less restricted formulation of the fitness function and makes fitness computation fast and efficient. This makes it feasible to use larger population sizes and allows MGNN to have a relatively wide search coverage of the architecture space. MGNN Implements a stopping criterion where over fitness occurrences are monitored through "sliding-windows" to avoid premature learning and overlearning. Statistical analysis of its performance to some well-known classification problems demonstrate its good generalization capability. It also reveals that locally adapting or scheduling the strategy parameters embedded in each individual network may provide a proper balance between the local and global searching capabilities of MGNN. [ABSTRACT FROM AUTHOR]

This article presents some efficient training algorithms, based on first-order, second-order, and conjugate gradient optimization methods, for a class of convolutional neural networks (CoNNs), known as shunting inhibitory convolution neural networks. Furthermore, a new hybrid method is proposed, which is derived from the principles of Quickprop, Rprop, SuperSAB, and least squares (LS). Experimental results show that the, new hybrid method can perform as well as the Levenberg-Marquardt (LM) algorithm, but at a much lower computational cost and less memory storage. For comparison sake, the visual pattern recognition task of face/nonface discrimination is chosen as a classification problem to evaluate the performance of the training algorithms. Sixteen training algorithms are implemented for the three different variants of the proposed CoNN architecture: binary-, Toeplitz- and fully connected architectures. All Implemented algorithms can train the three network architectures successfully, but their convergence speed vary markedly in particular, the combination of LS with the new hybrid method and LS with the LM method achieve the best convergence rates in terms of number of training epochs. In addition, the classification accuracies of all three architectures are assessed using ten-fold cross validation. The results show that the binary- and Toeplitz-connected architectures outperform slightly the fully connected architecture: the lowest error rates across all training algorithms are 1.95% for Toeplitz-connected, 2.10% for the binary-connected, and 2.20% for the fully connected network. in general, the modified Broyden-Fletcher-Goldfarb--Shanno (BFGS) methods, the three variants of LM algorithm, and the new hybrid/LS method perform consistently well, achieving error rates of less than 3% averaged across all three architectures. [ABSTRACT FROM AUTHOR]

A novel random-gradient-based algorithm is developed for online tracking the minor component (MC) associated with the smallest eigenvalue of the autocorrelation matrix of the input vector sequence. The five available learning algorithms for tracking one MC are extended to those for tracking multiple MCs or the minor subspace (MS). In order to overcome the dynamical divergence properties of some available random-gradient-based algorithms, we propose a modification of the Oja-type algorithms, called OJAm, which can work satisfactorily. The averaging differential equation and the energy function associated with the OJAm are given. It is shown that the avenging differential equation will globally asymptotically converge to an invariance set. The corresponding energy or Lyapunov functions exhibit a unique global minimum attained if and only if its state matrices span the MS of the autocorrelation matrix of a vector data stream. The other stationary points are saddle (unstable) points. The globally convergence of OJAm is also studied. The OJAm provides an efficient online learning for tracking the MS It can track an orthonormal basis of the MS while the other five available algorithms cannot track any orthonormal basis of the MS. The performances of the relative algorithms are shown via computer simulations. [ABSTRACT FROM AUTHOR]

The prediction of protease cleavage sites in proteins is critical to effective drug design. One of the important issues in constructing an accurate and efficient predictor is how to present nonnumerical amino acids to a model effectively. As this issue has not yet been paid full attention and is closely related to model efficiency and accuracy, we present a novel neural learning algorithm aimed at improving the prediction accuracy and reducing the time involved in training. The algorithm is developed based on the conventional radial basis function neural networks (RBFNNs) and is referred to as a bio-basis function neural network (BBFNN). The basic principle is to replace the radial basis function used in RBFNNs by a novel bio-basis function. Each bio-basis is a feature dimension in a numerical feature space, to which a nonnumerical sequence space is mapped for analysis. The bio-basis function is designed using an amino acid mutation matrix verified in biology. Thus, the biological content in protein sequences can be maximally utilized for accurate modeling. Mutual information (MI) is used to select the most informative bio-bases and an ensemble method is used to enhance a decision-making process, hence, improving the prediction accuracy further. The algorithm has been successfully verified in two case studies, namely the prediction of Human Im- munodeficiency Virus (HIV) protease cleavage sites and trypsin cleavage sites in proteins. [ABSTRACT FROM AUTHOR]

The general problem of reconstructing an unknown function from a finite collection of samples is considered, in case the position of each input vector in the training set is not flied beforehand but is part of the learning process. In particular, the consistency of the empirical risk minimization (ERM) principle is analyzed, when the points in the input space are generated by employing a purely deterministic algorithm (deterministic learning). When the output generation is not subject to noise, classical number-theoretic results, involving discrepancy and variation, enable the establishment of a sufficient condition for the consistency of the ERM principle. In addition, the adoption of low-discrepancy sequences enables the achievement of a learning rate of O(1/L), with L being the size of the training set. An extension to the noisy case is provided, which Shows that the good properties of deterministic learning are preserved, if the level of noise at the output is not high. Simulation results confirm the validity of the proposed approach. [ABSTRACT FROM AUTHOR]

We consider a multilayer perceptron neural network (NN) receiver architecture for the recovery of the information bits of a direct-sequence code-division-multiple-access (DS-CDMA) user. We develop a fast converging adaptive training algorithm that minimizes the bit-error rate (BER) at the output of the receiver. The adaptive algorithm has three key features: i) it incorporates the BER, i.e., the ultimate performance evaluation measure, directly into the learning process, ii) it utilizes constraints that are derived from the properties of the optimum single-user decision boundary for additive white Gaussian noise (AWGN) multiple-access channels, and iii) it embeds importance sampling (IS) principles directly into the receiver optimization process. Simulation studies illustrate the BER performance of the proposed scheme. [ABSTRACT FROM AUTHOR]

Fuzzy logic systems have been recognized as a robust and attractive alternative to some classical control methods. The application of classical fuzzy logic (FL) technology to dynamic system control has been constrained by the nondynamic nature of popular FL architectures. Many difficulties include large rule bases (i.e., curse of dimensionality), long training times, etc. These problems can be overcome with a dynamic fuzzy network (DFN), a network with unconstrained connectivity and dynamic fuzzy processing units called "feurons." In this study, DFN as an optimal control trajectory priming system is considered as a nonlinear optimization with dynamic equality constraints. The overall algorithm operates as an autotrainer for DFN (a self-learning structure) and generates optimal feed-forward control trajectories in a significantly smaller number of iterations. For this, DEN encapsulates and generalizes the optimal control trajectories. By the algorithm, the time-varying optimal feedback gains are also generated along the trajectory as byproducts. This structure assists the speeding up of trajectory calculations for intelligent nonlinear optimal control. For this purpose, the direct-descent-curvature algorithm is used with some modifications [called modified-descend-controller (MDC) algorithm] for the nonlinear optimal control computations. The algorithm has numerically generated robust solutions with respect to conjugate points. The minimization of an integral quadratic cost functional subject to dynamic equality constraints (which is DFN) is considered for trajectory obtained by MDC tracking applications. The adjoint theory (whose computational complexity is significantly less than direct method) has been used in the training of DFN, which is as a quasilinear dynamic system. [ABSTRACT FROM AUTHOR]

Classification of underwater targets from the acoustic backscattered signals is considered here. Several different classification algorithms are tested and benchmarked not only for their performance but also to gain insight to the properties of the feature space. Results on a wideband 80-kHz acoustic backscattered data set collected for six different objects are presented in terms of the receiver operating characteristic (ROC) and robustness of the classifiers wrt reverberation. [ABSTRACT FROM AUTHOR]