Search

In this paper, we analyze the convergence of two general classes of optimization algorithms for regularized kernel methods with convex loss function and quadratic norm regularization. The first methodology is a new class of algorithms based on fixed-point iterations that are well-suited for a parallel implementation and can be used with any convex loss function. The second methodology is based on coordinate descent, and generalizes some techniques previously proposed for linear support vector machines. It exploits the structure of additively separable loss functions to compute solutions of line searches in closed form. The two methodologies are both very easy to implement. In this paper, we also show how to remove non-differentiability of the objective functional by exactly reformulating a convex regularization problem as an unconstrained differentiable stabilization problem.

This paper evaluates the four leading techniques proposed in the literature for construction of prediction intervals (PIs) for neural network point forecasts. The delta, Bayesian, bootstrap, and mean-variance estimation (MVE) methods are reviewed and their performance for generating high-quality PIs is compared. PI-based measures are proposed and applied for the objective and quantitative assessment of each method's performance. A selection of 12 synthetic and real-world case studies is used to examine each method's performance for PI construction. The comparison is performed on the basis of the quality of generated PIs, the repeatability of the results, the computational requirements and the PIs variability with regard to the data uncertainty. The obtained results in this paper indicate that: 1) the delta and Bayesian methods are the best in terms of quality and repeatability, and 2) the MVE and bootstrap methods are the best in terms of low computational load and the width variability of PIs. This paper also introduces the concept of combinations of PIs, and proposes a new method for generating combined PIs using the traditional PIs. Genetic algorithm is applied for adjusting the combiner parameters through minimization of a PI-based cost function subject to two sets of restrictions. It is shown that the quality of PIs produced by the combiners is dramatically better than the quality of PIs obtained from each individual method.

The algorithm and the results of a blind multiuser detector using a machine learning technique called support vector machine (SVM) on a chaos-based code division multiple access system is presented in this paper. Simulation results showed that the performance achieved by using SVM is comparable to existing minimum mean square error (MMSE) detector under both additive white Gaussian noise (AWGN) and Rayleigh fading conditions. However, unlike the MMSE detector, the SVM detector does not require the knowledge of spreading codes of other users in the system or the estimate of the channel noise variance. The optimization of this algorithm is considered in this paper and its complexity is compared with the MMSE detector. This detector is much more suitable to work in the forward link than MMSE. In addition, original theoretical bit-error rate expressions for the SVM detector under both AWGN and Rayleigh fading are derived to verify the simulation results.

Negative correlation learning (NCL) is a neural network ensemble learning algorithm that introduces a correlation penalty term to the cost function of each individual network so that each neural network minimizes its mean square error (MSE) together with the correlation of the ensemble. This paper analyzes NCL and reveals that the training of NCL (when lambda = 1) corresponds to training the entire ensemble as a single learning machine that only minimizes the MSE without regularization. This analysis explains the reason why NCL is prone to overfitting the noise in the training set. This paper also demonstrates that tuning the correlation parameter lambda in NCL by cross validation cannot overcome the overfitting problem. The paper analyzes this problem and proposes the regularized negative correlation learning (RNCL) algorithm which incorporates an additional regularization term for the whole ensemble. RNCL decomposes the ensemble's training objectives, including MSE and regularization, into a set of sub-objectives, and each sub-objective is implemented by an individual neural network. In this paper, we also provide a Bayesian interpretation for RNCL and provide an automatic algorithm to optimize regularization parameters based on Bayesian inference. The RNCL formulation is applicable to any nonlinear estimator minimizing the MSE. The experiments on synthetic as well as real-world data sets demonstrate that RNCL achieves better performance than NCL, especially when the noise level is nontrivial in the data set.

It has been proven that there are synchrony (or phase-locking) phenomena present in multiple oscillating systems such as electrical circuits, lasers, chemical reactions, and human neurons. If the measurements of these systems cannot detect the individual oscillators but rather a superposition of them, as in brain electrophysiological signals (electro- and magneoencephalogram), spurious phase locking will be detected. Current source-extraction techniques attempt to undo this superposition by assuming properties on the data, which are not valid when underlying sources are phase-locked. Statistical independence of the sources is one such invalid assumption, as phase-locked sources are dependent. In this paper, we introduce methods for source separation and clustering which make adequate assumptions for data where synchrony is present, and show with simulated data that they perform well even in cases where independent component analysis and other well-known source-separation methods fail. The results in this paper provide a proof of concept that synchrony-based techniques are useful for low-noise applications.

This paper proposes three novel training methods, two of them based on the backpropagation approach and a third one based on information theory for multilayer perceptron (MLP) binary classifiers. Both backpropagation methods are based on the maximal-margin (MM) principle. The first one, based on the gradient descent with adaptive learning rate algorithm (GDX) and named maximum-margin GDX (MMGDX), directly increases the margin of the MLP output-layer hyperplane. The proposed method jointly optimizes both MLP layers in a single process, backpropagating the gradient of an MM-based objective function, through the output and hidden layers, in order to create a hidden-layer space that enables a higher margin for the output-layer hyperplane, avoiding the testing of many arbitrary kernels, as occurs in case of support vector machine (SVM) training. The proposed MM-based objective function aims to stretch out the margin to its limit. An objective function based on Lp-norm is also proposed in order to take into account the idea of support vectors, however, overcoming the complexity involved in solving a constrained optimization problem, usually in SVM training. In fact, all the training methods proposed in this paper have time and space complexities O(N) while usual SVM training methods have time complexity O(N (3)) and space complexity O(N (2)) , where N is the training-data-set size. The second approach, named minimization of interclass interference (MICI), has an objective function inspired on the Fisher discriminant analysis. Such algorithm aims to create an MLP hidden output where the patterns have a desirable statistical distribution. In both training methods, the maximum area under ROC curve (AUC) is applied as stop criterion. The third approach offers a robust training framework able to take the best of each proposed training method. The main idea is to compose a neural model by using neurons extracted from three other neural networks, each one previously trained by MICI, MMGDX, and Levenberg-Marquard (LM), respectively. The resulting neural network was named assembled neural network (ASNN). Benchmark data sets of real-world problems have been used in experiments that enable a comparison with other state-of-the-art classifiers. The results provide evidence of the effectiveness of our methods regarding accuracy, AUC, and balanced error rate.

This paper studies a general class of dynamical neural networks with lateral inhibition, exhibiting winner-take-all (WTA) behavior. These networks are motivated by a metal-oxide-semiconductor field effect transistor (MOSFET) implementation of neural networks, in which mutual competition plays a very important role. We show that for a fairly general class of competitive neural networks, WTA behavior exists. Sufficient conditions for the network to have a WTA equilibrium are obtained, and rigorous convergence analysis is carried out. The conditions for the network to have the WTA behavior obtained in this paper provide design guidelines for the network implementation and fabrication. We also demonstrate that whenever the network gets into the WTA region, it will stay in that region and settle down exponentially fast to the WTA point. This provides a speeding procedure for the decision making: as soon as it gets into the region, the winner can be declared. Finally, we show that this WTA neural network has a self-resetting property, and a resetting principle is proposed.

Motivated by the reportedly strong performance of particle filters (PFs) for noise reduction on essentially linear speech production models, and the mounting evidence that the introduction of nonlinearities can lead to a refined speech model, this paper presents a study of PF solutions to the problem of speech enhancement in the context of nonlinear, neural-type speech models. Several variations of a global model are presented (single/multiple neurons; bias/no bias), and corresponding PF solutions are derived. Different importance functions are given when beneficial, Rao-Blackwellization is proposed when possible, and dual/nondual versions of each algorithms are presented. The method shown can handle both white and colored noise. Using a variety of speech and noise signals and different objective quality measures, the performance of these algorithms are evaluated against other PF solutions running on linear models, as well as some traditional enhancement algorithms. A certain hierarchy in performance is established between each algorithm in the paper. Depending on the experimental conditions, the best-performing algorithms are a classical Rao-Blackwellized particle filter (RBPF) running on a linear model, and a proposed PF employing a nondual, nonlinear model with multiple neurons and no biases. With consistence, the neural-network-based PF outperforms RBPF at low signal-to-noise ratio (SNR).

The face is a complex multidimensional visual model and developing a computational model for face recognition is difficult. In this paper, we present a method for face recognition based on parallel neural networks. Neural networks (NNs) have been widely used in various fields. However, the computing efficiency decreases rapidly if the scale of the NN increases. In this paper, a new method of face recognition based on fuzzy clustering and parallel NNs is proposed. The face patterns are divided into several small-scale neural networks based on fuzzy clustering and they are combined to obtain the recognition result. In particular, the proposed method achieved a 98.75 % recognition accuracy for 240 patterns of 20 registrants and a 99.58% rejection rate for 240 patterns of 20 nonregistrants. Experimental results show that the performance of our new face-recognition method is better than those of the backpropagation NN (BPNN) system, the hard c-means (HCM) and parallel NNs system, and the pattern-matching system.

With the aim of getting understandable symbolic rules to explain a given phenomenon, we split the task of learning these rules from sensory data in two phases: a multilayer perceptron maps features into propositional variables and a set of subsequent layers operated by a PAC-like algorithm learns Boolean expressions on these variables. The special features of this procedure are that: i) the neural network is trained to produce a Boolean output having the principal task of discriminating between classes of inputs; ii) the symbolic part is directed to compute rules within a family that is not known a priori; iii) the welding point between the two learning systems is represented by a feedback based on a suitability evaluation of the computed rules. The procedure we propose is based on a computational learning paradigm set up recently in some papers in the fields of theoretical computer science, artificial intelligence and cognitive systems. The present article focuses on information management aspects of the procedure. We deal with the lack of prior information about the rules through learning strategies that affect both the meaning of the variables and the description length of the rules into which they combine. The paper uses the task of learning to formally discriminate among several emotional states as both a working example and a test bench for a comparison with previous symbolic and subsymbolic methods in the field.

There are a number of spiking and bursting neuron models with varying levels of complexity, ranging from the simple integrate-and-fire model to the more complex Hodgkin-Huxley model. The simpler models tend to be easily implemented in silicon but yet not biologically plausible. Conversely, the more complex models tend to occupy a large area although they are more biologically plausible. In this paper, we present the 0.5 μm complementary metal-oxide-semiconductor (CMOS) implementation of the Mihalaş-Niebur neuron model--a generalized model of the leaky integrate-and-fire neuron with adaptive threshold--that is able to produce most of the known spiking and bursting patterns that have been observed in biology. Our implementation modifies the original proposed model, making it more amenable to CMOS implementation and more biologically plausible. All but one of the spiking properties--tonic spiking, class 1 spiking, phasic spiking, hyperpolarized spiking, rebound spiking, spike frequency adaptation, accommodation, threshold variability, integrator and input bistability--are demonstrated in this model.

This paper presents an energy-efficient fast independent component analysis (FastICA) implementation with an early determination scheme for eight-channel electroencephalogram (EEG) signal separation. The main contributions are as follows: (1) energy-efficient FastICA using the proposed early determination scheme and the corresponding architecture; (2) cost-effective FastICA using the proposed preprocessing unit architecture with one coordinate rotation digital computer-based eigenvalue decomposition processor and the proposed one-unit architecture with the hardware reuse scheme; and (3) low-computation-time FastICA using the four parallel one-units architecture. The resulting power dissipation of the FastICA implementation for eight-channel EEG signal separation is 16.35 mW at 100 MHz at 1.0 V. Compared with the design without early determination, the proposed FastICA architecture implemented in united microelectronics corporation 90 nm 1P9M complementary metal-oxide-semiconductor process with a core area of 1.221 × 1.218 mm2 can achieve average energy reduction by 47.63%. From the post-layout simulation results, the maximum computation time is 0.29 s.

We introduce an ensemble of classifiers-based approach for incremental learning of concept drift, characterized by nonstationary environments (NSEs), where the underlying data distributions change over time. The proposed algorithm, named Learn(++). NSE, learns from consecutive batches of data without making any assumptions on the nature or rate of drift; it can learn from such environments that experience constant or variable rate of drift, addition or deletion of concept classes, as well as cyclical drift. The algorithm learns incrementally, as other members of the Learn(++) family of algorithms, that is, without requiring access to previously seen data. Learn(++). NSE trains one new classifier for each batch of data it receives, and combines these classifiers using a dynamically weighted majority voting. The novelty of the approach is in determining the voting weights, based on each classifier's time-adjusted accuracy on current and past environments. This approach allows the algorithm to recognize, and act accordingly, to the changes in underlying data distributions, as well as to a possible reoccurrence of an earlier distribution. We evaluate the algorithm on several synthetic datasets designed to simulate a variety of nonstationary environments, as well as a real-world weather prediction dataset. Comparisons with several other approaches are also included. Results indicate that Learn(++). NSE can track the changing environments very closely, regardless of the type of concept drift. To allow future use, comparison and benchmarking by interested researchers, we also release our data used in this paper.

A novel framework using a Bayesian approach for content-based phishing web page detection is presented. Our model takes into account textual and visual contents to measure the similarity between the protected web page and suspicious web pages. A text classifier, an image classifier, and an algorithm fusing the results from classifiers are introduced. An outstanding feature of this paper is the exploration of a Bayesian model to estimate the matching threshold. This is required in the classifier for determining the class of the web page and identifying whether the web page is phishing or not. In the text classifier, the naive Bayes rule is used to calculate the probability that a web page is phishing. In the image classifier, the earth mover's distance is employed to measure the visual similarity, and our Bayesian model is designed to determine the threshold. In the data fusion algorithm, the Bayes theory is used to synthesize the classification results from textual and visual content. The effectiveness of our proposed approach was examined in a large-scale dataset collected from real phishing cases. Experimental results demonstrated that the text classifier and the image classifier we designed deliver promising results, the fusion algorithm outperforms either of the individual classifiers, and our model can be adapted to different phishing cases.

To reduce the computational complexity of the bilinear recurrent neural network (BLRNN), a novel low-complexity nonlinear adaptive filter with a pipelined bilinear recurrent neural network (PBLRNN) is presented in this paper. The PBLRNN, inheriting the modular architectures of the pipelined RNN proposed by Haykin and Li, comprises a number of BLRNN modules that are cascaded in a chained form. Each module is implemented by a small-scale BLRNN with internal dynamics. Since those modules of the PBLRNN can be performed simultaneously in a pipelined parallelism fashion, it would result in a significant improvement of computational efficiency. Moreover, due to nesting module, the performance of the PBLRNN can be further improved. To suit for the modular architectures, a modified adaptive amplitude real-time recurrent learning algorithm is derived on the gradient descent approach. Extensive simulations are carried out to evaluate the performance of the PBLRNN on nonlinear system identification, nonlinear channel equalization, and chaotic time series prediction. Experimental results show that the PBLRNN provides considerably better performance compared to the single BLRNN and RNN models.

The identification of nonlinear spatiotemporal systems is of significance to engineering practice, since it can always provide useful insight into the underlying nonlinear mechanism and physical characteristics under study. In this paper, nonlinear spatiotemporal system models are transformed into a class of multi-input-multi-output (MIMO) partially linear systems (PLSs), and an effective online identification algorithm is therefore proposed by using a pruning error minimization principle and least square support vector machines. It is shown that many benchmark physical and engineering systems can be transformed into MIMO-PLSs which keep some important physical spatiotemporal relationships and are very helpful in the identification and analysis of the underlying system. Compared with several existing methods, the advantages of the proposed method are that it can make full use of some prior structural information about system physical models, can realize online estimation of the system dynamics, and achieve accurate characterization of some important nonlinear physical characteristics of the system. This would provide an important basis for state estimation, control, optimal analysis, and design of nonlinear distributed parameter systems. The proposed algorithm can also be applied to identification problems of stochastic spatiotemporal dynamical systems. Numeral examples and comparisons are given to demonstrate our results.

In this paper, two adaptive neural network (NN) decentralized output feedback control approaches are proposed for a class of uncertain nonlinear large-scale systems with immeasurable states and unknown time delays. Using NNs to approximate the unknown nonlinear functions, an NN state observer is designed to estimate the immeasurable states. By combining the adaptive backstepping technique with decentralized control design principle, an adaptive NN decentralized output feedback control approach is developed. In order to overcome the problem of "explosion of complexity" inherent in the proposed control approach, the dynamic surface control (DSC) technique is introduced into the first adaptive NN decentralized control scheme, and a simplified adaptive NN decentralized output feedback DSC approach is developed. It is proved that the two proposed control approaches can guarantee that all the signals of the closed-loop system are semi-globally uniformly ultimately bounded, and the observer errors and the tracking errors converge to a small neighborhood of the origin. Simulation results are provided to show the effectiveness of the proposed approaches.

This paper describes an artificial neuron structure and an efficient learning procedure in the complex domain. This artificial neuron aims at incorporating an improved aggregation operation on the complex-valued signals. The aggregation operation is based on the idea underlying the weighted root-power mean of input signals. This aggregation operation allows modeling the degree of compensation in a natural manner and includes various aggregation operations as its special cases. The complex resilient propagation algorithm ([Formula: see text]-RPROP) with error-dependent weight backtracking step accelerates the training speed significantly and provides better approximation accuracy. Finally, performance evaluation of the proposed complex root-power mean neuron with the [Formula: see text]-RPROP learning algorithm on various typical examples is given to understand the motivation.

Bayesian reasoning provides an ideal basis for representing and manipulating uncertain knowledge, with the result that many interesting algorithms in machine learning are based on Bayesian inference. In this paper, we use the Bayesian approach with one and two levels of inference to model the semisupervised learning problem and give its application to the successful kernel classifier support vector machine (SVM) and its variant least-squares SVM (LS-SVM). Taking advantage of Bayesian interpretation of LS-SVM, we develop a semisupervised learning algorithm for Bayesian LS-SVM using our approach based on two levels of inference. Experimental results on both artificial and real pattern recognition problems show the utility of our method.

In this paper, we apply an immune-inspired approach to design ensembles of heterogeneous neural networks for classification problems. Our proposal, called Bayesian artificial immune system, is an estimation of distribution algorithm that replaces the traditional mutation and cloning operators with a probabilistic model, more specifically a Bayesian network, representing the joint distribution of promising solutions. Among the additional attributes provided by the Bayesian framework inserted into an immune-inspired search algorithm are the automatic control of the population size along the search and the inherent ability to promote and preserve diversity among the candidate solutions. Both are attributes generally absent from alternative estimation of distribution algorithms, and both were shown to be useful attributes when implementing the generation and selection of components of the ensemble, thus leading to high-performance classifiers. Several aspects of the design are illustrated in practical applications, including a comparative analysis with other attempts to synthesize ensembles.

Projection techniques are frequently used as the principal means for the implementation of feature extraction and dimensionality reduction for machine learning applications. A well established and broad class of such projection techniques is the projection pursuit (PP). Its core design parameter is a projection index, which is the driving force in obtaining the transformation function via optimization, and represents in an explicit or implicit way the user's perception of the useful information contained within the datasets. This paper seeks to address the problem related to the design of PP index functions for the linear feature extraction case. We achieve this using an evolutionary search framework, capable of building new indices to fit the properties of the available datasets. The high expressive power of this framework is sustained by a rich set of function primitives. The performance of several PP indices previously proposed by human experts is compared with these automatically generated indices for the task of classification, and results show a decrease in the classification errors.

Backpropagation is the most widely used neural network learning technique. It is based on the mathematical notion of an ordered derivative. In this paper, we present a formulation of ordered derivatives and the backpropagation training algorithm using the important emerging area of mathematics known as the time scales calculus. This calculus, with its potential for application to a wide variety of inter-disciplinary problems, is becoming a key area of mathematics. It is capable of unifying continuous and discrete analysis within one coherent theoretical framework. Using this calculus, we present here a generalization of backpropagation which is appropriate for cases beyond the specifically continuous or discrete. We develop a new multivariate chain rule of this calculus, define ordered derivatives on time scales, prove a key theorem about them, and derive the backpropagation weight update equations for a feedforward multilayer neural network architecture. By drawing together the time scales calculus and the area of neural network learning, we present the first connection of two major fields of research.

Recently, new neuro-fuzzy inference algorithms have been developed to deal with the time-varying behavior and uncertainty of many complex systems. This paper presents the design and application of a novel transductive neuro-fuzzy inference method to control force in a high-performance drilling process. The main goal is to study, analyze, and verify the behavior of a transductive neuro-fuzzy inference system for controlling this complex process, specifically addressing the dynamic modeling, computational efficiency, and viability of the real-time application of this algorithm as well as assessing the topology of the neuro-fuzzy system (e.g., number of clusters, number of rules). A transductive reasoning method is used to create local neuro-fuzzy models for each input/output data set in a case study. The direct and inverse dynamics of a complex process are modeled using this strategy. The synergies among fuzzy, neural, and transductive strategies are then exploited to deal with process complexity and uncertainty through the application of the neuro-fuzzy models within an internal model control (IMC) scheme. A comparative study is made of the adaptive neuro-fuzzy inference system (ANFIS) and the suggested method inspired in a transductive neuro-fuzzy inference strategy. The two neuro-fuzzy strategies are evaluated in a real drilling force control problem. The experimental results demonstrated that the transductive neuro-fuzzy control system provides a good transient response (without overshoot) and better error-based performance indices than the ANFIS-based control system. In particular, the IMC system based on a transductive neuro-fuzzy inference approach reduces the influence of the increase in cutting force that occurs as the drill depth increases, reducing the risk of rapid tool wear and catastrophic tool breakage.

We propose a multiple incremental decremental algorithm of support vector machines (SVM). In online learning, we need to update the trained model when some new observations arrive and/or some observations become obsolete. If we want to add or remove single data point, conventional single incremental decremental algorithm can be used to update the model efficiently. However, to add and/or remove multiple data points, the computational cost of current update algorithm becomes inhibitive because we need to repeatedly apply it for each data point. In this paper, we develop an extension of incremental decremental algorithm which efficiently works for simultaneous update of multiple data points. Some analyses and experimental results show that the proposed algorithm can substantially reduce the computational cost. Our approach is especially useful for online SVM learning in which we need to remove old data points and add new data points in a short amount of time.

In the last two decades, many online fault/noise injection algorithms have been developed to attain a fault tolerant neural network. However, not much theoretical works related to their convergence and objective functions have been reported. This paper studies six common fault/noise-injection-based online learning algorithms for radial basis function (RBF) networks, namely 1) injecting additive input noise, 2) injecting additive/multiplicative weight noise, 3) injecting multiplicative node noise, 4) injecting multiweight fault (random disconnection of weights), 5) injecting multinode fault during training, and 6) weight decay with injecting multinode fault. Based on the Gladyshev theorem, we show that the convergence of these six online algorithms is almost sure. Moreover, their true objective functions being minimized are derived. For injecting additive input noise during training, the objective function is identical to that of the Tikhonov regularizer approach. For injecting additive/multiplicative weight noise during training, the objective function is the simple mean square training error. Thus, injecting additive/multiplicative weight noise during training cannot improve the fault tolerance of an RBF network. Similar to injective additive input noise, the objective functions of other fault/noise-injection-based online algorithms contain a mean square error term and a specialized regularization term.

One of the most vexing issues in design of a high-speed computer is the wide gap of access times between the memory and the disk. To solve this problem, static file access predictors have been used. In this paper, we propose dynamic file access predictors using neural networks to significantly improve upon the accuracy, success-per-reference, and effective-success-rate-per-reference by using neural-network-based file access predictor with proper tuning. In particular, we verified that the incorrect prediction has been reduced from 53.11% to 43.63% for the proposed neural network prediction method with a standard configuration than the recent popularity (RP) method. With manual tuning for each trace, we are able to improve upon the misprediction rate and effective-success-rate-per-reference using a standard configuration. Simulations on distributed file system (DFS) traces reveal that exact fit radial basis function (RBF) gives better prediction in high end system whereas multilayer perceptron (MLP) trained with Levenberg-Marquardt (LM) backpropagation outperforms in system having good computational capability. Probabilistic and competitive predictors are the most suitable for work stations having limited resources to deal with and the former predictor is more efficient than the latter for servers having maximum system calls. Finally, we conclude that MLP with LM backpropagation algorithm has better success rate of file prediction than those of simple perceptron, last successor, stable successor, and best k out of m predictors.

In this paper, a training method for the formation of topology preserving maps is introduced. The proposed approach presents a sequential formulation of the self-organizing map (SOM), which is based on a new model of the neuron, or processing unit. Each neuron acts as a finite impulse response (FIR) system, and the coefficients of the filters are adaptively estimated during the sequential learning process, in order to minimize a distortion measure of the map. The proposed FIR-SOM model deals with static distributions and it computes an ordered set of centroids. Additionally, the FIR-SOM estimates the learning dynamic of each prototype using an adaptive FIR model. A noteworthy result is that the optimized coefficients of the FIR processes tend to represent a moving average filter, regardless of the underlying input distribution. The convergence of the resulting model is analyzed numerically and shows good properties with respect to the classic SOM and other unsupervised neural models. Finally, the optimal FIR coefficients are shown to be useful for visualizing the cluster densities.

This paper proposes a multiclassification algorithm using multilayer perceptron neural network models. It tries to boost two conflicting main objectives of multiclassifiers: a high correct classification rate level and a high classification rate for each class. This last objective is not usually optimized in classification, but is considered here given the need to obtain high precision in each class in real problems. To solve this machine learning problem, we use a Pareto-based multiobjective optimization methodology based on a memetic evolutionary algorithm. We consider a memetic Pareto evolutionary approach based on the NSGA2 evolutionary algorithm (MPENSGA2). Once the Pareto front is built, two strategies or automatic individual selection are used: the best model in accuracy and the best model in sensitivity (extremes in the Pareto front). These methodologies are applied to solve 17 classification benchmark problems obtained from the University of California at Irvine (UCI) repository and one complex real classification problem. The models obtained show high accuracy and a high classification rate for each class.

In this paper, a generalized Brain-State-in-a-Box (gBSB)-based hybrid neural network is proposed for storing and retrieving pattern sequences. The hybrid network consists of autoassociative and heteroassociative parts. Then, a large-scale image storage and retrieval neural system is constructed using the gBSB-based hybrid neural network and the pattern decomposition concept. The notion of the deadbeat stability is employed to describe the stability property of the vertices of the hypercube to which the trajectories of the gBSB neural system are constrained. Extensive simulations of large scale pattern and image storing and retrieval are presented to illustrate the results obtained.

Generally, training neural networks with the global extended Kalman filter (GEKF) technique exhibits excellent performance at the expense of a large increase in computational costs which can become prohibitive even for networks of moderate size. This drawback was previously addressed by heuristically decoupling some of the weights of the networks. Inevitably, ad hoc decoupling leads to a degradation in the quality (accuracy) of the resultant neural networks. In this paper, we present an algorithm that emulates the accuracy of GEKF, but avoids the construction of the state covariance matrix-the source of the computational bottleneck in GEKF. In the proposed algorithm, all the synaptic weights remain connected while the amount of computer memory required is similar to (or cheaper than) the memory requirements in the decoupling schemes. We also point out that the new method can be extended to derivative-free nonlinear Kalman filters, such as the unscented Kalman filter and ensemble Kalman filters.

In this paper, a novel, unified model-based fault-detection and prediction (FDP) scheme is developed for nonlinear multiple-input-multiple-output (MIMO) discrete-time systems. The proposed scheme addresses both state and output faults by considering separate time profiles. The faults, which could be incipient or abrupt, are modeled using input and output signals of the system. The fault-detection (FD) scheme comprises online approximator in discrete time (OLAD) with a robust adaptive term. An output residual is generated by comparing the FD estimator output with that of the measured system output. A fault is detected when this output residual exceeds a predefined threshold. Upon detecting the fault, the robust adaptive terms and the OLADs are initiated wherein the OLAD approximates the unknown fault dynamics online while the robust adaptive terms help in ensuring asymptotic stability of the FD design. Using the OLAD outputs, a fault diagnosis scheme is introduced. A stable parameter update law is developed not only to tune the OLAD parameters but also to estimate the time to failure (TTF), which is considered as a first step for prognostics. The asymptotic stability of the FDP scheme enhances the detection and TTF accuracy. The effectiveness of the proposed approach is demonstrated using a fourth-order MIMO satellite system.

International audience; Independent component analysis (ICA) aims at decomposing an observed random vector into statistically independent variables. Deflation-based implementations, such as the popular one-unit FastICA algorithm and its variants, extract the independent components one after another. A novel method for deflationary ICA, referred to as RobustICA, is put forward in this paper. This simple technique consists of performing exact line search optimization of the kurtosis contrast function. The step size leading to the global maximum of the contrast along the search direction is found among the roots of a fourth-degree polynomial. This polynomial rooting can be performed algebraically, and thus at low cost, at each iteration. Among other practical benefits, RobustICA can avoid prewhitening and deals with real- and complex-valued mixtures of possibly noncircular sources alike. The absence of prewhitening improves asymptotic performance. The algorithm is robust to local extrema and shows a very high convergence speed in terms of the computational cost required to reach a given source extraction quality, particularly for short data records. These features are demonstrated by a comparative numerical analysis on synthetic data. RobustICA's capabilities in processing real-world data involving noncircular complex strongly super-Gaussian sources are illustrated by the biomedical problem of atrial activity (AA) extraction in atrial fibrillation (AF) electrocardiograms (ECGs), where it outperforms an alternative ICA-based technique.

Fuzzy rule-based systems (FRBSs) have been successfully applied to many areas. However, traditional fuzzy systems are often manually crafted, and their rule bases that represent the acquired knowledge are static and cannot be trained to improve the modeling performance. This subsequently leads to intensive research on the autonomous construction and tuning of a fuzzy system directly from the observed training data to address the knowledge acquisition bottleneck, resulting in well-established hybrids such as neural-fuzzy systems (NFSs) and genetic fuzzy systems (GFSs). However, the complex and dynamic nature of real-world problems demands that fuzzy rule-based systems and models be able to adapt their parameters and ultimately evolve their rule bases to address the nonstationary (time-varying) characteristics of their operating environments. Recently, considerable research efforts have been directed to the study of evolving Tagaki-Sugeno (T-S)-type NFSs based on the concept of incremental learning. In contrast, there are very few incremental learning Mamdani-type NFSs reported in the literature. Hence, this paper presents the evolving neural-fuzzy semantic memory (eFSM) model, a neural-fuzzy Mamdani architecture with a data-driven progressively adaptive structure (i.e., rule base) based on incremental learning. Issues related to the incremental learning of the eFSM rule base are carefully investigated, and a novel parameter learning approach is proposed for the tuning of the fuzzy set parameters in eFSM. The proposed eFSM model elicits highly interpretable semantic knowledge in the form of Mamdani-type if-then fuzzy rules from low-level numeric training data. These Mamdani fuzzy rules define the computing structure of eFSM and are incrementally learned with the arrival of each training data sample. New rules are constructed from the emergence of novel training data and obsolete fuzzy rules that no longer describe the recently observed data trends are pruned. This enables eFSM to maintain a current and compact set of Mamdani-type if-then fuzzy rules that collectively generalizes and describes the salient associative mappings between the inputs and outputs of the underlying process being modeled. The learning and modeling performances of the proposed eFSM are evaluated using several benchmark applications and the results are encouraging.

This paper proposes a new document retrieval (DR) and plagiarism detection (PD) system using multilayer self-organizing map (MLSOM). A document is modeled by a rich tree-structured representation, and a SOM-based system is used as a computationally effective solution. Instead of relying on keywords/lines, the proposed scheme compares a full document as a query for performing retrieval and PD. The tree-structured representation hierarchically includes document features as document, pages, and paragraphs. Thus, it can reflect underlying context that is difficult to acquire from the currently used word-frequency information. We show that the tree-structured data is effective for DR and PD. To handle tree-structured representation in an efficient way, we use an MLSOM algorithm, which was previously developed by the authors for the application of image retrieval. In this study, it serves as an effective clustering algorithm. Using the MLSOM, local matching techniques are developed for comparing text documents. Two novel MLSOM-based PD methods are proposed. Detailed simulations are conducted and the experimental results corroborate that the proposed approach is computationally efficient and accurate for DR and PD.

In this paper, we consider a learning strategy that allows one to transmit information between two coupled phase oscillator systems (called teaching and learning systems) via frequency adaptation. The dynamics of these systems can be modeled with reference to a number of partially synchronized cluster states and transitions between them. Forcing the teaching system by steady but spatially nonhomogeneous inputs produces cyclic sequences of transitions between the cluster states, that is, information about inputs is encoded via a ldquowinnerless competitionrdquo process into spatio-temporal codes. The large variety of codes can be learned by the learning system that adapts its frequencies to those of the teaching system. We visualize the dynamics using ldquoweighted order parameters (WOPs)rdquo that are analogous to ldquolocal field potentialsrdquo in neural systems. Since spatio-temporal coding is a mechanism that appears in olfactory systems, the developed learning rules may help to extract information from these neural ensembles.

This paper presents a novel blood glucose regulation for type I (insulin-dependent) diabetes mellitus patients using biologically inspired TSK0-FCMAC, a fuzzy cerebellar model articulation controller (CMAC) based on the zero-ordered Takagi-Sugeno-Kang (TSK) fuzzy inference scheme. TSK0 -FCMAC is capable of performing localized online training with an effective fuzzy inference scheme that could respond swiftly to changing environment such as human's endocrine system. Without prior knowledge of disturbance (e.g., food intake), the proposed fuzzy CMAC is able to capture the glucose-insulin dynamics of individuals under different dietary profiles. Preliminary simulations show that the blood glucose level is kept within the state of euglycemia. The design of the proposed system follows closely to what is available in real life and is suitable for animal and clinical pilot testing in the near future.

Advancements in biometrics-based authentication have led to its increasing prominence and are being incorporated into everyday tasks. Existing vehicle security systems rely only on alarms or smart card as forms of protection. A biometric driver recognition system utilizing driving behaviors is a highly novel and personalized approach and could be incorporated into existing vehicle security system to form a multimodal identification system and offer a greater degree of multilevel protection. In this paper, detailed studies have been conducted to model individual driving behavior in order to identify features that may be efficiently and effectively used to profile each driver. Feature extraction techniques based on Gaussian mixture models (GMMs) are proposed and implemented. Features extracted from the accelerator and brake pedal pressure were then used as inputs to a fuzzy neural network (FNN) system to ascertain the identity of the driver. Two fuzzy neural networks, namely, the evolving fuzzy neural network (EFuNN) and the adaptive network-based fuzzy inference system (ANFIS), are used to demonstrate the viability of the two proposed feature extraction techniques. The performances were compared against an artificial neural network (NN) implementation using the multilayer perceptron (MLP) network and a statistical method based on the GMM. Extensive testing was conducted and the results show great potential in the use of the FNN for real-time driver identification and verification. In addition, the profiling of driver behaviors has numerous other potential applications for use by law enforcement and companies dealing with buses and truck drivers.

Wearable human movement measurement systems are increasingly popular as a means of capturing human movement data in real-world situations. Previous work has attempted to estimate segment kinematics during walking from foot acceleration and angular velocity data. In this paper, we propose a novel neural network [GRNN with Auxiliary Similarity Information (GASI)] that estimates joint kinematics by taking account of proximity and gait trajectory slope information through adaptive weighting. Furthermore, multiple kernel bandwidth parameters are used that can adapt to the local data density. To demonstrate the value of the GASI algorithm, hip, knee, and ankle joint motions are estimated from acceleration and angular velocity data for the foot and shank, collected using commercially available wearable sensors. Reference hip, knee, and ankle kinematic data were obtained using externally mounted reflective markers and infrared cameras for subjects while they walked at different speeds. The results provide further evidence that a neural net approach to the estimation of joint kinematics is feasible and shows promise, but other practical issues must be addressed before this approach is mature enough for clinical implementation. Furthermore, they demonstrate the utility of the new GASI algorithm for making estimates from continuous periodic data that include noise and a significant level of variability.

In this paper, a supervised neural network training technique based on constrained optimization is developed for preserving prior knowledge of an input-output mapping during repeated incremental training sessions. The prior knowledge, referred to as long-term memory (LTM), is expressed in the form of equality constraints obtained by means of an algebraic training technique. Incremental training, which may be used to learn new short-term memories (STMs) online, is then formulated as an error minimization problem subject to equality constraints. The solution of this problem is simplified by implementing an adjoined error gradient that circumvents direct substitution and exploits classical backpropagation. A target application is neural network function approximation in adaptive critic designs. For illustrative purposes, constrained training is implemented to update an adaptive critic flight controller, while preserving prior knowledge of an established performance baseline that consists of classical gain-scheduled controllers. It is shown both analytically and numerically that the LTM is accurately preserved while the controller is repeatedly trained over time to assimilate new STMs.

Fast independent component analysis (FastICA) algorithm separates the independent sources from their mixtures by measuring non-Gaussian. FastICA is a common offline method to identify artifact and interference from their mixtures such as electroencephalogram (EEG), magnetoencephalography (MEG), and electrocardiogram (ECG). Therefore, it is valuable to implement FastICA for real-time signal processing. In this paper, the FastICA algorithm is implemented in a field-programmable gate array (FPGA), with the ability of real-time sequential mixed signals processing by the proposed pipelined FastICA architecture. Moreover, in order to increase the numbers precision, the hardware floating-point (FP) arithmetic units had been carried out in the hardware FastICA. In addition, the proposed pipeline FastICA provides the high sampling rate (192 kHz) capability by hand coding the hardware FastICA in hardware description language (HDL). To verify the features of the proposed hardware FastICA, simulations are first performed, then real-time signal processing experimental results are presented using the fabricated platform. Experimental results demonstrate the effectiveness of the presented hardware FastICA as expected.

Global convergence of the sequential minimal optimization (SMO) algorithm for support vector regression (SVR) is studied in this paper. Given l training samples, SVR is formulated as a convex quadratic programming (QP) problem with l pairs of variables. We prove that if two pairs of variables violating the optimality condition are chosen for update in each step and subproblems are solved in a certain way, then the SMO algorithm always stops within a finite number of iterations after finding an optimal solution. Also, efficient implementation techniques for the SMO algorithm are presented and compared experimentally with other SMO algorithms.

In this paper, we propose a multiresolution approach for surface reconstruction from clouds of unorganized points representing an object surface in 3-D space. The proposed method uses a set of mesh operators and simple rules for selective mesh refinement, with a strategy based on Kohonen's self-organizing map (SOM). Basically, a self-adaptive scheme is used for iteratively moving vertices of an initial simple mesh in the direction of the set of points, ideally the object boundary. Successive refinement and motion of vertices are applied leading to a more detailed surface, in a multiresolution, iterative scheme. Reconstruction was experimented on with several point sets, including different shapes and sizes. Results show generated meshes very close to object final shapes. We include measures of performance and discuss robustness.

Large-margin methods, such as support vector machines (SVMs), have been very successful in classification problems. Recently, maximum margin discriminant analysis (MMDA) was proposed that extends the large-margin idea to feature extraction. It often outperforms traditional methods such as kernel principal component analysis (KPCA) and kernel Fisher discriminant analysis (KFD). However, as in the SVM, its time complexity is cubic in the number of training points m, and is thus computationally inefficient on massive data sets. In this paper, we propose an (1+epsilon)(2)-approximation algorithm for obtaining the MMDA features by extending the core vector machine. The resultant time complexity is only linear in m, while its space complexity is independent of m. Extensive comparisons with the original MMDA, KPCA, and KFD on a number of large data sets show that the proposed feature extractor can improve classification accuracy, and is also faster than these kernel-based methods by over an order of magnitude.

A magnetic-levitation (maglev) transportation system including levitation and propulsion control is a subject of considerable scientific interest because of highly nonlinear and unstable behaviors. In this paper, the dynamic model of a maglev transportation system including levitated electromagnets and a propulsive linear induction motor (LIM) based on the concepts of mechanical geometry and motion dynamics is developed first. Then, a model-based sliding-mode control (SMC) strategy is introduced. In order to alleviate chattering phenomena caused by the inappropriate selection of uncertainty bound, a simple bound estimation algorithm is embedded in the SMC strategy to form an adaptive sliding-mode control (ASMC) scheme. However, this estimation algorithm is always a positive value so that tracking errors introduced by any uncertainty will cause the estimated bound increase even to infinity with time. Therefore, it further designs an adaptive fuzzy-neural-network control (AFNNC) scheme by imitating the SMC strategy for the maglev transportation system. In the model-free AFNNC, online learning algorithms are designed to cope with the problem of chattering phenomena caused by the sign action in SMC design, and to ensure the stability of the controlled system without the requirement of auxiliary compensated controllers despite the existence of uncertainties. The outputs of the AFNNC scheme can be directly supplied to the electromagnets and LIM without complicated control transformations for relaxing strict constrains in conventional model-based control methodologies. The effectiveness of the proposed control schemes for the maglev transportation system is verified by numerical simulations, and the superiority of the AFNNC scheme is indicated in comparison with the SMC and ASMC strategies.

This paper presents evolvable block-based neural networks (BbNNs) for personalized ECG heartbeat pattern classification. A BbNN consists of a 2-D array of modular component NNs with flexible structures and internal configurations that can be implemented using reconfigurable digital hardware such as field-programmable gate arrays (FPGAs). Signal flow between the blocks determines the internal configuration of a block as well as the overall structure of the BbNN. Network structure and the weights are optimized using local gradient-based search and evolutionary operators with the rates changing adaptively according to their effectiveness in the previous evolution period. Such adaptive operator rate update scheme ensures higher fitness on average compared to predetermined fixed operator rates. The Hermite transform coefficients and the time interval between two neighboring R-peaks of ECG signals are used as inputs to the BbNN. A BbNN optimized with the proposed evolutionary algorithm (EA) makes a personalized heartbeat pattern classifier that copes with changing operating environments caused by individual difference and time-varying characteristics of ECG signals. Simulation results using the Massachusetts Institute of Technology/Beth Israel Hospital (MIT-BIH) arrhythmia database demonstrate high average detection accuracies of ventricular ectopic beats (98.1%) and supraventricular ectopic beats (96.6%) patterns for heartbeat monitoring, being a significant improvement over previously reported electrocardiogram (ECG) classification results.

This paper presents incremental hierarchical discriminant regression (IHDR) which incrementally builds a decision tree or regression tree for very high-dimensional regression or decision spaces by an online, real-time learning system. Biologically motivated, it is an approximate computational model for automatic development of associative cortex, with both bottom-up sensory inputs and top-down motor projections. At each internal node of the IHDR tree, information in the output space is used to automatically derive the local subspace spanned by the most discriminating features. Embedded in the tree is a hierarchical probability distribution model used to prune very unlikely cases during the search. The number of parameters in the coarse-to-fine approximation is dynamic and data-driven, enabling the IHDR tree to automatically fit data with unknown distribution shapes (thus, it is difficult to select the number of parameters up front). The IHDR tree dynamically assigns long-term memory to avoid the loss-of-memory problem typical with a global-fitting learning algorithm for neural networks. A major challenge for an incrementally built tree is that the number of samples varies arbitrarily during the construction process. An incrementally updated probability model, called sample-size-dependent negative-log-likelihood (SDNLL) metric is used to deal with large sample-size cases, small sample-size cases, and unbalanced sample-size cases, measured among different internal nodes of the IHDR tree. We report experimental results for four types of data: synthetic data to visualize the behavior of the algorithms, large face image data, continuous video stream from robot navigation, and publicly available data sets that use human defined features.

In this paper, we use a two-stage sparse factorization approach for blindly estimating the channel parameters and then estimating source components for electroencephalogram (EEG) signals. EEG signals are assumed to be linear mixtures of source components, artifacts, etc. Therefore, a raw EEG data matrix can be factored into the product of two matrices, one of which represents the mixing matrix and the other the source component matrix. Furthermore, the components are sparse in the time-frequency domain, i.e., the factorization is a sparse factorization in the time frequency domain. It is a challenging task to estimate the mixing matrix. Our extensive analysis and computational results, which were based on many sets of EEG data, not only provide firm evidences supporting the above assumption, but also prompt us to propose a new algorithm for estimating the mixing matrix. After the mixing matrix is estimated, the source components are estimated in the time frequency domain using a linear programming method. In an example of the potential applications of our approach, we analyzed the EEG data that was obtained from a modified Sternberg memory experiment. Two almost uncorrelated components obtained by applying the sparse factorization method were selected for phase synchronization analysis. Several interesting findings were obtained, especially that memory-related synchronization and desynchronization appear in the alpha band, and that the strength of alpha band synchronization is related to memory performance.

Recursive least square (RLS) is an efficient approach to neural network training. However, in the classical RLS algorithm, there is no explicit decay in the energy function. This will lead to an unsatisfactory generalization ability for the trained networks. In this paper, we propose a generalized RLS (GRLS) model which includes a general decay term in the energy function for the training of feedforward neural networks. In particular, four different weight decay functions, namely, the quadratic weight decay, the constant weight decay and the newly proposed multimodal and quartic weight decay are discussed. By using the GRLS approach, not only the generalization ability of the trained networks is significantly improved but more unnecessary weights are pruned to obtain a compact network. Furthermore, the computational complexity of the GRLS remains the same as that of the standard RLS algorithm. The advantages and tradeoffs of using different decay functions are analyzed and then demonstrated with examples. Simulation results show that our approach is able to meet the design goals: improving the generalization ability of the trained network while getting a compact network.

Selective visual attention in a natural environment can be seen as the interaction between the external visual stimulus and task specific knowledge of the required behavior. This interaction between the bottom-up stimulus and the top-down, task-related knowledge is crucial for what is selected in the space and time within the scene. In this paper, we propose a computational model for selective attention for a visual search task. We go beyond simple saliency-based attention models to model selective attention guided by top-down visual cues, which are dynamically integrated with the bottom-up information. In this way, selection of a location is accomplished by interaction between bottom-up and top-down information. First, the general structure of our model is briefly introduced and followed by a description of the top-down processing of task-relevant cues. This is then followed by a description of the processing of the external images to give three feature maps that are combined to give an overall bottom-up map. Second, the development of the formalism for our novel interactive spiking neural network (ISNN) is given, with the interactive activation rule that calculates the integration map. The learning rule for both bottom-up and top-down weight parameters are given, together with some further analysis of the properties of the resulting ISNN. Third, the model is applied to a face detection task to search for the location of a specific face that is cued. The results show that the trajectories of attention are dramatically changed by interaction of information and variations of cues, giving an appropriate, task-relevant search pattern. Finally, we discuss ways in which these results can be seen as compatible with existing psychological evidence.

Motivated by the human autonomous development process from infancy to adulthood, we have built a robot that develops its cognitive and behavioral skills through real-time interactions with the environment. We call such a robot a developmental robot. In this paper, we present the theory and the architecture to implement a developmental robot and discuss the related techniques that address an array of challenging technical issues. As an application, experimental results on a real robot, self-organizing, autonomous, incremental learner (SAIL), are presented with emphasis on its audition perception and audition-related action generation. In particular, the SAIL robot conducts the auditory learning from unsegmented and unlabeled speech streams without any prior knowledge about the auditory signals, such as the designated language or the phoneme models. Neither available before learning starts are the actions that the robot is expected to perform. SAIL learns the auditory commands and the desired actions from physical contacts with the environment including the trainers.