Search

In multiple instance learning problems, patterns are often given as bags and each bag consists of some instances. Most of existing research in the area focuses on multiple instance classification and multiple instance regression, while very limited work has been conducted for multiple instance clustering (MIC). This paper formulates a novel framework, maximum margin multiple instance clustering (M(3)IC), for MIC. However, it is impractical to directly solve the optimization problem of M(3)IC. Therefore, M(3)IC is relaxed in this paper to enable an efficient optimization solution with a combination of the constrained concave-convex procedure and the cutting plane method. Furthermore, this paper presents some important properties of the proposed method and discusses the relationship between the proposed method and some other related ones. An extensive set of empirical results are shown to demonstrate the advantages of the proposed method against existing research for both effectiveness and efficiency.

This paper introduces a general theory of extended Hamiltonian (second-order) learning on Riemannian manifolds, as an instance of learning by constrained criterion optimization. The dynamical learning equations are derived within the general framework of extended-Hamiltonian stationary-action principle and are expressed in a coordinate-free fashion. A theoretical analysis is carried out in order to compare the features of the dynamical learning theory with the features exhibited by the gradient-based ones. In particular, gradient-based learning is shown to be an instance of dynamical learning, and the classical gradient-based learning modified by a "momentum" term is shown to resemble discrete-time dynamical learning. Moreover, the convergence features of gradient-based and dynamical learning are compared on a theoretical basis. This paper discusses cases of learning by dynamical systems on manifolds of interest in the scientific literature, namely, the Stiefel manifold, the special orthogonal group, the Grassmann manifold, the group of symmetric positive definite matrices, the generalized flag manifold, and the real symplectic group of matrices.

In safety critical applications, control architectures based on adaptive neural networks (NNs) must satisfy strict design specifications. This paper presents a practical approach for designing a mixed linear/adaptive model reference controller that recovers the performance of a reference model, and guarantees the boundedness of the tracking error within an a priori specified compact domain, in the presence of bounded uncertainties. The linear part of the controller results from the solution of an optimization problem where specifications are expressed as linear matrix inequality constraints. The linear controller is then augmented with a general adaptive NN that compensates for the uncertainties. The only requirement for the NN is that its output must be confined within pre-specified saturation limits. Toward this end a specific NN output confinement algorithm is proposed in this paper. The main advantages of the proposed approach are that requirements in terms of worst-case performance can be easily defined during the design phase, and that the design of the adaptation mechanism is largely independent from the synthesis of the linear controller. A numerical example is used to illustrate the design methodology.

Most partitioning algorithms iteratively partition a space into cells that contain underlying linear or nonlinear structures using linear partitioning strategies. The compactness of each cell depends on how well the (locally) linear partitioning strategy approximates the intrinsic structure. To partition a compact structure for complex data in a nonlinear context, this paper proposes a nonlinear partition strategy. This is a principal curve tree (PC-tree), which is implemented iteratively. Given that a PC passes through the middle of the data distribution, it allows for partitioning based on the arc length of the PC. To enhance the partitioning of a given space, a residual version of the PC-tree algorithm is developed, denoted here as the principal component analysis tree (PCR-tree) algorithm. Because of its residual property, the PCR-tree can yield the intrinsic dimension of high-dimensional data. Comparisons presented in this paper confirm that the proposed PC-tree and PCR-tree approaches show a better performance than several other competing partitioning algorithms in terms of vector quantization error and nearest neighbor search. The comparison also shows that the proposed algorithms outperform competing linear methods in total average coverage which measures the nonlinear compactness of partitioning algorithms.

This paper introduces the Clifford support vector machines (CSVM) as a generalization of the real and complex-valued support vector machines using the Clifford geometric algebra. In this framework, we handle the design of kernels involving the Clifford or geometric product. In this approach, one redefines the optimization variables as multivectors. This allows us to have a multivector as output. Therefore, we can represent multiple classes according to the dimension of the geometric algebra in which we work. We show that one can apply CSVM for classification and regression and also to build a recurrent CSVM. The CSVM is an attractive approach for the multiple input multiple output processing of high-dimensional geometric entities. We carried out comparisons between CSVM and the current approaches to solve multiclass classification and regression. We also study the performance of the recurrent CSVM with experiments involving time series. The authors believe that this paper can be of great use for researchers and practitioners interested in multiclass hypercomplex computing, particularly for applications in complex and quaternion signal and image processing, satellite control, neurocomputation, pattern recognition, computer vision, augmented virtual reality, robotics, and humanoids.

Soft computing (SC) emerged as an integrating framework for a number of techniques that could complement one another quite well (artificial neural networks, fuzzy systems, evolutionary algorithms, probabilistic reasoning). Since its inception, a distinctive goal has been to dig out the deep relationships among their components. This paper considers two wide families of SC models. On the one hand, the regime-switching autoregressive paradigm is a recent development in statistical time series modeling, and it includes a set of models closely related to artificial neural networks. On the other hand, we consider fuzzy rule-based systems in the framework of time series analysis. This paper discloses original results establishing functional equivalences between models of these two classes, and hence opens the door to a productive line of research where results and techniques from one area can be applied in the other. As a consequence of the equivalences presented in this paper, we prove the asymptotic stationarity of a class of fuzzy rule-based systems. Simulations based on information criteria show the importance of the selection of the proper membership function.

This paper is concerned with the problem of exponential stability analysis of continuous-time switched delayed neural networks. By using the average dwell time approach together with the piecewise Lyapunov function technique and by combining a novel Lyapunov-Krasovskii functional, which benefits from the delay partitioning method, with the free-weighting matrix technique, sufficient conditions are proposed to guarantee the exponential stability for the switched neural networks with constant and time-varying delays, respectively. Moreover, the decay estimates are explicitly given. The results reported in this paper not only depend upon the delay but also depend upon the partitioning, which aims at reducing the conservatism. Numerical examples are presented to demonstrate the usefulness of the derived theoretical results.

Support vector data description (SVDD) has become a very attractive kernel method due to its good results in many novelty detection problems. However, the decision function of SVDD is expressed in terms of the kernel expansion, which results in a run-time complexity linear in the number of support vectors. For applications where fast real-time response is needed, how to speed up the decision function is crucial. This paper aims at dealing with the issue of reducing the testing time complexity of SVDD. A method called fast SVDD (F-SVDD) is proposed. Unlike the traditional methods which all try to compress a kernel expansion into one with fewer terms, the proposed F-SVDD directly finds the preimage of a feature vector, and then uses a simple relationship between this feature vector and the SVDD sphere center to re-express the center with a single vector. The decision function of F-SVDD contains only one kernel term, and thus the decision boundary of F-SVDD is only spherical in the original space. Hence, the run-time complexity of the F-SVDD decision function is no longer linear in the support vectors, but is a constant, no matter how large the training set size is. In this paper, we also propose a novel direct preimage-finding method, which is noniterative and involves no free parameters. The unique preimage can be obtained in real time by the proposed direct method without taking trial-and-error. For demonstration, several real-world data sets and a large-scale data set, the extended MIT face data set, are used in experiments. In addition, a practical industry example regarding liquid crystal display micro-defect inspection is also used to compare the applicability of SVDD and our proposed F-SVDD when faced with mass data input. The results are very encouraging.

In sparse representation, two important sparse solutions, the 0-norm and 1-norm solutions, have been receiving much of attention. The 0-norm solution is the sparsest, however it is not easy to obtain. Although the 1-norm solution may not be the sparsest, it can be easily obtained by the linear programming method. In many cases, the 0-norm solution can be obtained through finding the 1-norm solution. Many discussions exist on the equivalence of the two sparse solutions. This paper analyzes two conditions for the equivalence of the two sparse solutions. The first condition is necessary and sufficient, however, difficult to verify. Although the second is necessary but is not sufficient, it is easy to verify. In this paper, we analyze the second condition within the stochastic framework and propose a variant. We then prove that the equivalence of the two sparse solutions holds with high probability under the variant of the second condition. Furthermore, in the limit case where the 0-norm solution is extremely sparse, the second condition is also a sufficient condition with probability 1.

A crucial issue in designing learning machines is to select the correct model parameters. When the number of available samples is small, theoretical sample-based generalization bounds can prove effective, provided that they are tight and track the validation error correctly. The maximal discrepancy (MD) approach is a very promising technique for model selection for support vector machines (SVM), and estimates a classifier's generalization performance by multiple training cycles on random labeled data. This paper presents a general method to compute the generalization bounds for SVMs, which is based on referring the SVM parameters to an unsupervised solution, and shows that such an approach yields tight bounds and attains effective model selection. When one estimates the generalization error, one uses an unsupervised reference to constrain the complexity of the learning machine, thereby possibly decreasing sharply the number of admissible hypothesis. Although the methodology has a general value, the method described in the paper adopts vector quantization (VQ) as a representation paradigm, and introduces a biased regularization approach in bound computation and learning. Experimental results validate the proposed method on complex real-world data sets.

In the literature, it was reported that the chaotic artificial neural network model with sinusoidal activation functions possesses a large memory capacity as well as a remarkable ability of retrieving the stored patterns, better than the conventional chaotic model with only monotonic activation functions such as sigmoidal functions. This paper, from the viewpoint of the anti-integrable limit, elucidates the mechanism inducing the superiority of the model with periodic activation functions that includes sinusoidal functions. Particularly, by virtue of the anti-integrable limit technique, this paper shows that any finite-dimensional neural network model with periodic activation functions and properly selected parameters has much more abundant chaotic dynamics that truly determine the model's memory capacity and pattern-retrieval ability. To some extent, this paper mathematically and numerically demonstrates that an appropriate choice of the activation functions and control scheme can lead to a large memory capacity and better pattern-retrieval ability of the artificial neural network models.

In this paper, a sparse learning algorithm, probabilistic classification vector machines (PCVMs), is proposed. We analyze relevance vector machines (RVMs) for classification problems and observe that adopting the same prior for different classes may lead to unstable solutions. In order to tackle this problem, a signed and truncated Gaussian prior is adopted over every weight in PCVMs, where the sign of prior is determined by the class label, i.e., +1 or -1. The truncated Gaussian prior not only restricts the sign of weights but also leads to a sparse estimation of weight vectors, and thus controls the complexity of the model. In PCVMs, the kernel parameters can be optimized simultaneously within the training algorithm. The performance of PCVMs is extensively evaluated on four synthetic data sets and 13 benchmark data sets using three performance metrics, error rate (ERR), area under the curve of receiver operating characteristic (AUC), and root mean squared error (RMSE). We compare PCVMs with soft-margin support vector machines (SVM(Soft)), hard-margin support vector machines (SVM(Hard)), SVM with the kernel parameters optimized by PCVMs (SVM(PCVM)), relevance vector machines (RVMs), and some other baseline classifiers. Through five replications of twofold cross-validation F test, i.e., 5 x 2 cross-validation F test, over single data sets and Friedman test with the corresponding post-hoc test to compare these algorithms over multiple data sets, we notice that PCVMs outperform other algorithms, including SVM(Soft), SVM(Hard), RVM, and SVM(PCVM), on most of the data sets under the three metrics, especially under AUC. Our results also reveal that the performance of SVM(PCVM) is slightly better than SVM(Soft), implying that the parameter optimization algorithm in PCVMs is better than cross validation in terms of performance and computational complexity. In this paper, we also discuss the superiority of PCVMs' formulation using maximum a posteriori (MAP) analysis and margin analysis, which explain the empirical success of PCVMs.

This paper presents a subgradient-based neural network to solve a nonsmooth nonconvex optimization problem with a nonsmooth nonconvex objective function, a class of affine equality constraints, and a class of nonsmooth convex inequality constraints. The proposed neural network is modeled with a differential inclusion. Under a suitable assumption on the constraint set and a proper assumption on the objective function, it is proved that for a sufficiently large penalty parameter, there exists a unique global solution to the neural network and the trajectory of the network can reach the feasible region in finite time and stay there thereafter. It is proved that the trajectory of the neural network converges to the set which consists of the equilibrium points of the neural network, and coincides with the set which consists of the critical points of the objective function in the feasible region. A condition is given to ensure the convergence to the equilibrium point set in finite time. Moreover, under suitable assumptions, the coincidence between the solution to the differential inclusion and the ldquoslow solutionrdquo of it is also proved. Furthermore, three typical examples are given to present the effectiveness of the theoretic results obtained in this paper and the good performance of the proposed neural network.

The self-organizing network (SON)-based multiple model system is a recently proposed method for identifying the dynamics of a general nonlinear system. It has been observed by researchers that cooperative learning among neighboring regions is sometimes important for the success of identification of a nonlinear system under the multiple model system framework. In this paper, we intend to formally evaluate the effects of cooperative learning and topology preservation in identification of multiple model system based on SON. The results of the mathematical analysis supports the heuristic that a good learning strategy for identifying the local model parameters of a SON-based multiple model system is to choose a neighborhood function whose effective region is initially wider and is reduced gradually during learning. An example of nonlinear function approximation is also provided at the end of this paper to demonstrate the results of the mathematical analysis.

In this paper, regularization path algorithms were proposed as a novel approach to the model selection problem by exploring the path of possibly all solutions with respect to some regularization hyperparameter in an efficient way. This approach was later extended to a support vector regression (SVR) model called epsilon-SVR. However, the method requires that the error parameter epsilon be set a priori. This is only possible if the desired accuracy of the approximation can be specified in advance. In this paper, we analyze the solution space for epsilon-SVR and propose a new solution path algorithm, called epsilon-path algorithm, which traces the solution path with respect to the hyperparameter epsilon rather than lambda. Although both two solution path algorithms possess the desirable piecewise linearity property, our epsilon-path algorithm overcomes some limitations of the original lambda-path algorithm and has more advantages. It is thus more appealing for practical use.

As is well known in statistics, the resulting linear regressors by using the rank-based Wilcoxon approach to linear regression problems are usually robust against (or insensitive to) outliers. This motivates us to introduce in this paper the Wilcoxon approach to the area of machine learning. Specifically, we investigate four new learning machines, namely Wilcoxon neural network (WNN), Wilcoxon generalized radial basis function network (WGRBFN), Wilcoxon fuzzy neural network (WFNN), and kernel-based Wilcoxon regressor (KWR). These provide alternative learning machines when faced with general nonlinear learning problems. Simple weights updating rules based on gradient descent will be derived. Some numerical examples will be provided to compare the robustness against outliers for various learning machines. Simulation results show that the Wilcoxon learning machines proposed in this paper have good robustness against outliers. We firmly believe that the Wilcoxon approach will provide a promising methodology for many machine learning problems.

This paper deals with adaptive tracking for discrete-time multiple-input-multiple-output (MIMO) nonlinear systems in presence of bounded disturbances. In this paper, a high-order neural network (HONN) structure is used to approximate a control law designed by the backstepping technique, applied to a block strict feedback form (BSFF). This paper also includes the respective stability analysis, on the basis of the Lyapunov approach, for the whole controlled system, including the extended Kalman filter (EKF)-based NN learning algorithm. Applicability of the scheme is illustrated via simulation for a discrete-time nonlinear model of an electric induction motor.

The existing approaches to the discrete-time nonlinear output regulation problem rely on the offline solution of a set of mixed nonlinear functional equations known as discrete regulator equations. For complex nonlinear systems, it is difficult to solve the discrete regulator equations even approximately. Moreover, for systems with uncertainty, these approaches cannot offer a reliable solution. By combining the approximation capability of the feedforward neural networks (NNs) with an online parameter optimization mechanism, we develop an approach to solving the discrete nonlinear output regulation problem without solving the discrete regulator equations explicitly. The approach of this paper can be viewed as a discrete counterpart of our previous paper on approximately solving the continuous-time nonlinear output regulation problem.

This paper is concerned with an enhanced independent component analysis (ICA) and its application to face recognition. Typically, face representations obtained by ICA involve unsupervised learning and high-order statistics. In this paper, we develop an enhancement of the generic ICA by augmenting this method by the Fisher linear discriminant analysis (LDA); hence, its abbreviation, FICA. The FICA is systematically developed and presented along with its underlying architecture. A comparative analysis explores four distance metrics, as well as classification with support vector machines (SVMs). We demonstrate that the FICA approach leads to the formation of well-separated classes in low-dimension subspace and is endowed with a great deal of insensitivity to large variation in illumination and facial expression. The comprehensive experiments are completed for the facial-recognition technology (FERET) face database; a comparative analysis demonstrates that FICA comes with improved classification rates when compared with some other conventional approaches such as eigenface, fisherface, and the ICA itself.

In this paper, we propose a new neural architecture for classification of visual patterns that is motivated by the two concepts of image pyramids and local receptive fields. The new architecture, called pyramidal neural network (PyraNet), has a hierarchical structure with two types of processing layers: Pyramidal layers and one-dimensional (1-D) layers. In the new network, nonlinear two-dimensional (2-D) neurons are trained to perform both image feature extraction and dimensionality reduction. We present and analyze five training methods for PyraNet [gradient descent (GD), gradient descent with momentum, resilient back-propagation (RPROP), Polak-Ribiere conjugate gradient (CG), and Levenberg-Marquadrt (LM)] and two choices of error functions [mean-square-error (mse) and cross-entropy (CE)]. In this paper, we apply PyraNet to determine gender from a facial image, and compare its performance on the standard facial recognition technology (FERET) database with three classifiers: The convolutional neural network (NN), the k-nearest neighbor (k-NN), and the support vector machine (SVM).

The inverse controller is traditionally assumed to be a deterministic function. This paper presents a pedagogical methodology for estimating the stochastic model of the inverse controller. The proposed method is based on Bayes' theorem. Using Bayes' rule to obtain the stochastic model of the inverse controller allows the use of knowledge of uncertainty from both the inverse and the forward model in estimating the optimal control signal. The paper presents the methodology for general nonlinear systems and is demonstrated on nonlinear single-input-single-output (SISO) and multiple-input-multiple-output (MIMO) examples.

In this paper, an adaptive wavelet-network-based control approach is proposed for highly nonlinear uncertain dynamical systems. Wavelet network, as a kind of universal approximator, has two novel properties--orthonormality and multiresolution. The orthonormal property ensures that adding a new resolution (new wavelets) does not affect the existing wavelet network that may have been well tuned. In the sequel, the online adjustment of the structure of the nonlinear adaptive wavelet controller (AWC) can be done in a constructive manner by gradually increasing the network resolution. The multiresolution property, on the other hand, assures a guaranteed improvement of the approximation precision when a new resolution is added. In real life problems we are unable to know the adequate size of a network, either a neural network (NN) or a wavelet network, to produce the required approximation precision. By virtue of the novel wavelet network properties, a coarse or very simple structure can be selected first. If the system fails to converge after the elapse of a dwell time, a new wavelet resolution is considered to be necessary and added directly. In this manner, the AWC can be easily constructed and tuned from the coarse to finer levels until the performance requirement is satisfied. The trial and error way of selecting the network structure, which may lead to either an inadequate or a highly redundant structure, can be avoided. In this paper, the proposed adaptive wavelet network is applied first to a class of nonlinear dynamical systems with a partially known model and an affine-in-input structure. Then, the adaptive wavelet network is applied to a class of nonlinear nonaffine dynamical systems.

In dealing with large data sets, the reduced support vector machine (RSVM) was proposed for the practical objective to overcome some computational difficulties as well as to reduce the model complexity. In this paper, we study the RSVM from the viewpoint of sampling design, its robustness, and the spectral analysis of the reduced kernel. We consider the nonlinear separating surface as a mixture of kernels. Instead of a full model, the RSVM uses a reduced mixture with kernels sampled from certain candidate set. Our main results center on two major themes. One is the robustness of the random subset mixture model. The other is the spectral analysis of the reduced kernel. The robustness is judged by a few criteria as follows: 1) model variation measure; 2) model bias (deviation) between the reduced model and the full model; and 3) test power in distinguishing the reduced model from the full one. For the spectral analysis, we compare the eigenstructures of the full kernel matrix and the approximation kernel matrix. The approximation kernels are generated by uniform random subsets. The small discrepancies between them indicate that the approximation kernels can retain most of the relevant information for learning tasks in the full kernel. We focus on some statistical theory of the reduced set method mainly in the context of the RSVM. The use of a uniform random subset is not limited to the RSVM. This approach can act as a supplemental algorithm on top of a basic optimization algorithm, wherein the actual optimization takes place on the subset-approximated data. The statistical properties discussed in this paper are still valid.

In some pattern recognition tasks, the dimension of the sample space is larger than the number of samples in the training set. This is known as the "small sample size problem". Linear discriminant analysis (LDA) techniques cannot be applied directly to the small sample size case. The small sample size problem is also encountered when kernel approaches are used for recognition. In this paper, we attempt to answer the question of "How should one choose the optimal projection vectors for feature extraction in the small sample size case?" Based on our findings, we propose a new method called the kernel discriminative common vector method. In this method, we first nonlinearly map the original input space to an implicit higher dimensional feature space, in which the data are hoped to be linearly separable. Then, the optimal projection vectors are computed in this transformed space. The proposed method yields an optimal solution for maximizing a modified Fisher's linear discriminant criterion, discussed in the paper. Thus, under certain conditions, a 100% recognition rate is guaranteed for the training set samples. Experiments on test data also show that, in many situations, the generalization performance of the proposed method compares favorably with other kernel approaches.

A new adaptive backpropagation (BP) algorithm based on Lyapunov stability theory for neural networks is developed in this paper. It is shown that the candidate of a Lyapunov function V(k) of the tracking error between the output of a neural network and the desired reference signal is chosen first, and the weights of the neural network are then updated, from the output layer to the input layer, in the sense that deltaV(k) = V(k) - V(k - 1)0. The output tracking error can then asymptotically converge to zero according to Lyapunov stability theory. Unlike gradient-based BP training algorithms, the new Lyapunov adaptive BP algorithm in this paper is not used for searching the global minimum point along the cost-function surface in the weight space, but it is aimed at constructing an energy surface with a single global minimum point through the adaptive adjustment of the weights as the time goes to infinity. Although a neural network may have bounded input disturbances, the effects of the disturbances can be eliminated, and asymptotic error convergence can be obtained. The new Lyapunov adaptive BP algorithm is then applied to the design of an adaptive filter in the simulation example to show the fast error convergence and strong robustness with respect to large bounded input disturbances.

This paper investigates new learning algorithms (LF I and LF II) based on Lyapunov function for the training of feedforward neural networks. It is observed that such algorithms have interesting parallel with the popular backpropagation (BP) algorithm where the fixed learning rate is replaced by an adaptive learning rate computed using convergence theorem based on Lyapunov stability theory. LF II, a modified version of LF I, has been introduced with an aim to avoid local minima. This modification also helps in improving the convergence speed in some cases. Conditions for achieving global minimum for these kind of algorithms have been studied in detail. The performances of the proposed algorithms are compared with BP algorithm and extended Kalman filtering (EKF) on three bench-mark function approximation problems: XOR, 3-bit parity, and 8-3 encoder. The comparisons are made in terms of number of learning iterations and computational time required for convergence. It is found that the proposed algorithms (LF I and II) are much faster in convergence than other two algorithms to attain same accuracy. Finally, the comparison is made on a complex two-dimensional (2-D) Gabor function and effect of adaptive learning rate for faster convergence is verified. In a nutshell, the investigations made in this paper help us better understand the learning procedure of feedforward neural networks in terms of adaptive learning rate, convergence speed, and local minima.

In pattern recognition, feature extraction techniques are widely employed to reduce the dimensionality of data and to enhance the discriminatory information. Principal component analysis (PCA) and linear discriminant analysis (LDA) are the two most popular linear dimensionality reduction methods. However, PCA is not very effective for the extraction of the most discriminant features, and LDA is not stable due to the small sample size problem . In this paper, we propose some new (linear and nonlinear) feature extractors based on maximum margin criterion (MMC). Geometrically, feature extractors based on MMC maximize the (average) margin between classes after dimensionality reduction. It is shown that MMC can represent class separability better than PCA. As a connection to LDA, we may also derive LDA from MMC by incorporating some constraints. By using some other constraints, we establish a new linear feature extractor that does not suffer from the small sample size problem, which is known to cause serious stability problems for LDA. The kernelized (nonlinear) counterpart of this linear feature extractor is also established in the paper. Our extensive experiments demonstrate that the new feature extractors are effective, stable, and efficient.

In this paper, we obtain new sufficient conditions ensuring existence, uniqueness, and global asymptotic stability (GAS) of the equilibrium point for a general class of delayed neural networks (DNNs) via nonsmooth analysis, which makes full use of the Lipschitz property of functions defining DNNs. Based on this new tool of nonsmooth analysis, we first obtain a couple of general results concerning the existence and uniqueness of the equilibrium point. Then those results are applied to show that existence assumptions on the equilibrium point in some existing sufficient conditions ensuring GAS are actually unnecessary; and some strong assumptions such as the boundedness of activation functions in some other existing sufficient conditions can be actually dropped. Finally, we derive some new sufficient conditions which are easy to check. Comparison with some related existing results is conducted and advantages are illustrated with examples. Throughout our paper, spectral properties of the matrix (A + A/sup /spl tau//) play an important role, which is a distinguished feature from previous studies. Here, A and A/sup /spl tau// are, respectively, the feedback and the delayed feedback matrix defining the neural network under consideration.

The problem of optimal asymmetric Hopfield-type associative memory (HAM) design based on perceptron-type learning algorithms is considered. It is found that most of the existing methods considered the design problem as either 1) finding optimal hyperplanes according to normal distance from the prototype vectors to the hyperplane surface or 2) obtaining weight matrix W=[w/sub ij/] by solving a constraint optimization problem. In this paper, we show that since the state space of the HAM consists of only bipolar patterns, i.e., V=(v/sub 1/,v/sub 2/,...,v/sub N/)/sup T//spl isin/{-1,+1}/sup N/, the basins of attraction around each prototype (training) vector should be expanded by using Hamming distance measure. For this reason, in this paper, the design problem is considered from a different point of view. Our idea is to systematically increase the size of the training set according to the desired basin of attraction around each prototype vector. We name this concept the higher order Hamming stability and show that conventional minimum-overlap algorithm can be modified to incorporate this concept. Experimental results show that the recall capability as well as the number of spurious memories are all improved by using the proposed method. Moreover, it is well known that setting all self-connections w/sub ii//spl forall/i to zero has the effect of reducing the number of spurious memories in state space. From the experimental results, we find that the basin width around each prototype vector can be enlarged by allowing nonzero diagonal elements on learning of the weight matrix W. If the magnitude of w/sub ii/ is small for all i, then the condition w/sub ii/=0/spl forall/i can be relaxed without seriously affecting the number of spurious memories in the state space. Therefore, the method proposed in this paper can be used to increase the basin width around each prototype vector with the cost of slightly increasing the number of spurious memories in the state space.

A novel feature selection method using the concept of mutual information (MI) is proposed in this paper. In all MI based feature selection methods, effective and efficient estimation of high-dimensional MI is crucial. In this paper, a pruned Parzen window estimator and the quadratic mutual information (QMI) are combined to address this problem. The results show that the proposed approach can estimate the MI in an effective and efficient way. With this contribution, a novel feature selection method is developed to identify the salient features one by one. Also, the appropriate feature subsets for classification can be reliably estimated. The proposed methodology is thoroughly tested in four different classification applications in which the number of features ranged from less than 10 to over 15,000. The presented results are very promising and corroborate the contribution of the proposed feature selection methodology.

The decision-making process of many binary classification systems is based on the likelihood ratio (LR) scores of test patterns. This paper shows that LR scores can be expressed in terms of the similarity between the supervectors (SVs) formed by stacking the mean vectors of Gaussian mixture models corresponding to the test patterns, the target model, and the background model. By interpreting the support vector machine (SVM) kernels as a specific similarity (or discriminant) function between SVs, this paper shows that LR scoring is a special case of SVM scoring and that most sequence kernels can be obtained by assuming a specific form for the similarity function of SVs. This paper further shows that this assumption can be relaxed to derive a new general kernel. The kernel function is general in that it is a linear combination of any kernels belonging to the reproducing kernel Hilbert space. The combination weights are obtained by optimizing the ability of a discriminant function to separate the positive and negative classes using either regression analysis or SVM training. The idea was applied to both high-and low-level speaker verification. In both cases, results show that the proposed kernels achieve better performance than several state-of-the-art sequence kernels. Further performance enhancement was also observed when the high-level scores were combined with acoustic scores.

In this paper, we consider a new periodic activation function for the multivalued neuron (MVN). The MVN is a neuron with complex-valued weights and inputs/output, which are located on the unit circle. Although the MVN outperforms many other neurons and MVN-based neural networks have shown their high potential, the MVN still has a limited capability of learning highly nonlinear functions. A periodic activation function, which is introduced in this paper, makes it possible to learn nonlinearly separable problems and non-threshold multiple-valued functions using a single multivalued neuron. We call this neuron a multivalued neuron with a periodic activation function (MVN-P). The MVN-Ps functionality is much higher than that of the regular MVN. The MVN-P is more efficient in solving various classification problems. A learning algorithm based on the error-correction rule for the MVN-P is also presented. It is shown that a single MVN-P can easily learn and solve those benchmark classification problems that were considered unsolvable using a single neuron. It is also shown that a universal binary neuron, which can learn nonlinearly separable Boolean functions, and a regular MVN are particular cases of the MVN-P.

This paper is concerned with the problem of exponential stability for a class of Markovian jump impulsive stochastic Cohen-Grossberg neural networks with mixed time delays and known or unknown parameters. The jumping parameters are determined by a continuous-time, discrete-state Markov chain, and the mixed time delays under consideration comprise both time-varying delays and continuously distributed delays. To the best of the authors' knowledge, till now, the exponential stability problem for this class of generalized neural networks has not yet been solved since continuously distributed delays are considered in this paper. The main objective of this paper is to fill this gap. By constructing a novel Lyapunov-Krasovskii functional, and using some new approaches and techniques, several novel sufficient conditions are obtained to ensure the exponential stability of the trivial solution in the mean square. The results presented in this paper generalize and improve many known results. Finally, two numerical examples and their simulations are given to show the effectiveness of the theoretical results.

Recurrent neural networks for solving constrained least absolute deviation (LAD) problems or L(1)-norm optimization problems have attracted much interest in recent years. But so far most neural networks can only deal with some special linear constraints efficiently. In this paper, two neural networks are proposed for solving LAD problems with various linear constraints including equality, two-sided inequality and bound constraints. When tailored to solve some special cases of LAD problems in which not all types of constraints are present, the two networks can yield simpler architectures than most existing ones in the literature. In particular, for solving problems with both equality and one-sided inequality constraints, another network is invented. All of the networks proposed in this paper are rigorously shown to be capable of solving the corresponding problems. The different networks designed for solving the same types of problems possess the same structural complexity, which is due to the fact these architectures share the same computing blocks and only differ in connections between some blocks. By this means, some flexibility for circuits realization is provided. Numerical simulations are carried out to illustrate the theoretical results and compare the convergence rates of the networks.

This paper describes an improved algorithm for the numerical solution to the support vector machine (SVM) classification problem for all values of the regularization parameter C . The algorithm is motivated by the work of Hastie and follows the main idea of tracking the optimality conditions of the SVM solution for ascending value of C . It differs from Hastie's approach in that the tracked path is not assumed to be 1-D. Instead, a multidimensional feasible space for the optimality condition is used to solve the tracking problem. Such a treatment allows the algorithm to properly handle data sets which Hastie's approach fails. These data sets are characterized by the presence of linearly dependent points (in the kernel space), duplicate points, or nearly duplicate points. Such data sets are quite common among many real-world data, especially those with nominal features. Other contributions of this paper include a unifying formulation of the tracking process in the form of a linear programming problem, update formula for the linear programs, considerations that guard against accumulation of errors resulting from the use of incremental updates, and routines to speed up the algorithm. The algorithm is implemented under the Matlab environment and is available for download. Experiments with several data sets including data set having up to several thousand data points are reported.

This paper proposes an uncorrelated multilinear principal component analysis (UMPCA) algorithm for unsupervised subspace learning of tensorial data. It should be viewed as a multilinear extension of the classical principal component analysis (PCA) framework. Through successive variance maximization, UMPCA seeks a tensor-to-vector projection (TVP) that captures most of the variation in the original tensorial input while producing uncorrelated features. The solution consists of sequential iterative steps based on the alternating projection method. In addition to deriving the UMPCA framework, this work offers a way to systematically determine the maximum number of uncorrelated multilinear features that can be extracted by the method. UMPCA is compared against the baseline PCA solution and its five state-of-the-art multilinear extensions, namely two-dimensional PCA (2DPCA), concurrent subspaces analysis (CSA), tensor rank-one decomposition (TROD), generalized PCA (GPCA), and multilinear PCA (MPCA), on the tasks of unsupervised face and gait recognition. Experimental results included in this paper suggest that UMPCA is particularly effective in determining the low-dimensional projection space needed in such recognition tasks.

The concepts of permitted and forbidden sets enable a new perspective of the memory in neural networks. Such concepts exhibit interesting dynamics in recurrent neural networks. This paper studies the basic theories of permitted and forbidden sets of the linear threshold discrete-time recurrent neural networks. The linear threshold transfer function has been regarded as an adequate transfer function for recurrent neural networks. Networks with this transfer function form a class of hybrid analog and digital networks which are especially useful for perceptual computations. Networks in discrete time can directly provide algorithms for efficient implementation in digital hardware. The main contribution of this paper is to establish foundations of permitted and forbidden sets. Necessary and sufficient conditions for the linear threshold discrete-time recurrent neural networks are obtained for complete convergence, existence of permitted and forbidden sets, as well as conditionally multiattractivity, respectively. Simulation studies explore some possible interesting practical applications.

An unsupervised competitive neural network for efficient clustering of Gaussian probability density function (GPDF) data of continuous density hidden Markov models (CDHMMs) is proposed in this paper. The proposed unsupervised competitive neural network, called the divergence-based centroid neural network (DCNN), employs the divergence measure as its distance measure and utilizes the statistical characteristics of observation densities in the HMM for speech recognition problems. While the conventional clustering algorithms used for the vector quantization (VQ) codebook design utilize only the mean values of the observation densities in the HMM, the proposed DCNN utilizes both the mean and the covariance values. When compared with other conventional unsupervised neural networks, the DCNN successfully allocates more code vectors to the regions where GPDF data are densely distributed while it allocates fewer code vectors to the regions where GPDF data are sparsely distributed. When applied to Korean monophone recognition problems as a tool to reduce the size of the codebook, the DCNN reduced the number of GPDFs used for code vectors by 65.3% while preserving recognition accuracy. Experimental results with a divergence-based k-means algorithm and a divergence-based self-organizing map algorithm are also presented in this paper for a performance comparison.

Earlier clustering techniques such as the modified learning vector quantization (MLVQ) and the fuzzy Kohonen partitioning (FKP) techniques have focused on the derivation of a certain set of parameters so as to define the fuzzy sets in terms of an algebraic function. The fuzzy membership functions thus generated are uniform, normal, and convex. Since any irregular training data is clustered into uniform fuzzy sets (Gaussian, triangular, or trapezoidal), the clustering may not be exact and some amount of information may be lost. In this paper, two clustering techniques using a Kohonen-like self-organizing neural network architecture, namely, the unsupervised discrete clustering technique (UDCT) and the supervised discrete clustering technique (SDCT), are proposed. The UDCT and SDCT algorithms reduce this data loss by introducing nonuniform, normal fuzzy sets that are not necessarily convex. The training data range is divided into discrete points at equal intervals, and the membership value corresponding to each discrete point is generated. Hence, the fuzzy sets obtained contain pairs of values, each pair corresponding to a discrete point and its membership grade. Thus, it can be argued that fuzzy membership functions generated using this kind of a discrete methodology provide a more accurate representation of the actual input data. This fact has been demonstrated by comparing the membership functions generated by the UDCT and SDCT algorithms against those generated by the MLVQ, FKP, and pseudofuzzy Kohonen partitioning (PFKP) algorithms. In addition to these clustering techniques, a novel pattern classifying network called the Yager fuzzy neural network (FNN) is proposed in this paper. This network corresponds completely to the Yager inference rule and exhibits remarkable generalization abilities. A modified version of the pseudo-outer product (POP)-Yager FNN called the modified Yager FNN is introduced that eliminates the drawbacks of the earlier network and yi- elds superior performance. Extensive experiments have been conducted to test the effectiveness of these two networks, using various clustering algorithms. It follows that the SDCT and UDCT clustering algorithms are particularly suited to networks based on the Yager inference rule.

In this paper, we consider delayed bidirectional associative memory (BAM) neural networks (NNs) with Lipschitz continuous activation functions. By applying Young's inequality and Hoelder's inequality techniques together with the properties of monotonic continuous functions, global exponential stability criteria are established for BAM NNs with time delays. This is done through the use of a new Lyapunov functional and an M-matrix. The results obtained in this paper extend and improve previous results.

Adaptive approximation-based control typically uses approximators with a predefined set of basis functions. Recently, spatially dependent methods have defined self-organizing approximators where new locally supported basis elements were incorporated when existing basis elements were insufficiently excited. In this paper, performance-dependent self-organizing approximators will be defined. The designer specifies a positive tracking error criteria. The self-organizing approximation-based controller then monitors the tracking performance and adds basis elements only as needed to achieve the tracking specification. The method of this paper is applicable to general nth-order input-state feedback linearizable systems. This paper includes a complete stability analysis and a detailed simulation example.

This paper deals with the problem of multiuser detection in direct-sequence code-division multiple-access (DS-CDMA) systems in multipath environments. The existing multiuser detectors can be divided into two categories: (1) low-complexity poor-performance linear detectors and (2) high-complexity good-performance nonlinear detectors. In particular, in channels where the orthogonality of the code sequences is destroyed by multipath, detectors with linear complexity perform much worse than the nonlinear detectors. In this paper, we propose an enhanced multisurface method (EMSM) for multiuser detection in multipath channels. EMSM is an intermediate piecewise linear detection scheme with a run-time complexity linear in the number of users. Its bit error rate performance is compared with existing linear detectors, a nonlinear radial basis function detector trained by the new support vector learning algorithm, and Verdu's optimal detector. Simulations in multipath channels, for both synchronous and asynchronous cases, indicate that it always outperforms all other linear detectors, performing nearly as well as nonlinear detectors.

A novel radial basis function neural network for discriminant analysis is presented in this paper. In contrast to many other researches, this work focuses on the exploitation of the weight structure of radial basis function neural networks using the Bayesian method. It is expected that the performance of a radial basis function neural network with a well-explored weight structure can be improved. As the weight structure of a radial basis function neural network is commonly unknown, the Bayesian method is, therefore, used in this paper to study this a priori structure. Two weight structures are investigated in this study, i.e., a single-Gaussian structure and a two-Gaussian structure. An expectation-maximization learning algorithm is used to estimate the weights. The simulation results showed that the proposed radial basis function neural network with a weight structure of two Gaussians outperformed the other algorithms.

In this paper, we analyze Boolean functions using a recently proposed measure of their complexity. This complexity measure, motivated by the aim of relating the complexity of the functions with the generalization ability that can be obtained when the functions are implemented in feed-forward neural networks, is the sum of a number of components. We concentrate on the case in which we use the first two of these components. The first is related to the "average sensitivity" of the function and the second is, in a sense, a measure of the "randomness" or lack of structure of the function. In this paper, we investigate the importance of using the second term in the complexity measure, and we consider to what extent these two terms suffice as an indicator of how difficult it is to learn a Boolean function. We also explore the existence of very complex Boolean functions, considering, in particular, the symmetric Boolean functions.

In this paper, we study cellular neural networks with almost periodic variable coefficients and time-varying delays. By using the existence theorem of almost periodic solution for general functional differential equations, introducing many real parameters and applying the Lyapunov functional method and the technique of Young inequality, we obtain some sufficient conditions to ensure the existence, uniqueness, and global exponential stability of almost periodic solution. The results obtained in this paper are new, useful, and extend and improve the existing ones in previous literature.

The convergence of Oja's principal component analysis (PCA) learning algorithms is a difficult topic for direct study and analysis. Traditionally, the convergence of these algorithms is indirectly analyzed via certain deterministic continuous time (DCT) systems. Such a method will require the learning rate to converge to zero, which is not a reasonable requirement to impose in many practical applications. Recently, deterministic discrete time (DDT) systems have been proposed instead to indirectly interpret the dynamics of the learning algorithms. Unlike DCT systems, DDT systems allow learning rates to be constant (which can be a nonzero). This paper will provide some important results relating to the convergence of a DDT system of Oja's PCA learning algorithm. It has the following contributions: 1) A number of invariant sets are obtained, based on which we can show that any trajectory starting from a point in the invariant set will remain in the set forever. Thus, the nondivergence of the trajectories is guaranteed. 2) The convergence of the DDT system is analyzed rigorously. It is proven, in the paper, that almost all trajectories of the system starting from points in an invariant set will converge exponentially to the unit eigenvector associated with the largest eigenvalue of the correlation matrix. In addition, exponential convergence rate are obtained, providing useful guidelines for the selection of fast convergence learning rate. 3) Since the trajectories may diverge, the careful choice of initial vectors is an important issue. This paper suggests to use the domain of unit hyper sphere as initial vectors to guarantee convergence. 4) Simulation results will be furnished to illustrate the theoretical results achieved.

In this paper, an observer-based direct adaptive fuzzy-neural control scheme is presented for nonaffine nonlinear systems in the presence of unknown structure of nonlinearities. A direct adaptive fuzzy-neural controller and a class of generalized nonlinear systems, which are called nonaffine nonlinear systems, are instead of the indirect one and affine nonlinear systems given by Leu et al. By using implicit function theorem and Taylor series expansion, the observer-based control law and the weight update law of the fuzzy-neural controller are derived for the nonaffine nonlinear systems. Based on strictly-positive-real (SPR) Lyapunov theory, the stability of the closed-loop system can be verified. Moreover, the overall adaptive scheme guarantees that all signals involved are bounded and the output of the closed-loop system will asymptotically track the desired output trajectory. To demonstrate the effectiveness of the proposed method, simulation results are illustrated in this paper.

In this letter, we point out that the linear matrix inequality (LMI)-based criterion obtained in the above paper (Singh, IEEE Trans. Neural Netw., vol. 15, no. 1, p. 223-5, 2004) for the global exponential stability of the delayed neural networks can be simplified to a simpler but equivalent form and, thus, show that it is not necessary to have such complex form of condition in the above paper. As a result, we also answer the question raised by the author of the above paper.

This paper is concerned with the passivity and stability problems of reaction-diffusion neural networks (RDNNs) in which the input and output variables are varied with the time and space variables. By utilizing the Lyapunov functional method combined with the inequality techniques, some sufficient conditions ensuring the passivity and global exponential stability are derived. Furthermore, when the parameter uncertainties appear in RDNNs, several criteria for robust passivity and robust global exponential stability are also presented. Finally, a numerical example is provided to illustrate the effectiveness of the proposed criteria.

In this paper, a one-layer recurrent neural network is presented for solving pseudoconvex optimization problems subject to linear equality constraints. The global convergence of the neural network can be guaranteed even though the objective function is pseudoconvex. The finite-time state convergence to the feasible region defined by the equality constraints is also proved. In addition, global exponential convergence is proved when the objective function is strongly pseudoconvex on the feasible region. Simulation results on illustrative examples and application on chemical process data reconciliation are provided to demonstrate the effectiveness and characteristics of the neural network.