Search

Unlike the first and the second generation artificial neural networks, spiking neural networks (SNNs) model the human brain by incorporating not only synaptic state but also a temporal component into their operating model. However, their intrinsic properties require expensive computation during training. This paper presents a novel algorithm to SpikeProp for SNN by introducing smoothing L 1∕2 regularization term into the error function. This algorithm makes the network structure sparse, with some smaller weights that can be eventually removed. Meanwhile, the convergence of this algorithm is proved under some reasonable conditions. The proposed algorithms have been tested for the convergence speed, the convergence rate and the generalization on the classical XOR-problem, Iris problem and Wisconsin Breast Cancer classification., (Copyright © 2018 Elsevier Ltd. All rights reserved.)

Weight elimination offers a simple and efficient improvement of training algorithm of feedforward neural networks. It is a general regularization technique in terms of the flexible scaling parameters. Actually, the weight elimination technique also contains the weight decay regularization for a large scaling parameter. Many applications of this technique and its improvements have been reported. However, there is little research concentrated on its convergence behavior. In this paper, we theoretically analyze the weight elimination for cyclic learning method and determine the conditions for the uniform boundedness of weight sequence, and weak and strong convergence. Based on the assumed network parameters, the optimal choice for the scaling parameter can also be determined. Moreover, two illustrative simulations have been done to support the theoretical explorations as well., (Copyright © 2016 Elsevier Ltd. All rights reserved.)

The aim of this paper is to develop a novel method to prune feedforward neural networks by introducing an L1/2 regularization term into the error function. This procedure forces weights to become smaller during the training and can eventually removed after the training. The usual L1/2 regularization term involves absolute values and is not differentiable at the origin, which typically causes oscillation of the gradient of the error function during the training. A key point of this paper is to modify the usual L1/2 regularization term by smoothing it at the origin. This approach offers the following three advantages: First, it removes the oscillation of the gradient value. Secondly, it gives better pruning, namely the final weights to be removed are smaller than those produced through the usual L1/2 regularization. Thirdly, it makes it possible to prove the convergence of the training. Supporting numerical examples are also provided., (Copyright © 2013 Elsevier Ltd. All rights reserved.)

Rule extraction from neural networks (NNs) solves two fundamental problems: it gives insight into the logic behind the network and in many cases, it improves the network's ability to generalize the acquired knowledge. This paper presents a novel eclectic approach to rule extraction from NNs, named LOcal Rule Extraction (LORE), suited for multilayer perceptron networks with discrete (logical or categorical) inputs. The extracted rules mimic network behavior on the training set and relax this condition on the remaining input space. First, a multilayer perceptron network is trained under standard regime. It is then transformed into an equivalent form, returning the same numerical result as the original network, yet being able to produce rules generalizing the network output for cases similar to a given input. The partial rules extracted for every training set sample are then merged to form a decision diagram (DD) from which logic rules can be extracted. A rule format explicitly separating subsets of inputs for which an answer is known from those with an undetermined answer is presented. A special data structure, the decision diagram, allowing efficient partial rule merging is introduced. With regard to rules' complexity and generalization abilities, LORE gives results comparable to those reported previously. An algorithm transforming DDs into interpretable boolean expressions is described. Experimental running times of rule extraction are proportional to the network's training time.

Multiclass cancer classification on microarray data has provided the feasibility of cancer diagnosis across all of the common malignancies in parallel. Using multiclass cancer feature selection approaches, it is now possible to identify genes relevant to a set of cancer types. However, besides identifying the relevant genes for the set of all cancer types, it is deemed to be more informative to biologists if the relevance of each gene to specific cancer or subset of cancer types could be revealed or pinpointed. In this paper, we introduce two new definitions of multiclass relevancy features, i.e., full class relevant (FCR) and partial class relevant (PCR) features. Particularly, FCR denotes genes that serve as candidate biomarkers for discriminating all cancer types. PCR, on the other hand, are genes that distinguish subsets of cancer types. Subsequently, a Markov blanket embedded memetic algorithm is proposed for the simultaneous identification of both FCR and PCR genes. Results obtained on commonly used synthetic and real-world microarray data sets show that the proposed approach converges to valid FCR and PCR genes that would assist biologists in their research work. The identification of both FCR and PCR genes is found to generate improvement in classification accuracy on many microarray data sets. Further comparison study to existing state-of-the-art feature selection algorithms also reveals the effectiveness and efficiency of the proposed approach.

This paper presents an optimization framework for improving case-based computer-aided decision (CB-CAD) systems. The underlying hypothesis of the study is that each example in the knowledge database of a medical decision support system has different importance in the decision making process. A new decision algorithm incorporating an importance weight for each example is proposed to account for these differences. The search for the best set of importance weights is defined as an optimization problem and a genetic algorithm is employed to solve it. The optimization process is tailored to maximize the system's performance according to clinically relevant evaluation criteria. The study was performed using a CAD system developed for the classification of regions of interests (ROIs) in mammograms as depicting masses or normal tissue. The system was constructed and evaluated using a dataset of ROIs extracted from the Digital Database for Screening Mammography (DDSM). Experimental results show that, according to receiver operator characteristic (ROC) analysis, the proposed method significantly improves the overall performance of the CAD system as well as its average specificity for high breast mass detection rates.

Single-class minimax probability machines (MPMs) offer robust novelty detection with distribution-free worst case bounds on the probability that a pattern will fall inside the normal region. However, in practice, they are too cautious in labeling patterns as outlying and so have a high false negative rate (FNR). In this paper, we propose a more aggressive version of the single-class MPM that bounds the best case probability that a pattern will fall inside the normal region. These two MPMs can then be used together to delimit the solution space. By using the hyperplane lying in the middle of this pair of MPMs, a better compromise between false positives (FPs) and false negatives (FNs), and between recall and precision can be obtained. Experiments on the real-world data sets show encouraging results.

We present a technique that enhances computer-assisted decision (CAD) systems with the ability to assess the reliability of each individual decision they make. Reliability assessment is achieved by measuring the accuracy of a CAD system with known cases similar to the one in question. The proposed technique analyzes the feature space neighborhood of the query case to dynamically select an input-dependent set of known cases relevant to the query. This set is used to assess the local (query-specific) accuracy of the CAD system. The estimated local accuracy is utilized as a reliability measure of the CAD response to the query case. The underlying hypothesis of the study is that CAD decisions with higher reliability are more accurate. The above hypothesis was tested using a mammographic database of 1337 regions of interest (ROIs) with biopsy-proven ground truth (681 with masses, 656 with normal parenchyma). Three types of decision models, (i) a back-propagation neural network (BPNN), (ii) a generalized regression neural network (GRNN), and (iii) a support vector machine (SVM), were developed to detect masses based on eight morphological features automatically extracted from each ROI. The performance of all decision models was evaluated using the Receiver Operating Characteristic (ROC) analysis. The study showed that the proposed reliability measure is a strong predictor of the CAD system's case-specific accuracy. Specifically, the ROC area index for CAD predictions with high reliability was significantly better than for those with low reliability values. This result was consistent across all decision models investigated in the study. The proposed case-specific reliability analysis technique could be used to alert the CAD user when an opinion that is unlikely to be reliable is offered. The technique can be easily deployed in the clinical environment because it is applicable with a wide range of classifiers regardless of their structure and it requires neither additional training nor building multiple decision models to assess the case-specific CAD accuracy.

Estimation of the dynamic spinal forces from kinematics data is very complicated because it involves the handling of the relationship between kinematic variables and electromyography (EMG) signals, as well as the relationship between EMG signals and the forces. A recurrent fuzzy neural network (RFNN) model is proposed to establish the kinematics-EMG-force relationship and model the dynamics of muscular activities. The EMG signals are used as an intermediate output and are fed back to the input layer. Since EMG is a direct reflection of muscular activities, the feedback of this model has a physical meaning. It expresses the dynamics of muscular activities in a straightforward way and takes advantage from the recurrent property. The trained model can then have the forces predicted directly from kinematic variables while bypassing the costly procedure of measuring EMG signals and avoiding the use of a biomechanics model. A learning algorithm is derived for the RFNN model.

Kernel methods, such as the support vector machine (SVM), are often formulated as quadratic programming (QP) problems. However, given m training patterns, a naive implementation of the QP solver takes O(m3) training time and at least O(m2) space. Hence, scaling up these QPs is a major stumbling block in applying kernel methods on very large data sets, and a replacement of the naive method for finding the QP solutions is highly desirable. Recently, by using approximation algorithms for the minimum enclosing ball (MEB) problem, we proposed the core vector machine (CVM) algorithm that is much faster and can handle much larger data sets than existing SVM implementations. However, the CVM can only be used with certain kernel functions and kernel methods. For example, the very popular support vector regression (SVR) cannot be used with the CVM. In this paper, we introduce the center-constrained MEB problem and subsequently extend the CVM algorithm. The generalized CVM algorithm can now be used with any linear/nonlinear kernel and can also be applied to kernel methods such as SVR and the ranking SVM. Moreover, like the original CVM, its asymptotic time complexity is again linear in m and its space complexity is independent of m. Experiments show that the generalized CVM has comparable performance with state-of-the-art SVM and SVR implementations, but is faster and produces fewer support vectors on very large data sets.

We propose an embedded/integrated feature selection method based on neural networks with Group Lasso penalty. Group Lasso regularization is considered to produce sparsity on the inputs to the network, i.e., for selection of useful features. Lasso based feature selection using a multi-layer perceptron usually requires an additional set of weights, while our Group Lasso formulation does not require that. However, Group Lasso penalty is non-differentiable at the origin. This may lead to oscillations in numerical simulations and make it difficult to analyze theoretically. To address this issue, four smoothing Group Lasso penalties are introduced. A rigorous proof for the convergence of the proposed algorithm is presented under suitable assumptions. To verify the effectiveness, a three-step algorithmic architecture is adopted in implementation. Experimental results on several datasets validate the theoretical results and demonstrate the competitive performance of the proposed method. [ABSTRACT FROM AUTHOR]

Identification of the significance of input variables is very important for complex systems with high-dimensional input space. In this paper, a method using fuzzy average with fuzzy cluster distribution is proposed. To avoid the interference of different distributions of the sampling data, the distribution of fuzzy clusters in the sampling data is considered, instead of the original data set. To discover the input-output relationship, the methods of fuzzy rules and fuzzy C-means are first used to partition the original sampling data set into fuzzy clusters. A new data set with the same distribution of the fuzzy clusters is produced. The fuzzy average method is then applied to the new data set. By doing so, the interference of distribution of the original sampling data is removed. This method is straightforward and computationally easy. The performance is tested on both benchmark data and real-world data. [ABSTRACT FROM AUTHOR]

Estimation of the dynamic spinal forces from kinematics data is very complicated because it involves the handling, of the relationship between kinematic variables and electromyography (EMG) signals, as well as the relationship between EMG signals and the forces. A recurrent fuzzy neural network (RFNN) model is proposed to establish the kinematics-EMG—force relationship and model the dynamics of muscular activities. The EMG signals are used as an intermediate output and are fed back to the input layer. Since EMG is a direct reflection of muscular activities, the feedback of this model has a physical meaning. It expresses the dynamics of muscular activities in a straightforward way and takes advantage from the recurrent property. The trained model can then have the forces predicted directly from kinematic variables while bypassing the costly procedure of measuring EMG signals and avoiding the use of a biomechanics model. A learning algorithm is derived for the RFNN model. [ABSTRACT FROM AUTHOR]

Kernel methods, such as the support vector machine (SVM), are often formulated as quadratic programming (QP) problems. However, given m training patterns, a naive implementation of the QP solver takes O(m3) training time and at least O(m2) space. Hence, scaling up these QPs is a major stumbling block in applying kernel methods on very large data sets, and a replacement of the naive method for finding the QP solutions is highly desirable. Recently, by using approximation algorithms for the minimum enclosing ball (MEB) problem, we proposed the core vector machine (CVM) algorithm that is much faster and can handle much larger data sets than existing SVM implementations. However, the CVM can only be used with certain kernel functions and kernel methods. For example, the very popular support vector regression (SVR) cannot be used with the CVM. In this paper, we introduce the center-constrained MEB problem and subsequently extend the CVM algorithm. The generalized CVM algorithm can now be used with any linear/nonlinear kernel and can also be applied to kernel methods such as SVR and the ranking SVM. Moreover, like the original CVM, its asymptotic time complexity is again linear in m and its space complexity is independent of m. Experiments show that the generalized CVM has comparable performance with state-of-the-art SVM and SVR implementations, but is faster and produces fewer support vectors on very large data sets. [ABSTRACT FROM AUTHOR]

The ability of bifurcating processing units and their networks to rapidly switch between different dynamic modes has been used in recent research efforts to model new computational properties of neural systems. In this spirit, we devise a bifurcating neuron based on control of chaos collapsing to a period-3 orbit in the dynamics of a quadratic logistic map (QLM). Proposed QLM3 neuron is constructed with the third iterate of QLM and uses an external input, which governs its dynamics. The input shifts the neuron''s dynamics from chaos to one of the stable fixed points. This way the inputs from certain ranges (clusters) are mapped to stable fixed points, while the rest of the inputs is mapped to chaotic or periodic output dynamics. It has been shown that QLM3 neuron is able to learn a specific mapping by adaptively adjusting its bifurcation parameter, the idea of which is based on the principles of parametric control of logistic maps [Proceedings of the International Symposium on Nonlinear Theory and its Applications (NOLTA''97), Honolulu, HI, 1997; Proceedings of SPIE, 2000]. Learning algorithm for the bifurcation parameter is proposed, which employs the error gradient descent method. [Copyright &y& Elsevier]

Presents a study that tested an approach for extracting rules from trained neural networks for nonlinear regression. Overview of related works for function approximation, pruning and rule extraction; Description of the neural training and pruning algorithm; Examination of the effectiveness of the proposed approach for function approximation.

Describes the structural learning with forgetting algorithm, with focus on the development of a continuous dynamical system model of regularization in which the associated regularization parameter is generalized to be a time-varying function. Features of the algorithm; Implications of the results; Conclusions reached.

To enhance the sparseness of the network, improve its generalization ability and accelerate its training, we propose a novel pruning approach for sigma-pi-sigma neural network (SPSNN) under the relaxed condition by adding smoothing group L 1 / 2 regularization and adaptive momentum. The main strength of this method is that it can prune both the redundant nodes between groups in the network, and also the redundant weights of the non-redundant nodes within the group, so as to achieve the sparseness of the network. Another strength is that the non-smooth absolute value function in the traditional L 1 / 2 regularization method is replaced by a smooth function. This reduces the oscillations of learning and enables us to more effectively prove the convergence of the proposed algorithm. Finally, the numerical simulation results demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Abstract: Conjugate gradient methods have many advantages in real numerical experiments, such as fast convergence and low memory requirements. This paper considers a class of conjugate gradient learning methods for backpropagation neural networks with three layers. We propose a new learning algorithm for almost cyclic learning of neural networks based on PRP conjugate gradient method. We then establish the deterministic convergence properties for three different learning modes, i.e., batch mode, cyclic and almost cyclic learning. The two deterministic convergence properties are weak and strong convergence that indicate that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. It is shown that the deterministic convergence results are based on different learning modes and dependent on different selection strategies of learning rate. Illustrative numerical examples are given to support the theoretical analysis. [Copyright &y& Elsevier]