Search

Abstract A smooth group regularization method is proposed to identify and remove the redundant input nodes of feedforward neural networks, or equivalently the redundant dimensions of the input data of a given data set. This is achieved by introducing a smooth group L 1/2 regularizer with respect to the input nodes into the error function to drive some weight vectors of the input nodes to zero. The main advantage of the method is that it can remove not only the redundant nodes, but also some redundant weights of the surviving nodes. As a comparison, the L 1 regularization (Lasso) is mainly designed for removing the redundant weights, and it is not very good at removing the redundant nodes. And the group Lasso can remove the redundant nodes, but not any weight of the surviving nodes. Another advantage of the proposed method is that it uses a smooth function to replace the non-smooth absolute value function in the common L 1/2 regularizer, and thus it reduces the oscillation caused by the non-smoothness and enables us to prove the convergence properties of the proposed training algorithm. Numerical simulations are performed to illustrate the efficiency of the algorithm. [ABSTRACT FROM AUTHOR]

Unlike the first and the second generation artificial neural networks, spiking neural networks (SNNs) model the human brain by incorporating not only synaptic state but also a temporal component into their operating model. However, their intrinsic properties require expensive computation during training. This paper presents a novel algorithm to SpikeProp for SNN by introducing smoothing L 1 ∕ 2 regularization term into the error function. This algorithm makes the network structure sparse, with some smaller weights that can be eventually removed. Meanwhile, the convergence of this algorithm is proved under some reasonable conditions. The proposed algorithms have been tested for the convergence speed, the convergence rate and the generalization on the classical XOR-problem, Iris problem and Wisconsin Breast Cancer classification. [ABSTRACT FROM AUTHOR]

This paper proposes new techniques for data representation in the context of deep learning using agglomerative clustering. Existing autoencoder-based data representation techniques tend to produce a number of encoding and decoding receptive fields of layered autoencoders that are duplicative, thereby leading to extraction of similar features, thus resulting in filtering redundancy. We propose a way to address this problem and show that such redundancy can be eliminated. This yields smaller networks and produces unique receptive fields that extract distinct features. It is also shown that autoencoders with nonnegativity constraints on weights are capable of extracting fewer redundant features than conventional sparse autoencoders. The concept is illustrated using conventional sparse autoencoder and nonnegativity-constrained autoencoders with MNIST digits recognition, NORB normalized-uniform object data and Yale face dataset. [ABSTRACT FROM AUTHOR]

This paper discusses the stability condition for discrete-time multi-valued recurrent neural networks (MVNRNNs) in asynchronous update mode. In the existing research literature, an MVNRNN in asynchronous update mode has been found convergent if its weight matrix is Hermitian with nonnegative diagonal entries. However, our finding has been that the weight matrix with zero diagonal entries cannot guarantee the network stability. Furthermore, the new stability condition and proof is offered to allow diagonal entries to be complex-valued, which extends previous theoretical result. Simulation results are used to illustrate the theory. [ABSTRACT FROM AUTHOR]

Abstract: Minimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L 2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing regularization term. For normal regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal regularization term can be addressed. This paper shows the strong convergence results for the smoothing regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L 2 and normal regularizations respectively. [Copyright &y& Elsevier]

Abstract: This paper discusses a Competitive Layer Model (CLM) for a class of recurrent Cellular Neural Networks (CNNs) from continuous-time type to discrete-time type. The objective of the CLM is to partition a set of input features into salient groups. The complete convergence of such networks in continuous-time type has been discussed first. We give a necessary condition, and a necessary and sufficient condition, which allow the CLM performance existence in our networks. We also discuss the properties of such networks of discrete-time type, and propose a novel CLM iteration method. Such method shows similar performance and storage allocation but faster convergence compared with the previous CLM iteration method (). Especially for a large scale network with many features and layers, it can significantly reduce the computing time. Examples and simulation results are used to illustrate the developed theory, the comparison between two CLM iteration methods, and the application in image segmentation. [Copyright &y& Elsevier]

Abstract: This paper discusses a class of discrete time recurrent neural networks with multivalued neurons (MVN), which have complex-valued weights and an activation function defined as a function of the argument of a weighted sum. Complementing state-of-the-art of such networks, our research focuses on the convergence analysis of the networks in synchronous update mode. Two related theorems are presented and simulation results are used to illustrate the theory. [Copyright &y& Elsevier]

Abstract: The paper describes an application of artificial neural networks (ANN) for natural language text reasoning. The task of knowledge discovery in text from a database, represented with a database file consisting of sentences with similar meanings but different lexico-grammatical patterns, was solved with ANNs which recognize the meaning of the text using training files with limited dictionary. The paper features recognition algorithms of text meaning from a selected source using 3-layer ANNs. Tests of the new method have also been described. [Copyright &y& Elsevier]

Abstract: Superposition of radial basis functions centered at given prototype patterns constitutes one of the most suitable energy forms for gradient systems that perform nearest neighbor classification with real-valued static prototypes. It is shown in this paper that a continuous-time dynamical neural network model, employing a radial basis function and a sigmoid multi-layer perceptron sub-networks, is capable of maximizing such an energy form locally, thus performing almost perfectly nearest neighbor classification, when initiated by a distorted pattern. The proposed design scheme allows for explicit representation of prototype patterns as network parameters, as well as augmenting additional or forgetting existing memory patterns. The dynamical classification scheme implemented by the network eliminates all comparisons, which are the vital steps of the conventional nearest neighbor classification process. The performance of the proposed network model is demonstrated on binary and gray-scale image reconstruction applications. [Copyright &y& Elsevier]

The ability of bifurcating processing units and their networks to rapidly switch between different dynamic modes has been used in recent research efforts to model new computational properties of neural systems. In this spirit, we devise a bifurcating neuron based on control of chaos collapsing to a period-3 orbit in the dynamics of a quadratic logistic map (QLM). Proposed QLM3 neuron is constructed with the third iterate of QLM and uses an external input, which governs its dynamics. The input shifts the neuron''s dynamics from chaos to one of the stable fixed points. This way the inputs from certain ranges (clusters) are mapped to stable fixed points, while the rest of the inputs is mapped to chaotic or periodic output dynamics. It has been shown that QLM3 neuron is able to learn a specific mapping by adaptively adjusting its bifurcation parameter, the idea of which is based on the principles of parametric control of logistic maps [Proceedings of the International Symposium on Nonlinear Theory and its Applications (NOLTA''97), Honolulu, HI, 1997; Proceedings of SPIE, 2000]. Learning algorithm for the bifurcation parameter is proposed, which employs the error gradient descent method. [Copyright &y& Elsevier]

This paper presents an efficient technique to reduce the inference cost of deep and/or wide convolutional neural network models by pruning redundant features (or filters). Previous studies have shown that over-sized deep neural network models tend to produce a lot of redundant features that are either shifted version of one another or are very similar and show little or no variations, thus resulting in filtering redundancy. We propose to prune these redundant features along with their related feature maps according to their relative cosine distances in the feature space, thus leading to smaller networks with reduced post-training inference computational costs and competitive performance. We empirically show on select models (VGG-16, ResNet-56, ResNet-110, and ResNet-34) and dataset (MNIST Handwritten digits, CIFAR-10, and ImageNet) that inference costs (in FLOPS) can be significantly reduced while overall performance is still competitive with the state-of-the-art. [ABSTRACT FROM AUTHOR]

To enhance the sparseness of the network, improve its generalization ability and accelerate its training, we propose a novel pruning approach for sigma-pi-sigma neural network (SPSNN) under the relaxed condition by adding smoothing group L 1 / 2 regularization and adaptive momentum. The main strength of this method is that it can prune both the redundant nodes between groups in the network, and also the redundant weights of the non-redundant nodes within the group, so as to achieve the sparseness of the network. Another strength is that the non-smooth absolute value function in the traditional L 1 / 2 regularization method is replaced by a smooth function. This reduces the oscillations of learning and enables us to more effectively prove the convergence of the proposed algorithm. Finally, the numerical simulation results demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Abstract: This paper presents how commonly used machine learning classifiers can be analyzed using a common framework of convex optimization. Four classifier models, the Support Vector Machine (SVM), the Least-Squares SVM (LSSVM), the Extreme Learning Machine (ELM), and the Margin Loss ELM (MLELM) are discussed to demonstrate how specific parametrizations of a general problem statement affect the classifier design and performance, and how ideas from the four different classifiers can be mixed and used together. Furthermore, 21 public domain benchmark datasets are used to experimentally evaluate five performance metrics of each model and corroborate the theoretical analysis. Comparison of classification accuracies under a nested cross-validation evaluation shows that with an exception all four models perform similarly on the evaluated datasets. However, the four classifiers command different amounts of computational resources for both testing and training. These requirements are directly linked to their formulations as different convex optimization problems. [Copyright &y& Elsevier]

Abstract: The aim of this paper is to develop a novel method to prune feedforward neural networks by introducing an regularization term into the error function. This procedure forces weights to become smaller during the training and can eventually removed after the training. The usual regularization term involves absolute values and is not differentiable at the origin, which typically causes oscillation of the gradient of the error function during the training. A key point of this paper is to modify the usual regularization term by smoothing it at the origin. This approach offers the following three advantages: First, it removes the oscillation of the gradient value. Secondly, it gives better pruning, namely the final weights to be removed are smaller than those produced through the usual regularization. Thirdly, it makes it possible to prove the convergence of the training. Supporting numerical examples are also provided. [Copyright &y& Elsevier]

Abstract: Weight decay method as one of classical complexity regularization methods is simple and appears to work well in some applications for backpropagation neural networks (BPNN). This paper shows results for the weak and strong convergence for cyclic and almost cyclic learning BPNN with penalty term (CBP-P and ACBP-P). The convergence is guaranteed under certain relaxed conditions for activation functions, learning rate and under the assumption for the stationary set of error function. Furthermore, the boundedness of the weights in the training procedure is obtained in a simple and clear way. Numerical simulations are implemented to support our theoretical results and demonstrate that ACBP-P has better performance than CBP-P on both convergence speed and generalization ability. [Copyright &y& Elsevier]

Abstract: Conjugate gradient methods have many advantages in real numerical experiments, such as fast convergence and low memory requirements. This paper considers a class of conjugate gradient learning methods for backpropagation neural networks with three layers. We propose a new learning algorithm for almost cyclic learning of neural networks based on PRP conjugate gradient method. We then establish the deterministic convergence properties for three different learning modes, i.e., batch mode, cyclic and almost cyclic learning. The two deterministic convergence properties are weak and strong convergence that indicate that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. It is shown that the deterministic convergence results are based on different learning modes and dependent on different selection strategies of learning rate. Illustrative numerical examples are given to support the theoretical analysis. [Copyright &y& Elsevier]

Abstract: This study investigates the effect of class imbalance in training data when developing neural network classifiers for computer-aided medical diagnosis. The investigation is performed in the presence of other characteristics that are typical among medical data, namely small training sample size, large number of features, and correlations between features. Two methods of neural network training are explored: classical backpropagation (BP) and particle swarm optimization (PSO) with clinically relevant training criteria. An experimental study is performed using simulated data and the conclusions are further validated on real clinical data for breast cancer diagnosis. The results show that classifier performance deteriorates with even modest class imbalance in the training data. Further, it is shown that BP is generally preferable over PSO for imbalanced training data especially with small data sample and large number of features. Finally, it is shown that there is no clear preference between oversampling and no compensation approach and some guidance is provided regarding a proper selection. [Copyright &y& Elsevier]

Abstract: In this paper, we present a new approach of designing adaptive neural network controllers for uncertain systems containing nonsmooth nonlinearities in the actuator device. The controllers are designed by introducing certain well-defined sign functions and neural network approximations as well as by using the backstepping technique. The salient feature of the approach is that no knowledge is assumed on unknown system parameters and nonlinearities. It is shown that the proposed controller not only can guarantee semi-global stability, but also excellent transient performance can be achieved. [Copyright &y& Elsevier]

Analysis of backscatter in the ultrasound echo envelope, in conjunction with ultrasound B-scans, can provide important information for tissue characterization and pathology diagnosis. Statistical models have often proven useful in modeling backscatter. In this paper, an innovative approach to backscatter analysis based on generalized entropies and neural function approximation is presented. Entropy measures are shown to provide accurate estimates of scatterer density, regularity, and SNR of the amplitude distribution. Specific scattering distributions need not be assumed. Experimental results on ground truth envelopes show that generalized entropies can be used to accurately estimate backscatter properties. [Copyright &y& Elsevier]

Background and Objective Anemia is a common comorbidity in patients with chronic kidney disease (CKD) and is frequently associated with decreased physical component of quality of life, as well as adverse cardiovascular events. Current treatment methods for renal anemia are mostly population-based approaches treating individual patients with a one-size-fits-all model. However, FDA recommendations stipulate individualized anemia treatment with precise control of the hemoglobin concentration and minimal drug utilization. In accordance with these recommendations, this work presents an individualized drug dosing approach to anemia management by leveraging the theory of optimal control. Methods A Multiple Receding Horizon Control (MRHC) approach based on the RBF-Galerkin optimization method is proposed for individualized anemia management in CKD patients. Recently developed by the authors, the RBF-Galerkin method uses the radial basis function approximation along with the Galerkin error projection to solve constrained optimal control problems numerically. The proposed approach is applied to generate optimal dosing recommendations for individual patients. Results Performance of the proposed approach (MRHC) is compared in silico to that of a population-based anemia management protocol and an individualized multiple model predictive control method for two case scenarios: hemoglobin measurement with and without observational errors. In silico comparison indicates that hemoglobin concentration with MRHC method has less variation among the methods, especially in presence of measurement errors. In addition, the average achieved hemoglobin level from the MRHC is significantly closer to the target hemoglobin than that of the other two methods, according to the analysis of variance (ANOVA) statistical test. Furthermore, drug dosages recommended by the MRHC are more stable and accurate and reach the steady-state value notably faster than those generated by the other two methods. Conclusions The proposed method is highly efficient for the control of hemoglobin level, yet provides accurate dosage adjustments in the treatment of CKD anemia. [ABSTRACT FROM AUTHOR]

Weight elimination offers a simple and efficient improvement of training algorithm of feedforward neural networks. It is a general regularization technique in terms of the flexible scaling parameters. Actually, the weight elimination technique also contains the weight decay regularization for a large scaling parameter. Many applications of this technique and its improvements have been reported. However, there is little research concentrated on its convergence behavior. In this paper, we theoretically analyze the weight elimination for cyclic learning method and determine the conditions for the uniform boundedness of weight sequence, and weak and strong convergence. Based on the assumed network parameters, the optimal choice for the scaling parameter can also be determined. Moreover, two illustrative simulations have been done to support the theoretical explorations as well. [ABSTRACT FROM AUTHOR]

The universal sequel to chronic kidney condition (CKD) is anemia. Patients of anemia have kidneys that are incapable of performing certain basic functions such as sensing of oxygen levels to secrete erythropoietin when red blood cell counts are low. Under such conditions, external administration of human recombinant erythropoietin (EPO) is administered as alternative to improve conditions of CKD patients by increasing their hemoglobin (Hb) levels to a given therapeutic range. Presently, EPO dosing strategies extensively depend on packet inserts and on “average” responses to the medication from previous patients. Clearly dosage strategies based on these approaches are, at best, nonoptimal to EPO medication and potentially dangerous to patients that do not adhere to the notion of expected “average” response. In this work, a technique called semi-blind robust identification is provided to uniquely identify models of the individual patients of anemia based on their actual Hb responses and EPO administration. Using the a priori information and the measured input-output data of the individual patients, the procedure identifies a unique model consisting of a nominal model and the associated model uncertainty for the patients. By incorporating the effects of unknown system initial conditions, considerably small measurement samples can be used in the modeling process. [ABSTRACT FROM AUTHOR]

Abstract: Effective management of anemia due to renal failure poses many challenges to physicians. Individual response to treatment varies across patient populations and, due to the prolonged character of the therapy, changes over time. In this work, a Reinforcement Learning-based approach is proposed as an alternative method for individualization of drug administration in the treatment of renal anemia. Q-learning, an off-policy approximate dynamic programming method, is applied to determine the proper dosing strategy in real time. Simulations compare the proposed methodology with the currently used dosing protocol. Presented results illustrate the ability of the proposed method to achieve the therapeutic goal for individuals with different response characteristics and its potential to become an alternative to currently used techniques. [Copyright &y& Elsevier]

This work presents a pharmacodynamic population analysis in chronic renal failure patients using Artificial Neural Networks (ANNs). In pursuit of an effective and cost-efficient strategy for drug delivery in patients with renal failure, two different types of ANN are applied to perform drug dose-effect modeling and their performance compared. Applied in a clinical environment, such models will allow for prediction of patient response to the drug at the effect site and, subsequently, for adjusting the dosing regimen. [Copyright &y& Elsevier]