Search

Abstract A smooth group regularization method is proposed to identify and remove the redundant input nodes of feedforward neural networks, or equivalently the redundant dimensions of the input data of a given data set. This is achieved by introducing a smooth group L 1/2 regularizer with respect to the input nodes into the error function to drive some weight vectors of the input nodes to zero. The main advantage of the method is that it can remove not only the redundant nodes, but also some redundant weights of the surviving nodes. As a comparison, the L 1 regularization (Lasso) is mainly designed for removing the redundant weights, and it is not very good at removing the redundant nodes. And the group Lasso can remove the redundant nodes, but not any weight of the surviving nodes. Another advantage of the proposed method is that it uses a smooth function to replace the non-smooth absolute value function in the common L 1/2 regularizer, and thus it reduces the oscillation caused by the non-smoothness and enables us to prove the convergence properties of the proposed training algorithm. Numerical simulations are performed to illustrate the efficiency of the algorithm. [ABSTRACT FROM AUTHOR]

This paper discusses the stability condition for discrete-time multi-valued recurrent neural networks (MVNRNNs) in asynchronous update mode. In the existing research literature, an MVNRNN in asynchronous update mode has been found convergent if its weight matrix is Hermitian with nonnegative diagonal entries. However, our finding has been that the weight matrix with zero diagonal entries cannot guarantee the network stability. Furthermore, the new stability condition and proof is offered to allow diagonal entries to be complex-valued, which extends previous theoretical result. Simulation results are used to illustrate the theory. [ABSTRACT FROM AUTHOR]

Abstract: Minimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L 2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing regularization term. For normal regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal regularization term can be addressed. This paper shows the strong convergence results for the smoothing regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L 2 and normal regularizations respectively. [Copyright &y& Elsevier]