Search

This work proves the deterministic convergence of the Sigma-Pi-Sigma neural network based on the batch gradient learning algorithm under certain relaxed conditions. We establish strong and weak convergence results and prove that the error function decreases monotonically and tends to zero. The boundedness of weights is also proved simply and efficiently. In contrast to the usual requirements for the boundedness of weights, this work shows that such weight boundedness is no more one of the necessary conditions for ensuring convergence. In addition, we also show that the requirements on the learning rate and the stationary point set of the error function can be relaxed. Finally, the effectiveness of the proposed algorithm is validated by numerical experiments, followed by brief conclusions. [ABSTRACT FROM AUTHOR]

In this paper, we propose a sparse and accelerated method for Sigma-Pi-Sigma neural network training based on smoothing group lasso regularization and adaptive momentum. It is shown that group sparsity can more efficiently sparsity the network structure at the group level, and the adaptive momentum term can speed up the learning convergence during the iteration process. Also, another important contribution lies in the analysis of theoretical results. However, the group lasso regularization is not differentiable at the origin. This leads to oscillations observed in numerical experiments and poses a challenge to theoretical analysis. So, we overcome those problems by smoothing techniques. Under suitable assumptions, we strictly proved the monotonicity, and weak and strong convergence theorems of the new algorithm. Finally, the numerical experiments are presented to support our theoretical findings. [ABSTRACT FROM AUTHOR]

Elman network is a classical recurrent neural network with an internal delay feedback. In this paper, we propose a recalling-enhanced recurrent neural network (RERNN) which has a selective memory property. In addition, an improved conjugate algorithm with generalized Armijo search technique that speeds up the convergence rate is used to train the RERNN model. Further enhancement performance is achieved with adaptive learning coefficients. Finally, we prove weak and strong convergence of the presented algorithm. In other words, as the number of training steps increases, the following has been established for RERNN: (1) the gradient norm of the error function with respect to the weight vectors converges to zero, (2) the weight sequence approaches a fixed optimal point. We have carried out a number of simulations to illustrate and verify the theoretical results that demonstrate the efficiency of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In this paper, a new variant of feedforward neural networks has been proposed for a class of nonsmooth optimization problems. The penalty term of the presented neural networks stems from the Group Lasso method which selects hidden variables in a grouped manner. To deal with the non-differentiability of the original penalty term ( l 1 − l 2 norm) and avoid oscillations, smoothing techniques have been used to approximate the objective function. It is assumed that the training samples are supplied to the networks in a specific incremental way during training, that is, in each cycle samples are supplied in a fixed order. Then, under suitable assumptions on learning rate, penalization coefficients and smoothing parameters, the weak and strong convergence of the training process for the smoothing neural networks have been proved. The convergence analysis shows that the gradient of the smoothing error function approaches zero and the weight sequence converges to a fixed point, respectively. We demonstrate how the smoothing approximation parameter can be updated in the training procedure so as to guarantee the convergence of the procedure to a Clarke stationary point of the original optimization problem. In addition, we have proved that the original nonsmoothing algorithm with l 1 − l 2 norm penalty converges consistently to the same optimum solution with the corresponding smoothed algorithm. Numerical simulations demonstrate the convergence and effectiveness of the proposed training algorithm. [ABSTRACT FROM AUTHOR]