1. Two highly efficient second-order algorithms for training feedforward networks
- Author
-
Ampazis, Nikolaos and Perantonis, Stavros J.
- Subjects
Electrical engineering -- Research ,Neural networks -- Research ,Fuzzy logic -- Usage ,Least squares -- Usage ,Machine learning -- Research ,Business ,Computers ,Electronics ,Electronics and electrical industries - Abstract
In this paper, we present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Their implementation requires minimal additional computations compared to a standard LM iteration which are compensated, however, from their excellent convergence properties. Simulations to large scale classical neural-network benchmarks are presented which reveal the power of the two methods to obtain solutions in difficult problems, whereas other standard second-order techniques (including LM) fail to converge. Index Terms--Algorithms, multilayer feedforward neural networks, nonlinear least-squares, optimization, supervised learning.
- Published
- 2002