1. A Gradient-Guided Evolutionary Approach to Training Deep Neural Networks
- Author
-
Cheng He, Kay Chen Tan, Shangshang Yang, Xingyi Zhang, Yaochu Jin, and Ye Tian
- Subjects
Network complexity ,Optimization problem ,Computer Networks and Communications ,Computer science ,Evolutionary algorithm ,02 engineering and technology ,Overfitting ,Genetic operator ,Machine learning ,computer.software_genre ,Local optimum ,Artificial Intelligence ,Robustness (computer science) ,0202 electrical engineering, electronic engineering, information engineering ,Hyperparameter ,Artificial neural network ,business.industry ,Biological Evolution ,Computer Science Applications ,020201 artificial intelligence & image processing ,Neural Networks, Computer ,Artificial intelligence ,business ,computer ,Algorithms ,Software ,Curse of dimensionality - Abstract
It has been widely recognized that the efficient training of neural networks (NNs) is crucial to classification performance. While a series of gradient-based approaches have been extensively developed, they are criticized for the ease of trapping into local optima and sensitivity to hyperparameters. Due to the high robustness and wide applicability, evolutionary algorithms (EAs) have been regarded as a promising alternative for training NNs in recent years. However, EAs suffer from the curse of dimensionality and are inefficient in training deep NNs (DNNs). By inheriting the advantages of both the gradient-based approaches and EAs, this article proposes a gradient-guided evolutionary approach to train DNNs. The proposed approach suggests a novel genetic operator to optimize the weights in the search space, where the search direction is determined by the gradient of weights. Moreover, the network sparsity is considered in the proposed approach, which highly reduces the network complexity and alleviates overfitting. Experimental results on single-layer NNs, deep-layer NNs, recurrent NNs, and convolutional NNs (CNNs) demonstrate the effectiveness of the proposed approach. In short, this work not only introduces a novel approach for training DNNs but also enhances the performance of EAs in solving large-scale optimization problems.
- Published
- 2022