Back to Search
Start Over
Hessian regularization of deep neural networks: A novel approach based on stochastic estimators of Hessian trace.
- Source :
-
Neurocomputing . Jun2023, Vol. 536, p13-20. 8p. - Publication Year :
- 2023
-
Abstract
- [Display omitted] • Connecting Hessian trace with a generalization error bound. • Flat minima of loss landscape and stability analysis in dynamical systems. • Efficient Hessian trace regularization algorithm with Dropout. • Performance comparison on vision and language tasks. In this paper, we develop a novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by a recent guarantee bound of the generalization error. We explain its benefits in finding flat minima and avoiding Lyapunov stability in dynamical systems. We adopt the Hutchinson method as a classical unbiased estimator for the trace of a matrix and further accelerate its calculation using a Dropout scheme. Experiments demonstrate that our method outperforms existing regularizers and data augmentation methods, such as Jacobian, Confidence Penalty, Label Smoothing, Cutout, and Mixup. The code is available at https://github.com/Dean-lyc/Hessian-Regularization. [ABSTRACT FROM AUTHOR]
Details
- Language :
- English
- ISSN :
- 09252312
- Volume :
- 536
- Database :
- Academic Search Index
- Journal :
- Neurocomputing
- Publication Type :
- Academic Journal
- Accession number :
- 163017900
- Full Text :
- https://doi.org/10.1016/j.neucom.2023.03.017