A novel deep auto-encoder considering energy and label constraints for categorization

Deep auto-encoder (DAE) is one of the representative deep learning algorithms for feature extraction. However, it often shows relatively poor generalization performance to express data without considering the probability distribution of data. Additionally, it cannot be directly applied to classification, because label information is ignored in DAE to judge the given categories. To tackle these issues, in this paper, we propose an energy and label constrained DAE (ELDAE) by integrating energy and label constraints to improve the feature extraction ability of network for classification. Specifically, as the probability distribution for fitting data can be reflected by energy of a network, the energy constraint is designed in this study to improve the probability of ELDAE for fitting data, and make a better expression to data. Moreover, the label constraint is integrated in ELDAE using label information to describe categorization rule, contributing to enhancing the accuracy of classification. We first give the complexity analysis of ELDAE, which is crucial to the property of speed. To exhibit the performance of the proposed ELDAE, we perform comprehensive experiments on benchmark USPS and MNIST datasets, and parameter sensitivity analysis is then provided to investigate the effects of three key parameters including the balance coefficients of weight decay, energy constraint and label constraint. In addition, we compare ELDAE with six state-of-the-art algorithms including Auto-Encoder (AE), Sparse AE (SAE), Deep AE (DAE), Deep Belief Network (DBN), Noisy AE (NAE) and Semi-supervised AE (SSAE). The comparative experimental results demonstrate that ELDAE performs better than the other six competitors in terms of classification accuracy, and keeps the same order of magnitude in terms of training time and testing time.