Back to Search
Start Over
Geometric Regularization of Local Activations for Knowledge Transfer in Convolutional Neural Networks
- Source :
- Information, Vol 12, Iss 333, p 333 (2021), Information, Volume 12, Issue 8
- Publication Year :
- 2021
- Publisher :
- MDPI AG, 2021.
-
Abstract
- In this work, we propose a mechanism for knowledge transfer between Convolutional Neural Networks via the geometric regularization of local features produced by the activations of convolutional layers. We formulate appropriate loss functions, driving a “student” model to adapt such that its local features exhibit similar geometrical characteristics to those of an “instructor” model, at corresponding layers. The investigated functions, inspired by manifold-to-manifold distance measures, are designed to compare the neighboring information inside the feature space of the involved activations without any restrictions in the features’ dimensionality, thus enabling knowledge transfer between different architectures. Experimental evidence demonstrates that the proposed technique is effective in different settings, including knowledge-transfer to smaller models, transfer between different deep architectures and harnessing knowledge from external data, producing models with increased accuracy compared to a typical training. Furthermore, results indicate that the presented method can work synergistically with methods such as knowledge distillation, further increasing the accuracy of the trained models. Finally, experiments on training with limited data show that a combined regularization scheme can achieve the same generalization as a non-regularized training with 50% of the data in the CIFAR-10 classification task.
- Subjects :
- Scheme (programming language)
manifold regularization
business.industry
Computer science
Generalization
Feature vector
Pattern recognition
Information technology
knowledge transfer
T58.5-58.64
Regularization (mathematics)
Convolutional neural network
Distance measures
knowledge distillation
deep learning with limited data
Artificial intelligence
business
computer
Knowledge transfer
Information Systems
computer.programming_language
Curse of dimensionality
Subjects
Details
- Language :
- English
- ISSN :
- 20782489
- Volume :
- 12
- Issue :
- 333
- Database :
- OpenAIRE
- Journal :
- Information
- Accession number :
- edsair.doi.dedup.....ad6e3b3ea6a82af176f9c8a45879df68