Back to Search
Start Over
Zero-Shot Knowledge Distillation Using Label-Free Adversarial Perturbation With Taylor Approximation
- Source :
- IEEE Access, Vol 9, Pp 45454-45461 (2021)
- Publication Year :
- 2021
- Publisher :
- Institute of Electrical and Electronics Engineers (IEEE), 2021.
-
Abstract
- Knowledge distillation (KD) is one of the most effective neural network light-weighting techniques when training data is available. However, KD is seldom applicable to an environment where it is difficult or impossible to access training data. To solve this problem, a complete zero-shot KD (C-ZSKD) based on adversarial learning has been recently proposed, but the so-called biased sample generation problem limits the performance of C-ZSKD. To overcome this limitation, this paper proposes a novel C-ZSKD algorithm that utilizes a label-free adversarial perturbation. The proposed adversarial perturbation derives a constraint of the squared norm of gradient style by using the convolution of probability distributions and the 2nd order Taylor series approximation. The constraint serves to increase the variance of the adversarial sample distribution, which makes the student model learn the decision boundary of the teacher model more accurately without labeled data. Through analysis of the distribution of adversarial samples on the embedded space, this paper also provides an insight into the characteristics of adversarial samples that are effective for adversarial learning-based C-ZSKD.
- Subjects :
- General Computer Science
Computer science
010501 environmental sciences
01 natural sciences
Convolution
symbols.namesake
0502 economics and business
Taylor series
General Materials Science
050207 economics
0105 earth and related environmental sciences
Artificial neural network
05 social sciences
General Engineering
Constraint (information theory)
knowledge distillation
Sampling distribution
Norm (mathematics)
symbols
Decision boundary
Probability distribution
lcsh:Electrical engineering. Electronics. Nuclear engineering
lcsh:TK1-9971
Algorithm
adversarial learning
Zero-shot learning
Subjects
Details
- ISSN :
- 21693536
- Volume :
- 9
- Database :
- OpenAIRE
- Journal :
- IEEE Access
- Accession number :
- edsair.doi.dedup.....c59d9a3af92a9de2800b22e1e51ef5c6