Back to Search
Start Over
A Generalized Framework for Population Based Training
- Source :
- KDD
- Publication Year :
- 2019
-
Abstract
- Population Based Training (PBT) is a recent approach that jointly optimizes neural network weights and hyperparameters which periodically copies weights of the best performers and mutates hyperparameters during training. Previous PBT implementations have been synchronized glass-box systems. We propose a general, black-box PBT framework that distributes many asynchronous "trials" (a small number of training steps with warm-starting) across a cluster, coordinated by the PBT controller. The black-box design does not make assumptions on model architectures, loss functions or training procedures. Our system supports dynamic hyperparameter schedules to optimize both differentiable and non-differentiable metrics. We apply our system to train a state-of-the-art WaveNet generative model for human voice synthesis. We show that our PBT system achieves better accuracy, less sensitivity and faster convergence compared to existing methods, given the same computational resource.<br />9 pages
- Subjects :
- FOS: Computer and information sciences
Computer Science - Machine Learning
Computer science
Computer Science - Artificial Intelligence
Evolutionary algorithm
02 engineering and technology
Machine learning
computer.software_genre
Computational resource
Machine Learning (cs.LG)
Control theory
020204 information systems
0202 electrical engineering, electronic engineering, information engineering
Neural and Evolutionary Computing (cs.NE)
Differentiable function
Hyperparameter
Artificial neural network
business.industry
Computer Science - Neural and Evolutionary Computing
Generative model
Artificial Intelligence (cs.AI)
Computer Science - Distributed, Parallel, and Cluster Computing
020201 artificial intelligence & image processing
Distributed, Parallel, and Cluster Computing (cs.DC)
Artificial intelligence
business
computer
Subjects
Details
- Language :
- English
- Database :
- OpenAIRE
- Journal :
- KDD
- Accession number :
- edsair.doi.dedup.....0c7596d67ebb76da7226bd41823a905f