Back to Search Start Over

Gap Minimization for Knowledge Sharing and Transfer

Authors :
Wang, Boyu
Mendez, Jorge
Shui, Changjian
Zhou, Fan
Wu, Di
Xu, Gezheng
Gagné, Christian
Eaton, Eric
Publication Year :
2022
Publisher :
arXiv, 2022.

Abstract

Learning from multiple related tasks by knowledge sharing and transfer has become increasingly relevant over the last two decades. In order to successfully transfer information from one task to another, it is critical to understand the similarities and differences between the domains. In this paper, we introduce the notion of \emph{performance gap}, an intuitive and novel measure of the distance between learning tasks. Unlike existing measures which are used as tools to bound the difference of expected risks between tasks (e.g., $\mathcal{H}$-divergence or discrepancy distance), we theoretically show that the performance gap can be viewed as a data- and algorithm-dependent regularizer, which controls the model complexity and leads to finer guarantees. More importantly, it also provides new insights and motivates a novel principle for designing strategies for knowledge sharing and transfer: gap minimization. We instantiate this principle with two algorithms: 1. gapBoost, a novel and principled boosting algorithm that explicitly minimizes the performance gap between source and target domains for transfer learning; and 2. gapMTNN, a representation learning algorithm that reformulates gap minimization as semantic conditional matching for multitask learning. Our extensive evaluation on both transfer learning and multitask learning benchmark data sets shows that our methods outperform existing baselines.<br />Comment: Journal of Machine Learning Research (JMLR) accepted

Details

Database :
OpenAIRE
Accession number :
edsair.doi.dedup.....b6c3551faa61dcbfa97e78f58d496e6a
Full Text :
https://doi.org/10.48550/arxiv.2201.11231