Benchmark for Personalized Federated Learning

Federated learning is a distributed machine learning approach that allows a single server to collaboratively build machine learning models with multiple clients without sharing datasets. Since data distributions may differ across clients, data heterogeneity is a challenging issue in federated learning. To address this issue, numerous federated learning methods have been proposed to build personalized models for clients, referred to as personalized federated learning. Nevertheless, no studies comprehensively investigate the performance of personalized federated learning methods in various experimental settings such as datasets and client settings. Therefore, in this article, we aim to benchmark the performance of existing personalized federated learning methods in various settings. We first survey the experimental settings in existing studies. We then benchmark the performance of existing methods through comprehensive experiments to reveal their characteristics in computer vision and natural language processing tasks which are the most popular tasks based on our survey. Our experimental study shows that (i) large data heterogeneity often leads to highly accurate predictions and (ii) standard federated learning methods (e.g. FedAvg) with fine-tuning often outperform personalized federated learning methods.