Abstract
Heterogeneous Federated Learning (HtFL) enables collaborative learning on multiple clients with different model architectures while preserving privacy. Despite recent research progress, knowledge sharing in HtFL is still difficult due to data and model heterogeneity. To tackle this issue, we leverage the knowledge stored in public pre-trained generators and propose a new upload-efficient knowledge transfer scheme called Federated Knowledge-Transfer Loop $($FedKTL$)$. Our FedKTL can produce client-task-related prototypical image-vector pairs via the generator’s inference on the server. With these pairs, each client can transfer preexisting knowledge from the generator to its local model through an additional supervised local task. We conduct extensive experiments on four datasets under two types of data heterogeneity with 14 kinds of models including CNNs and ViTs. Results show that our upload-efficient FedKTL surpasses seven state-of-the-art methods by up to 7.31% in accuracy. Moreover, our knowledge transfer scheme is applicable in scenarios with only one edge client.
- $($Heterogeneous$)$ Federated Learning: Federated learning aims to train a deep neural network (DNN) model on distributed devices while maintaining data privacy. However, device and system heterogeneity pose challenges in terms of hardware variations, computing power differences, and limited bandwidth.
- Knowledge sharing is a crucial component in federated learning.
- The work proposes upload-efficient knowledge transfer scheme FedKTL.