FedDecay: Balancing Model Performance and Rapid Personalization in Federated Learning with Learning Rate Scheduling

Federated learning (FL) is a powerful technique for collaboratively training a single centralized model on distributed local data sources. By aggregating model information without disclosing the local training data, FL preserves data privacy. However, the inherent heterogeneity in local data sets challenges the performance of FL techniques, especially when data is very diverse across local sources. Personalized Federated Learning (PFL) can mitigate this challenge using multiple models but often requires additional memory and computation. This work does not propose a new PFL method but instead explores how learning rate decay within each local training round can improve model performance across local data sets after fine-tuning. We provide theoretical insights and empirical evidence of efficacy across diverse domains, including vision, text, and graph data. Our extensive experiments demonstrate that learning rate scheduling alone outperforms other FL methods regarding generalization to new data for both new and existing users. Moreover, it performs comparably to PFL methods, particularly for new users, while maintaining similar computational and memory requirements as FL techniques. Supplementary materials, including all code and instructions for reproducing the experiments, are available online.