Federated Learning (FL) presents a compelling distributed learning paradigm that enables resource-constrained clients to collaboratively train machine learning models while preserving data privacy. However, inter-client data heterogeneity poses fundamental challenges to federated optimization efficacy. Although Knowledge Distillation (KD) effectively addresses model performance alignment under heterogeneity, its inherent constraints restrict client parameter exploration capacity, thus, inducing confinement to suboptimal basins. To reconcile this trade-off, we introduce FedKDMR, a novel FL paradigm unifying KD constraints with exploration via model recombination. FedKDMR imposes global model consistency and robustness in training through dynamic KD while sufficiently harnessing model recombination-induced perturbations for diverse parameter exploration. We establish convergence guarantees for strongly convex and smooth objectives. Extensive experiments on FL benchmark datasets demonstrate that FedKDMR achieves a superior accuracy-robustness trade-off against state-of-the-art methods when tackling non-independent and identically distributed and heterogeneous data in FL environments.
Li et al. (Mon,) studied this question.