Weighted Federated Distillation: A knowledge-quality-aware, teacher-less strategy

The rapid advancements in Artificial Intelligence (AI) and edge devices have driven the proliferation of smart-world applications, many of which are deployed in distributed environments. In contexts where data privacy must be maintained, Federated Learning (FL) has emerged as a promising privacy-preserving collaborative learning paradigm, offering significant potential to harness the distributed data available at the network’s edge. However, the heterogeneity of devices and the non-Independent and Identically Distributed (IID) nature of data across them pose significant challenges, undermining the performance of FL systems. To address these issues, Knowledge Distillation (KD) has proven to be a valid solution. Initially developed as a teacher-student training paradigm, KD enhances the performance of smaller student networks by transferring knowledge from larger, more powerful teacher networks. When integrated with FL, KD offers additional benefits: it mitigates the effects of data and model heterogeneity and compensates for the absence of a centralized teacher model. This is achieved by creating a global knowledge representation derived from the aggregated knowledge of individual clients. Building on these principles, we propose Weighted-FD, a novel framework that introduces a quality-aware approach to global knowledge computation. Unlike conventional methods, Weighted-FD evaluates the quality of knowledge contributed by each client and dynamically adjusts their influence on the global knowledge representation. This ensures a more accurate and effective aggregation process. We detail the mathematical foundation of our framework and validate its efficacy through extensive experiments conducted on MNIST, FashionMNIST, and CIFAR-10 under different data heterogeneity settings (IID, weak non-IID, and strong non-IID). The results demonstrate that the proposed method consistently outperforms existing federated distillation approaches. In particular, Weighted-FD achieves substantial improvements under strong non-IID conditions, reaching accuracy gains of up to 57.12% over FedMD and 49.34% over Selective-FD on CIFAR-10, while maintaining low computational and memory requirements, making it well suited for deployment in resource-constrained edge environments. The source code for Weighted-FD is publicly available at the following link: https://anonymous.4open.science/r/weighted-fd-C6EC/ .