An Adaptive Online Knowledge Distillation Algorithm for Edge Computing Models Enhanced by Elite-Students

In recent years, deep learning models have exhibited exceptional performance across several tasks. However, the substantial computational and storage demands impede implementation on edge devices with constrained resources. Online Knowledge Distillation (OKD) has arisen as an effective model compression strategy that removes the reliance on pre-trained teachers characteristic of conventional distillation approaches. Nonetheless, OKD persists in facing challenges, including substantial performance variances within student networks, insufficient learning capacity for difficult data, and network homogeneity. To address those issues, this paper proposes an Elite-Student-Enhanced Adaptive Online Knowledge Distillation (ESAKD) algorithm. ESAKD introduces a patience factor-based adaptive temperature scheduling mechanism to dynamically balance knowledge clarity and richness during knowledge transfer. This mechanism utilizes the performance benefits of elite-students, particularly during initial training phases, to offer superior supervision that successfully transcends the learning capacity limitations of current OKD approaches. This method promotes swift convergence and substantially enhances the ultimate precision of the standard-student models. Furthermore, a confidence-weighted ensemble student model is designed to improve collective decision-making. Experimental assessments indicate that ESAKD provides substantial performance improvements over supervised learning without distillation and other leading online distillation techniques. On the CIFAR-100 dataset, ESAKD improves the test accuracy of various models by 1.49–6% over the undistilled baselines, and by 0.27–2.18% compared to advanced online distillation algorithms. Moreover, it exhibits enhanced performance on the Tiny-ImageNet-200 dataset as well.