Beyond fixed hyper-parameters: Adaptive temperature and label smoothing for knowledge distillation

Current knowledge distillation (KD) methods predominantly rely on globally fixed temperature coefficients and label smoothing strengths. This uniform strategy overlooks the inherent differences among samples, thereby limiting the efficacy of knowledge transfer. To address this issue, we propose a unified and synergistic sample-wise adaptive knowledge distillation framework, termed ATLS, which leverages softmax entropy as a unified signal to jointly guide these two key hyperparameters. Specifically, for “easy samples” where the teacher model exhibits high confidence, ATLS automatically lowers the temperature and smoothing strength to preserve their discriminative information; for “hard samples” where the teacher is uncertain, it correspondingly increases these hyperparameters to explore richer inter-class knowledge structures. This design allows ATLS to serve as a simple, plug-and-play module that seamlessly integrates into existing KD frameworks, delivering substantial performance gains with minimal computational overhead. Extensive experiments on CIFAR-100 are conducted, with key results reported as mean ± standard deviation over multiple independent runs, validating the effectiveness and statistical reliability of our method.