A general self-knowledge distillation framework with noise injection

Knowledge distillation transfers knowledge from large teacher models to compact students, but training teachers is costly. Self-distillation removes this requirement by enabling models to learn from themselves. However, existing approaches still rely on auxiliary networks or cached models, introducing extra overhead and limiting scalability. We propose a lightweight and efficient self-distillation framework named Adaptive Noise-Based Self-Distillation (ANSD) to alleviate the substantial training overhead of existing methods. It constructs an additional student view via noise injection, forming a teacher–student paradigm with the original view. However, excessive noise may corrupt semantic information, while insufficient noise fails to ensure view diversity. To address this, we introduce an adaptive noise mechanism that adjusts the perturbation strength to balance view diversity and semantic fidelity. This design constructs a student view that is both distinct and semantically consistent, enabling more effective knowledge transfer. During training, we adopt a two-level distillation strategy that enforces distribution alignment at both the logit and feature levels, thereby exploiting complementary information across semantic levels. We evaluate our method on CIFAR-10, CIFAR-100, and ImageNet across diverse architectures, achieving consistent gains of up to 2.96% accuracy. Extensive visual analyses (CAM and t-SNE) demonstrate improved semantic extraction ability and enhanced inter-class separability. Moreover, ANSD is compatible with existing distillation methods and can further improve the performance boundaries of knowledge distillation.