Combined Knowledge Distillation Framework: Breaking Down Knowledge Barriers

Knowledge distillation, one of the most prominent methods in model compression, has successfully balanced small model sizes and high performance. However, it has been observed that knowledge distillation predominantly focuses on acquiring knowledge concealed within the dataset and the external knowledge imparted by the teacher. In contrast, self-distillation concerns itself with the utilization of internal network knowledge. Neither approach fails to fully harness the potential of knowledge. Therefore, this paper introduces the combined knowledge c framework that combines knowledge distillation with self-distillation. Within this framework, we introduce multiple shallow classifiers, combined with an attention module, to exploit internal and external knowledge. To enhance the efficiency with which the network utilizes knowledge. Experimental results demonstrate that by comprehensively leveraging network knowledge, distillation effectiveness can be enhanced, resulting in further improvements in network accuracy. Additionally, we applied the framework to lightweight neural networks with group convolution, the framework continues to perform exceptionally well.