The Evolution of Knowledge Distillation in Image Classification Tasks

With the start of the digital age, image classification has been one of the first problems in computer vision, that is to say, to identify and label all pictures according to various categories. High-performance models are usually very expensive to run on computers, so they are not suitable for the resource-constrained environment of real-time deployment. One of the many methods to address the problem of model compression is knowledge distillation. The goal is to reduce the size and complexity of the "teacher model" by optimising the parameters of the "student model" so that it can achieve similar performance to the "teacher model", but with a lower cost and without modifying the "student model" architecture. In this paper, we aim to systematically review the technical methods of knowledge distillation in image classification and discuss how to maximise the learning efficiency of student models with different forms of knowledge transfer across various dimensions, such as model output and feature map matching, structural relationships between the teacher and student models, etc. The three parts of the study are as follows: Output-based distillation is how to separate knowledge distillation from the problem of determining the boundary of a relationship graph; Feature-based distillation is related to Wasserstein distance and feature attention transfer; and Relationship-based distillation is based on virtual distillation techniques and Lipschitz continuity constraints. Based on the above analysis, although there is a problem of one-dimensional transfer, complementary effects can be obtained through the organic combination of output, feature and relationship transfer. A multi-dimensional fusion method can improve the accuracy and generalisation of the small model so that it does not need to be implemented on the high-performance vision system of the edge computing platform.