Optimizing the Learning Process with Limited Computational Resources

The paper considers the problem of optimizing deep learning models for their use in real-time applications on resource-constrained devices. To solve this problem, it is proposed to use knowledge distillation, which consists in transferring knowledge from an accurate but cumbersome neural network model to a more compact one. Different architectures of this approach, their disadvantages and advantages are described. An analysis of the effectiveness of knowledge distillation is given, identifying important patterns to achieve the desired result using this approach. The implementation and results of application to CNN for visual classification are described.