A novel decoupled knowledge distillation

Deep neural networks outperform their shallow counterparts, but they tend to add depth or width, leading to a surge in parameters and computations. It is well-established that excessively wide networks pose a heightened risk of overfitting, while overly deep networks demand a substantial computational burden. This paper introduces a novel narrow-deep ResNet architecture and applies the strategy of decoupled knowledge distillation. This innovation aims to enhance network depth while mitigating potential issues associated with excessive width. This involves establishing a trained teacher model to instruct the training of unaltered, wide, and narrow-deep ResNet(student) models, enabling students to learn from the teacher's output. In order to assess the efficiency of this proposed method, We conduct tests on the Cifar-100 and Pascal VOC datasets. Results indicate that the method in this paper empowers a smaller model to achieve nearly identical accuracy as larger models, while significantly reducing inference time and computational effort.