Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks

There is an increasing interest on accelerating neural networks for real-time. We study the student-teacher strategy, in which a small and fast network is trained with the auxiliary information learned from a large accurate teacher network. We propose to use conditional adversarial to learn the loss function to transfer knowledge from teacher to. The proposed method is particularly effective for relatively small networks. Moreover, experimental results show the effect of network when the modern networks are used as student. We empirically study the-off between inference time and classification accuracy, and provide on choosing a proper student network.