Contrastive Distillation With Regularized Knowledge for Deep Model Compression on Sensor-Based Human Activity Recognition

Deep learning (DL) approaches have been widely applied to sensor-based human activity recognition (HAR). However, existing approaches neglect to distinguish human activities which have similar sensory reading patterns, resulting in poor recognition accuracy on those activities. Moreover, these deep models often come with complex network architectures for performance improvement. Expensive computational cost and memory requirement hinder their feasibility to be deployed on resource-limited environments such as smartphones. As one of the most popular model compress techniques, knowledge distillation (KD) can be leveraged to boost the performance of a compact student with the knowledge from a cumbersome teacher. However, most of the existing KD works ignore the bias introduced by teacher's logits during distillation, leading to sub-optimal issue for student training. To address the above issues, we propose a novel Con trastive D istillation framework with R egularized K nowledge (ConDRK) for sensor-based human activity recognition. Particularly, we first utilize multiple intra-class samples to formulate a novel unbiased soft target as regularized knowledge which can significantly eliminate the bias introduced by teacher. Then, we propose a contrastive distillation scheme which employs the proposed unbiased soft target as positive pair and samples from other different classes as negative pairs to transfer the knowledge. Our proposed approach can not only reduce the intra-class variance but also maximize the inter-class distances, thus further enhancing the recognition performance of the compact student on similar activities. Extensive experimental results on two sensor-based HAR datasets clearly show that our proposed method can achieve superior and consistent performance over other state-of-the-art methods. The PyTorch implementation of our proposed method is available at https://github.com/xuqing88/ConDRK