VQ-Transformer: Lightweight IoT intrusion detection enhanced by optimal transport knowledge distillation

The proliferation of resource-constrained Internet of Things (IoT) devices has heightened their vulnerability to sophisticated cyberattacks, underscoring the need for intrusion detection systems (IDS) that achieve both high accuracy and computational efficiency. This paper proposes VQ-Transformer, a lightweight framework that employs token-level knowledge distillation from a high-performance teacher model to a compact student model. The teacher model enhances a Transformer encoder with vector quantization (VQ) and Top-K pooling to generate concise and discriminative sequential representations. The student model, consisting solely of an embedding layer, Global Average Pooling (GAP), and two dense layers, is trained by matching its latent token distributions with those of the teacher using an entropy-regularized Sinkhorn distance. Extensive experiments on the CIC-IDS2017 and ToN-IoT datasets demonstrate that both the teacher and the distilled student achieve accuracy, precision, recall, and F1-scores exceeding 99%, outperforming state-of-the-art lightweight models and knowledge distillation baselines by statistically significant margins.