Adversarial red teaming and imbalance correction in heterogeneous NIDS: A class-specific adaptive GAN framework

Modern network infrastructures face a structural long-tail imbalance where malicious events are statistically negligible against benign traffic. This bias causes Machine Learning-based Network Intrusion Detection Systems (NIDS) to systematically overlook rare, high-severity intrusions. While Generative Adversarial Networks (GANs) offer a solution for minority class synthesis, standard monolithic architectures suffer from majority-class gradient dominance, inevitably leading to conditional mode collapse. To address this, we propose the Adaptive Manifold-Decoupled GAN (AMD-GAN), a novel framework that applies class-wise architectural decoupling to feature manifolds and dynamically adapts its optimization regime to each category’s cardinality. This decoupled design mitigates inter-class interference and empirically reduces critic memorization. Evaluated across three heterogeneous NIDS benchmarks (CIC-IDS2017, UNSW-NB15, Edge-IIoTset 2022), AMD-GAN achieves a mean global empirical Wasserstein distance of (across five independent seeds), confirming robust distributional fidelity. Under a Train-Synthetic-Test-Real (TSTR) protocol, balanced synthetic augmentation consistently improves Multiclass Macro F1-Scores. Specifically, in CIC-IDS2017, it yields a 10.1 pp absolute improvement (+45.2% Botnet, +22.9% Web Attack), demonstrating statistically significant superiority over interpolation methods like ADASYN and Borderline-SMOTE. An extensive ablation study isolates the adaptive engine as the causal driver of manifold recovery, while Distance to Closest Record (DCR) analysis confirms the absence of exact-match memorization. Furthermore, interpretability techniques (SHAP and LIME) validate that the generated flows preserve authentic protocol semantics, enabling unambiguous attribution for Security Operations Centers (SOC). Deployed offensively as an automated Red Teaming engine, AMD-GAN exposes structural fragility in operational tree ensembles, inducing up to 0.909 Macro F1-Score degradation under volumetric stress scenarios. Finally, the decoupled sequential training architecture restricts peak GPU VRAM to 1,575 MB while generating 90,000 synthetic flows in 15.7 s, establishing AMD-GAN as a computationally efficient, low-VRAM solution potentially suitable for resource-constrained deployments for robust data augmentation and adversarial NIDS auditing.