Revisiting adversarial robustness in NIDS: a multi-dataset evaluation of test-time evasion attacks

Machine learning-based Network Intrusion Detection Systems (NIDS) remain highly vulnerable to test-time evasion (TTE) attacks, yet existing research relies heavily on legacy datasets and ignores the impact of evaluation scale. This study presents a systematic evaluation of TTE attacks using the Adversarial Robustness Toolbox (ART) across three modern, domain-diverse datasets: UNR-IDD, CICIOMT-2024, and UAV-IDS-2025. We evaluate a PyTorch-based fully connected neural network classifier and a 1D Convolutional Neural Network (CNN)-based classifier against white-box (AutoAttack, Auto-PGD, Carlini & Wagner) and black-box (HopSkipJump) threat models. Furthermore, we present a scalability analysis by evaluating performance across incremental test-set proportions from 10 to 100%. Results demonstrate severe susceptibility across models: despite achieving clean baseline accuracies up to 100%, Carlini& Wagner and HopSkipJump attacks reduce classification accuracy by nearly 80%. While 1D CNNs exhibit greater structural resilience than the PyTorch classifier, their adversarial robustness consistently deteriorates as the test set volume increases. These findings confirm that the evaluation scale and feature structure strongly influence adversarial performance, underscoring the urgent need for scalable evaluation standards and adaptive defenses in modern NIDS.