A deployment-efficient CNN-LSTM framework for real-time malicious traffic detection across heterogeneous, imbalanced, and encrypted network traffic

Modern enterprise networks, including those adopting Software-Defined Wide Area Network (SD-WAN) architectures, face increasing intrusion detection challenges due to encrypted, heterogeneous, and imbalanced traffic. This paper presents a deployment-aware and robustness-oriented intrusion detection framework motivated by the security requirements of enterprise WAN and SD-WAN deployment environments. The experimental evaluation uses three heterogeneous benchmark traffic domains that are relevant to enterprise WAN monitoring but are not SD-WAN-specific datasets, and that do not cover all operational conditions. The study evaluates a deployment-efficient hybrid CNN-LSTM model across three heterogeneous traffic domains, including laboratory-generated traffic (CICIDS-2017), highly imbalanced multi-attack traffic (UNSW-NB15), and real-world encrypted traffic (CERNET-2020, with a substantial proportion of TLS/SSL-encrypted flows). A lightweight Channel Attention mechanism is incorporated, intended to enhance feature discrimination. SD-WAN is treated strictly as an intended deployment context rather than a validated setting: no SD-WAN testbed or SD-WAN traffic was used. Extensive experiments demonstrate that the proposed framework achieves 98.1% accuracy, 97.6% F1-score, and 0.99 AUC on the encrypted CERNET dataset, and 99.2% accuracy on CICIDS-2017, achieving strong within-dataset detection performance and providing cross-domain transferability insights across three heterogeneous benchmark datasets. To evaluate cross-domain robustness, a shared-feature transfer setting is introduced between CICIDS-2017 and UNSW-NB15, where the model achieves over 80% accuracy without retraining, indicating partial cross-dataset transferability for this 22-feature aligned subset only. A constrained transferability probe on CERNET (10 shared features only) indicates non-trivial discriminative capability despite substantial feature-space heterogeneity. In addition, the framework is designed for CPU-only inference, with an amortized per-flow model-inference latency of 2.3 ms (batch size 64; traffic acquisition, feature extraction, and window construction excluded), which is not end-to-end detection latency. Overall, this work contributes a systematic robustness evaluation framework for intrusion detection in heterogeneous, encrypted, and imbalanced network environments, addressing the gap between laboratory research and practical enterprise WAN deployment. The CNN-LSTM with channel attention is an established combination; the contribution lies in the evaluation protocol rather than in a new architecture. Comparisons are limited to the baselines evaluated (DNN-LSTM-GRU, PSO-D-SEM, DRL-DQN, and classical machine learning; PSO-D-SEM and DRL-DQN on CERNET only); no Transformer or TCN baseline was included, and no superiority over current methods is claimed. All experiments address the binary task (benign versus malicious); fine-grained attack-family and attack-stage information, which supports threat attribution, response prioritization, and forensics, is not assessed and is lost under this formulation.