Optical-layer and control-plane attack detection in SDONs using an ensemble ML-based IDS: evaluation on OSDN and GÉANT telemetry datasets

Abstract Optical communication networks play a vital role in cloud computing, 5G and 6G backhaul, and hyperscale data centers. With the move toward Software-Defined Optical Networks (SDON) and OpenFlow-based control, the optical layer and control plane are exposed to cyber-physical attacks such as optical jamming, power denial of service (PDoS), wavelength spoofing, and malicious lightpath reconfiguration. Conventional IP-based Intrusion Detection Systems (IDSs) cannot monitor optical-layer parameters, such as Optical Signal-to-Noise Ratio (OSNR), Bit Error Rate (BER), optical power drift, and spectrum allocation behavior. This study presents an ensemble Intrusion Detection System (IDS) that combines decision tree, random forest, and XGBoost for detecting optical-layer and control-plane attacks. The ensemble IDS leverages optical telemetry (OSNR, BER, received power, Q-factor) and SDN control-plane logs in order to enhance anomaly detection. Evaluations are carried out on (1) the OSDN-2023 dataset – simulated optical SDN environments, and (2) the GÉANT pan-European optical backbone telemetry dataset. The ensemble model achieves 97.842 % accuracy and 0.9815 ROC-AUC on OSDN-2023 dataset and also achieves 99.996 % accuracy and 0.9998 ROC-AUC on GÉANT data, outperforming conventional ML models. Our findings demonstrate that optical-layer parameters (e.g., OSNR degradation and transient spikes in BER before attacks) improve detection and reduce false positives compared to IDS that solely rely on traffic.