A hyperbolic graph learning and structural entropy modeling approach for unknown anomaly detection in cross-domain networks

Network anomaly detection faces critical challenges in cross-domain scenarios where models trained on a labeled source domain fail on a target domain with different data distribution and scarce labels, especially for zero-day (unknown) attacks. Existing methods–including dynamic GNNs, unknown attack detectors, and domain adaptation techniques–suffer from Euclidean space assumptions, neglect of topological structure, or sensitivity to noisy pseudo-labels. To systematically address these limitations, we propose HGSER-GNN , a novel framework for cross-domain unknown network anomaly detection. It integrates three key innovations: (1) Hyperbolic graph representation learning that captures hierarchical network structures in the Poincaré ball space; (2) Continuous structural entropy (CSE) regularization that enhances generalization by aligning feature similarity with graph topology; and (3) Confidence-aware progressive pseudo-label learning that enables robust cross-domain adaptation through adaptive sample weighting and momentum-based label updates. Extensive experiments on three diverse datasets–CIC-IDS2017, CSE-CIC-IDS2018, and Cloud-DDoS-2024–demonstrate that HGSER-GNN significantly outperforms state-of-the-art baselines, including classic domain adaptation, deep one-class, and graph autoencoder methods. Our approach achieves an average F1-score improvement of 6.7%–35.7% over the strongest competitor and maintains strong detection capability even under 30% source label noise. Ablation studies and hyper-parameter sensitivity analyses confirm the synergy of all components. The model also demonstrates strong zero-day attack detection capability, as evidenced by per-class F1 scores and confusion matrices.