Open-set encrypted traffic recognition via contrastively-optimized multi-modal gating and extreme value theory

The widespread adoption of cryptographic protocols has rendered traditional payload inspection ineffective, necessitating advanced encrypted traffic analysis. Identifying unseen zero-day threats within encrypted streams, defined as the open-set problem, remains challenging due to adversarial feature manipulation, metric distortion induced by traditional cross-entropy optimization, and the resulting overconfidence of closed-set classifiers. In this paper, we propose Contrastively-Optimized Multi-Modal Gating with Extreme Value Theory (COMG-EVT), an open-set encrypted traffic recognition framework designed to establish strict rejection boundaries for unknown attacks. The framework integrates three core mechanisms: a dynamic GatedFusion module that leverages stable microscopic payload entropy to modulate vulnerable macroscopic topological graphs, mitigating shortcut learning against obfuscation; a two-stage graph contrastive optimization paradigm that improves representation consistency and class-aware latent-space organization to reduce metric distortion; and a calibrated multi-gate EVT engine that models extreme tail distributions to mathematically bound the open space. We conduct extensive experiments on two contemporary datasets, including the cipher-agnostic CipherSpectrum dataset. The analytical results demonstrate that COMG-EVT outperforms state-of-the-art open-set methodologies, effectively minimizing the false positive rate for zero-day attacks while maintaining high classification accuracy for known traffic categories, thereby exhibiting robust resilience against dynamic open-space risks. Not applicable.