SnifferDog: Comprehensively Learning Heterogeneous Features of Network Traffic to Identify Malicious Flows

Deep learning has recently attracted significant attention in the field of network intrusion detection. Despite a substantial number of efforts have been made, previous works struggle to comprehensively learn the features of network traffic, resulting in inconsistent performance across various environments and attacks. To address these limitation, this study presents SnifferDog, a novel network attack detection system that takes raw packets as input and rationally extracts and integrates heterogeneous features involved in packets, flows and topology. It formats the packets and flows concurrently to achieve a high-level throughout for feature learning. Then, a flow pretraining model consisting of a LSTM, a self-attention and cross-attention layers is developed to learn both sequential and nonsequential inter packet relation features as initial flow vectors. Subsequently, a node-to-node and a node-to-edge attention layers are implemented to enhance an inductive GNN model that dynamically embeds the flow-to-flow and flow-to-topology relation features into the flow vectors. The resulting flow vectors involve comprehensive information of packet-to-packet, flow-to-flow and flow-to-topology relations, enabling high detection performance. In-lab experiments across eight datasets from diverse environments demonstrate SnifferDog’s superior effectiveness over existing solutions. A scalable prototype deployed in our institute’s network achieves a false positive rate of only 0.08%, validating SnifferDog’s practicality in real-world scenarios.