Modern enterprise networks generate high-volume, high-velocity traffic, making timely and accurate intrusion detection increasingly challenging for traditional batch-oriented Intrusion Detection Systems (IDS). These systems often struggle to detect sophisticated and evolving cyber threats due to delayed analysis, limited scalability, and the inability to provide continuous monitoring under dynamic workloads. This paper presents a real-time Network Intrusion Detection framework built on a distributed stream-processing architecture using Zeek, Apache Kafka, and PySpark. Network packets are captured and transformed into structured logs using Zeek, streamed reliably through Kafka, and processed in real time using PySpark for feature extraction, rule-based analysis, and machine learning–based classification. A Random Forest model is employed to identify both benign and malicious traffic across multiple attack categories, while Elasticsearch and Grafana provide low-latency storage, visualization, and alerting capabilities for operational security monitoring. The proposed system enables end-to-end real-time detection with high throughput and low latency, helping bridge the gap between rapid attack execution and timely threat response. Experimental results demonstrate the feasibility of scalable prototype IDS deployment for continuous monitoring, rapid analytics, and actionable security insights, while indicating strong promise for broader deployment with further real-world validation under diverse operational conditions.
Shreya et al. (Tue,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: