Intrusion detection systems (IDS) are indispensable for defending networked systems, yet security analysts are often overwhelmed by the large volume of false alerts they generate. Despite numerous attempts to reduce false positives, even highly accurate IDSs can trigger excessive alerts due to concept drift, such as changes in benign behaviors. This flood of alerts presents two critical challenges: the burden of investigating a large number of alerts and the uncertainty of situational awareness when choosing defensive actions. Since false alerts are inevitable, analysts need to: i) continuously update IDSs to address concept drift, ii) prioritize informative alerts to support efficient investigations, and iii) respond quickly to potential threats under uncertainty. Therefore, security analysts consume a significant amount of time on the three tasks, highlighting the necessity of reducing time consumption. The papers included in this thesis are organized into two parts. The first part introduces an anomaly-based intrusion detection system using a transformer. Each field in a log line is mapped to a token using both field-based and global vocabularies. To reduce irrelevant noise, fields unrelated to attacks are excluded from the anomaly score computation. The model is trained in a self-supervised manner by masking tokens and attempting to infer the masked fields. An anomaly score is then computed from the inference error, and alerts are raised when the score exceeds a predefined threshold. Our simulations demonstrate that the bidirectional transformer with field-based vocabulary outperforms baseline LSTM approaches. Then, we explore an intrusion detection framework that integrates flow-level anomaly detection, flow-context-based anomaly detection, and concept drift detection. A generative model maps flows in the networked system into an anomaly detection-friendly latent space and identifies anomalous behavior in individual log lines, forming the basis of flow-level anomaly detection. To capture system-wide context, a graph neural network correlates each log line with others across time and hosts, producing flow-context-based anomaly scores. Through anomaly masking, comparing predictions with and without suspicious log lines, the framework highlights significant differences when malicious activities are present. The flow-level and flow-context scores are then combined into a final anomaly score. In parallel, once concept drift is detected by a second generative model, two sample selection strategies are employed to identify log lines for investigation, enabling analysts to update the model efficiently and maintain detection accuracy over time. The second part of this thesis models the networked system using a Markov framework to study operations under false alerts. We first employ likelihood ratio testing to detect intrusions and prioritize alerts that confirm attacker presence. By pruning attack-graph states to form multiple hypotheses and investigating alerts to maximally shift likelihood ratios, our approach achieves a faster mean time to detection. We then address the challenge of alert investigation for improved situational awareness by proposing two prioritization policies that rank alerts at each time step according to their informational value. To enhance practicality, we introduce methods that reduce computational complexity, and simulations show substantial gains in overall computational efficiency. Finally, we examine automated incident response that combines low-cost and high-cost defensive measures, including blocking and eviction. We introduce a decentralized belief-maintenance method that significantly reduces communication overhead, and extensive simulations demonstrate that this approach achieves low overall cost while remaining highly practical. Overall, the deep learning-based anomaly detection and Markov model-based operational strategies developed in this thesis provide practical support for security analysts by reducing false alerts, lowering the number of alerts to investigate, and enabling more effective defensive decision-making.
Yeongwoo Kim (Thu,) studied this question.