Automated APT Defense Using Reinforcement Learning and Attack Graph Risk-based Situation Awareness

Advanced Persistent Threats pose significant risks to communication and infrastructure systems. While both heuristic and reinforcement learning have been applied to address this challenge, current approaches rely on fixed assumptions or use simplistic state representations and reward functions, compromising adaptability and accuracy. In this paper, we propose a novel intelligent agent system that enhances security against APTs by leveraging MulVAL attack graphs and quantitative risk assessment. Our approach translates complex MulVAL attack graph states into a compact RL state representation, enabling efficient learning in dynamic network environments. We integrate quantitative attack graph-based risk assessment into the RL framework, employing a hybrid reward function that incorporates residual risk, risk reduction, control efficacy, and cost. This risk-based approach provides enhanced context and situation awareness to the RL agent, facilitating more informed long-term decision-making. Using both Q-learning and Proximal Policy Optimization algorithms, we train and evaluate our system on an emulated industrial control system environment under realistic APT attacks. Experimental results demonstrate the significant benefits of our risk-based approach in guiding RL agents. Compared to non-risk configurations, our method shows improved success rates in APT mitigation, reduced control costs, which imply better long-term strategic planning.