Autonomous Pentesting using Artificial Intelligence: From the Cybersecurity Point-of-View

Key Points

• This work aims to evaluate and compare traditional and AI-driven penetration testing tools in terms of effectiveness and applicability.
• Conducted a comparative analysis of traditional and AI-driven pentesting tools using the MITRE ATT&CK framework.
• Evaluated reinforcement learning simulators and emulators for their effectiveness in cybersecurity testing.
• Identified limitations in current reinforcement learning tools concerning emulation and scalability.
• Proposed a hybrid simulation-emulation environment to enhance training realism for AI agents.
• None of the simulators led in all evaluation areas of attack coverage and scalability.
• NASimEmu, Caldera, and Cyberwheel exhibited the highest environment fidelity.
• Caldera, Cyberwheel, and CybORG demonstrated the broadest coverage of the MITRE ATT&CK framework.
• NASim, MulVAL, and CyberBattleSim showed the best scalability with lightweight designs.