Getting pwn’d by AI: Penetration Testing with Large Language Models

The field of software security testing, more specifically penetration, is an activity that requires high levels of expertise and involves manual testing and analysis steps. This paper explores the potential usage large-language models, such as GPT3. 5, to augment penetration testers with sparring partners. We explore the feasibility of supplementing penetration with AI models for two distinct use cases: high-level task planning for testing assignments and low-level vulnerability hunting within a virtual machine. For the latter, we implemented a closed-feedback between LLM-generated low-level actions with a vulnerable virtual machine (connected through SSH) and allowed the LLM to analyze the machine state for and suggest concrete attack vectors which were automatically within the virtual machine. We discuss promising initial results, avenues for improvement, and close deliberating on the ethics of AI-based sparring partners.