PenQA: A Comprehensive Instructional Dataset for Enhancing Penetration Testing Capabilities in Language Models

Large language models’ domain-specific capabilities can be enhanced through specialized datasets, yet constructing comprehensive cybersecurity datasets remains challenging due to the field’s multidisciplinary nature. We present PenQA, a novel instructional dataset for penetration testing that integrates theoretical and practical knowledge. Leveraging authoritative sources like MITRE ATT&CK™ and Metasploit, we employ online large language models to generate approximately 50,000 question–answer pairs.We demonstrate PenQA’s efficacy by fine-tuning language models with fewer than 10 billion parameters. Evaluation metrics, including the BLEU, ROUGE, and BERTScore, show significant improvements in the models’ penetration testing capabilities. PenQA is designed to be compatible with various model architectures and updatable as new techniques emerge. This work has implications for automated penetration testing tools, cybersecurity education, and decision support systems. The PenQA dataset is available in our GitHub repository.