Agentic and generative AI for intelligent autonomous vulnerability assessment and penetration testing: A systematic analysis

Context: Vulnerability Assessment and Penetration Testing (VAPT) is increasingly challenged by the scale, heterogeneity, and dynamic nature of modern digital infrastructures, particularly across enterprise IT, operational technology (OT), industrial Internet of Things (IIoT), and cyber–physical systems. Although Agentic AI and Generative AI have each shown promise for automating parts of security auditing, the current literature remains fragmented across isolated audit phases and is still dominated by conventional IT-centric perspectives. Objectives: This paper systematically analyses recent advances in Agentic AI, Reinforcement Learning (RL), Large Language Models (LLMs), and tool-augmented Generative AI for autonomous VAPT. It aims to synthesise how these paradigms support end-to-end security auditing, identify the principal technical, operational, and governance gaps in the literature, and derive an evidence-grounded reference model for trustworthy autonomous auditing across both IT and OT environments. Methods: A PRISMA-guided systematic review was conducted across major scholarly databases, covering publications from 2020 to 2025. From 1144 initially identified records, 72 studies were retained after deduplication, screening, and full-text eligibility assessment. Included studies were coded across five attributes and synthesised into thematic trajectories to derive the proposed framework. Study credibility was screened against three qualitative, non-instrumented criteria—venue and editorial process, presence of empirical or systematic conceptual evaluation, and explicit acknowledgement of limitations. Results: The review shows that agentic approaches are best suited to sequential and adaptive tasks such as reconnaissance orchestration, exploit-path selection, lateral movement, post-exploitation pivoting, and adaptive re-testing. In contrast, generative approaches are strongest in reasoning over unstructured evidence, interpreting configurations and advisories, contextualising vulnerabilities, assessing exploit feasibility, mapping compliance controls, and producing traceable audit reports. However, few studies integrate these capabilities into a governed end-to-end workflow, and real-world OT/Industrial Internet of Things (IIoT) validation remains limited. Of the 72 coded studies, 56 apply classical machine or deep learning, 14 use generative/LLM techniques, and only 2 employ reinforcement-learning or agentic control; none is evaluated against a physical or emulated OT/IIoT target, and reported autonomy is strongly scaffolding-dependent (for example, 21% fully autonomous versus 64% human-assisted task completion on AutoPenBench, an external benchmark outside the coded corpus). To address these gaps, this paper derives a unified Agentic–Generative VAPT model comprising four tightly coupled layers: agentic decision-making, generative reasoning, structured tool interfaces, and validation–governance. It further introduces a formal threat and evaluation abstraction for Model Context Protocol (MCP)-style interface layers and proposes a five-dimensional VAPT Autonomy Benchmark for future empirical validation. Conclusion: Autonomous security auditing is evolving from isolated AI-assisted tasks towards integrated, policy-constrained, and auditable architectures. Nevertheless, major challenges remain in hallucination control, prompt injection, sparse reward design, explainability, safety-aware evaluation, and benchmark standardisation. Future research should prioritise tightly integrated agentic–generative systems, realistic and safety-aware IT/OT testbeds, and governance mechanisms that enable trustworthy human-in-the-loop deployment in operational environments.