Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing

Large language model (LLM) agents increasingly automate offensive-security tasks, but success on isolated challenges does not establish reliable long-horizon autonomy. We study this distinction in authorized capture-the-flag (CTF) and wargame environments and introduce a planner-directed architecture in which a persistent non-executing planner delegates on demand to transient, context-isolated specialists with retrievable skills, persistent execution, compact memory, guardrails, and configurable model escalation. On all 200 challenges of the NYU CTF Bench test split, the system obtains 41 accepted flags (20.5%) and 29 leak-audited genuine solves (14.5%), against 4 (2.0%) and 3 (1.5%) for a stateful single-agent baseline. Both start from gpt-4o-mini, but only the planner-directed system could escalate to gpt-5-mini, which it did on 199 of 200 challenges and for 72.5% of spend. A controlled architecture-by-escalation experiment over a pre-registered subset of 80 challenges, crossing a resource-matched monolithic agent with the planner under matched budgets, was run in two independent formal passes. M0 solves 4/80 then 3/80, M1 22/80 then 19/80, P0 4/80 then 3/80, and P1 14/80 then 15/80. Pooling the two formal passes with challenge as the repeated unit, a clustered GEE shows a strong escalation effect (OR 8.79, 95% CI 3.39−22.81, p = 8× 10⁻⁶ p = 8 × 10 - 6 ), no architecture main effect (OR 1.17, $$p = 0.34$$ p = 0.34 ), and a significant architecture-by-escalation interaction (OR 0.53, $$p = 0.005$$ p = 0.005 ). Under the spend-constrained 340-action ceiling, the coordination cost of specialist delegation, context restart, and evidence folding offsets any additional solve-rate advantage from the planner-directed structure. On OverTheWire Bandit, the planner-directed system reaches 30/33 standalone and 24/33 sequentially, compared with 27/33 and 23/33 for the strongest single-agent mini backbone.