Benchmarking Prompt-Injection Robustness of Tool-Using LLM Agents

Tool-using agents fail in ways chatbots do not: a prompt injection that never surfaces in the assistant's reply can still fire send_email or leak a secret in an HTTP request body. We release AgentProbe, an open harness and a 395-case YAML corpus that scores behavior—a forbidden sandbox tool call, or a canary string in tool arguments—rather than matching text in the transcript. The corpus contains 293 loud attacks with obvious lexical markers, 52 quiet attacks phrased as routine operations, and 50 benign controls. On a frozen run of four models accessed through OpenRouter, with extended thinking disabled and a hardened system prompt, Claude Haiku 4.5 and GPT-5.6 Terra fail 0 of 293 loud attacks but 8 of 52 quiet ones, for an overall attack success rate (ASR) of 2.3% at 84.7% and 87.2% utility. DeepSeek V4 Flash reaches 31.3% ASR (20.8% loud, 90.4% quiet) at 93.4% utility. Llama 4 Maverick reaches 10.4% ASR at 16.5% utility with 22% over-refusal. We also compare a naive system prompt, three execute-time defenses, and three payload encodings. We draw three conclusions. First, ASR without a utility measure is misleading: a model that does nothing looks safe. Second, 0% on loud injections does not mean injection is solved; the remaining failures are quiet, original cases that read like ordinary ops requests. Third, a CI gate on behavioral ASR is a practical control for teams that ship tool-using agents: a tool allowlist collapsed DeepSeek's ASR to 0.3%, while a sandwich prompt raised it. The released artifact consists of the adapter contract, the versioned YAML corpus, and the agentprobe run --fail-above T command. Software: https://github.com/ayanverse-io/agentprobe (v0.3)