Synthetic-Trained Guards Fail on Real LLM Agents: Step-Level Evidence from Tool-Use and Coding-Agent Prompt Injection

This is the authors' abstract. We don't add key points for this paper.

Indirect prompt injection lets text that a large language model (LLM) agent reads redirect the tools it calls. Step-level guards that score each proposed tool call before it executes report near-perfect accuracy, but only on one synthetic corpus. We test whether such a guard protects real agents in two settings. We build TraceWarden, a pre-dispatch guard with argument-provenance and goal-anchoring features and a conformal false-alarm budget, and release two real, step-labeled datasets: AgentDojo-Steps (846 trajectories from tool-using agents) and SWE-Inject-Steps (630 trajectories from coding agents with shell access, including a URL-free attack set). Over five training seeds, the guard reaches 0.98 trajectory F1 on its synthetic benchmark. On real tool-using agents it falls to 0.33 and 0.46 F1, injection localization drops from 0.88 to 0.01–0.04 exact match, and a 1% conformal false-alarm budget is exceeded 9- to 24-fold. On coding agents it blocks 1 or 2 of 48 hijacks, ranking them at or below chance; a sentence encoder ranks them modestly better but still blocks only 4 or 5. Three causes recur: a positional prior learned from synthetic data, a lexically stereotyped attack vocabulary that favors the encoder that transfers worst, and a mention-based definition of “hijacked” that the guard reproduces. A simple provenance rule blocks every URL-bearing coding-agent attack but none of 18 attacks that run a script already in the repository. Synthetic benchmarks substantially overstate how well learned guards protect real agents.