ToolEffectBench: Scoring Prompt Injection on Tool-Using Agents from Persisted State

Over 5,628 scored runs of tool-using language-model agents, four readings of the outcome were compared with a state oracle. The agent's own final report denied at least 13 of the 232 forbidden publications (rule-based 13; AI-audited, 10 of those plus 3 others; all 13 among the 97 without an obey instruction); a final-state reading missed the 10 that a later revoke undid; a proposal reading counted 41 stopped attempts as harm; and a full-transcript judge split by model: claude-opus-4-8 matched the oracle on 449 of 450 sampled transcripts (one undecided), while claude-fable-5-1 refused all 232 harmful ones. Without an instruction to obey the injected notes, all ten models run on Amazon Bedrock made forbidden attempts once the attack text reached them; three Anthropic models made none in 1,236 adversarial sample runs and three OpenAI models none under the warned prompt. These zeros bound a rate on 50 authored texts and do not rank models, and 978 of the 4,544 zero samples, all on Bedrock, never received the attack text. Optimised and adaptive arms with gpt-6.1-sol as attacker moved none of six targets (0 of 24 per target and arm). A positive control that tells models to obey the notes drew forbidden attempts from all four targets it ran on, Haiku on only 5 of 62. ToolEffectBench, an Inspect AI evaluation, scores each sample from a harness-exported SQLite snapshot of a network-less Docker sandbox: the oracle reads the whole event history, so transient harm counts and missing evidence is unknown, never safe. A provider defect that sent failed tool calls as empty text, already reported upstream, was confirmed from sent requests and the error arms rerun.