Compiling Operating Policies into Executable Test Environments for Tool-Calling Agents

Tool-calling agents work under operating policies that a general benchmark cannot know. We ask whether executable environments compiled from a deployment's written policy can produce solvable tests that catch policy violations by such agents. Our pipeline calls a language model only at build time. The model writes a state model and one function per tool from the policy and the tool schemas, and we keep what survives execution. We then discover faults by perturbing the compiled world, compose them into scenarios, and emit a scenario only with a replay certificate showing that a reference repair sequence solves it. On the telecom domain of τ³-bench, the compiled environment implements all 43 tools in 2,337 lines. On an earlier development build, generated scenarios require the same set of repair calls, by tool name, as 105 of 113 benchmark tasks, a development-set result. We ran a 24-scenario compiled suite, which contains 19 distinct tests, on the 2,337-line build. An agent given the policy completes 73.8% of graded actions and an otherwise identical agent given one sentence completes 43.8%, a separation of +30.0 percentage points (95% CI [24.7, 35.4] over the 19 distinct scenarios), with the policy arm ahead on 16 of 19. In a pre-registered experiment we removed one of six rules at a time from the agent's policy (3 repeats, 63 valid scenario blocks). The six rules are a purposive sample, chosen partly because the full-policy agent usually passed their criteria. One of the 6 removals meets the pre-registered criterion, and the count is best read by group. Two rules were tied to enough distinct scenarios for this design to detect them, and the suite caught one of them, the APN-reset rule, clearly (a 34.6-point drop on its tied criteria, p = 0.008). Two were tied to only 3 and 4 distinct scenarios, too few to reach significance at any effect size, and their removal still produced large drops (25.0 and 26.9 points). For the last two, the agent largely performed the step without the rule, so little or no violation arose. A pre-registered follow-up on 14 new scenarios did not catch either of the two underpowered rules on its own. Pooled with the first run in a secondary analysis, removal of the payment rule meets the criterion (21.1 points, p = 0.016). We also find that 29.8% (telecom), 30.7% (retail) and 34.0% (airline) of benchmark tasks change verdict across identical runs, which bears on how any such test should be read.