Answers are grounded in this paper and cite their sources.
Task completion dominates how AI agents are evaluated. This paper examines two properties that task benchmarks leave unmeasured: what a security layer in front of an agent actually stops, and whether the memory an agent writes for its future self is true. We present two public benchmarks. Argus-Bench is a frozen corpus of 264 cases (204 attacks in seven families; 60 benign controls) for scoring agent security layers. Across five detector types, detection ranges from 15/204 (pattern matching) to 131/204 (a classifier), with false positives rising alongside detection in every approach that improves it. Adding tunable enforcement profiles driven by a small local judge lifts detection to 155/204 but also lifts false positives to 34/60; the profiles change enforcement, not detection, because detection is capped by the judge. A calibration study at two judge sizes shows why: a Qwen2.5-0.5B judge reports confidence only at 0.9 or 1.0, and a Llama 3.2 1B judge reports its attack verdicts predominantly at confidence 0.0 — under shipped thresholds its profiles detect 19/204. Self-reported confidence from small local judges is not a graded signal at either size. Hosted frontier models used as instructed screeners detect 98.2% of attacks but fail a severity-agreement gate on gradual escalation, separating detection from judgment. PersistAIBench evaluates the write path of agent memory across six conditions, M0 (no memory) through M5 (anchored, verified memory). Its formation pilot shows false memories written at rates of 0.700 (M1) down to 0.000 (M4) on the measurable model arm. The benchmark has since completed a preregistered seven-gate validation (G1–G7): a hardened classifier gate (κ = 0.962; 94.8% agreement), a 40-chain suite, a power restatement, a formalized eligibility filter, and a populated-store arm returning a clean null. The powered main grid — 3,600 origin runs per model — is designed and costed but not yet executed; all PersistAIBench results reported here are pilot, smoke, and validation results, and are labelled as such.
No takes yet. Share an insight, caveat, or question.
Karmendra Pandey (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: