Answers are grounded in this paper and cite their sources.
The costliest failure for a shopping agent is buying something that does not fit the customer. We ran a pre-registered, sham-controlled study on our own sealed instrument shaped after Visa's published agentic-commerce benchmark: 15 sealed shopping episodes, five frontier models from four labs, three attempts, and five arms (bare agent, a structured-requirements instruction, a matched sham, an enforced variant of the instruction, and the enforced variant with a judge model in the loop). The pre-registered primary was falsified: the judge-in-the-loop arm passed fewer runs than the instruction alone (Δ −13.4 points, 95% CI −20.4 to −6.5, 216 paired cells), and the enforced variant, re-measured as a pre-registered replication, did not demonstrate an advantage over the instruction alone (Δ −1.4 points, 95% CI −6.9 to +4.2). A sensitivity read that was not pre-registered, scoring the faulted judge-arm runs as failures, gives −16.4 (CI −23.6 to −9.3). The pre-registered non-primary comparison found a large instruction effect: the structured-requirements instruction raised the pass rate from 31.9% to 85.6% (Δ +53.7 points, CI +46.3 to +60.6), while a sham of matched structure moved it +4.2 (CI −3.7 to +12.0). Purchases of an item outside the answer key fell from 27.3% of runs to 3.7%. Two limits bound every figure. Every customer in this study is a scripted simulator, so nothing here measures real shoppers. And our simulator is harsher than Visa's: on the one public episode our bare arm passed 1 of 15 runs against 8 of 15 in Visa's grid for the same model ids (two-sided Fisher p = 0.014). Our figures are therefore not comparable to Visa's, and every claim is a paired comparison inside one environment. The study does not measure performance on Visa's benchmark, real purchases, or memory across sessions. Preprint, version 1.0 (2026-11-10). The structured-requirements instruction and its enforced variant are patent pending. Analysis, review and drafting were performed by AI agent seats operated by the author; decisions and responsibility are the author's.
No takes yet. Share an insight, caveat, or question.
Tad MacPherson (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: