Answers are grounded in this paper and cite their sources.
Preprint. Many agent benchmarks now test what a system does when an instruction leaves something out or admits more than one reading, and each defines the problem in its own way. This evidence map codes 215 benchmarks and datasets dated 2022 to 2026, located by searches frozen on 19 November 2026, on how they define underspecification, construct it, label it, let the agent respond, simulate the user, score the outcome, and release the data. Most are recent: 93% are dated 2025 or 2026. 66% state a definition of the problem they test. 67% create the missing information by injecting it into existing tasks or by generating tasks, and 13% use natural or mixed origins. The agent may ask a question in 71%; where it asks, a language model usually plays the user. Task success is the most common metric (75%), but 144 benchmarks also report a metric outside the common families, and only 95 state how many tasks are underspecified. Resolving the gap needs a visual or other non-text referent in 5 benchmarks according to their sources and in 14 when coder inferences are counted; it is not reported in 22. A data license is stated for 50%. Blind second coding of a seeded sample agreed with the initial coding on 90% of origin codes. The map describes the works located by these searches, not the field. AI-use disclosure: Claude (Anthropic) assisted with literature discovery, screening, data extraction, classification, analysis and manuscript preparation. The research methods and quality-control procedures are described in Section 4. The author reviewed the output.
No takes yet. Share an insight, caveat, or question.
Ankur Shrestha (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: