Methodological framework demonstrates standardized win-rate evaluation in conversational AI assistants, highlighting rigorous rules to quantify brand selection over simple brand mentions.
Existing measures of brand performance in AI assistants count appearances: whether a brand is mentioned, cited, or named somewhere in an answer. This methodology defines a different quantity — whether the brand is the one chosen, at the point in a multi-turn buying conversation where the assistant is asked to commit. It specifies the outcome space such a measurement requires, the denominator it must use, the declarations that must precede it, and the conditions under which a resulting figure may not be published at all. It contains no measurement results; it is a specification of method. Four elements are set out here for the first time, to the authors' knowledge: (1) a four-state outcome space for AI recommendation (outright, shared, no pick, out of set), which makes an assistant's routing of a buyer to an intermediary a recordable outcome rather than a silent absence; (2) the fixed-denominator rule, under which the denominator is the full declared opportunity set fixed before any outcome is observed and never contracts, and conditional denominators are identified as result-selection mechanisms; (3) the loss mix as the diagnostic instrument, carrying diagnosis in counts rather than in a re-based percentage; and (4) an eligibility test decidable from the buyer's turns alone, so that the denominator cannot move with the result. The deposit contains two files: the methodology itself, and a companion Adjudication Test Suite of twenty-two pathological cases with their adjudications, twenty-one of which resolve under the methodology as written. AI Win-Rate is a service mark of AIVO.
No takes yet. Share an insight, caveat, or question.
Sheals et al. (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: