Answers are grounded in this paper and cite their sources.
While large language models (LLMs) are increasingly deployed as decision-makers, current evaluation practice emphasizes benchmark accuracy rather than the structural properties that make decisions coherent and interpretable. We introduce a measurement-theoretic framework for evaluating whether the choices of an LLM are transitively rationalizable, focusing on the choice axiom of transitivity of preference. When transitivity is violated over a choice set, alternatives can no longer be coherently ordered as globally better or worse, and a stated preference between one pair constrains nothing about how the model would choose between any others. We apply our framework across 20 distinct LLMs, systematically varying question format, generation temperature, and contextual memory, using classic choice paradigms previously shown to induce intransitive preferences in humans. We find substantial and systematic transitivity violations in many models, although some patterns partially mirror known features of human decision behavior. Testing the transitivity axiom thus offers a principled, complementary foundation for evaluating the consistency of AI decision making beyond benchmark accuracy.
No takes yet. Share an insight, caveat, or question.
Song et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: