Answers are grounded in this paper and cite their sources.
Background: Large language models (LLMs) are increasingly used as automated evaluators, but concerns remain about systematic biases such as self-preference.Prior findings may be confounded by stylistic similarity, shared quality preferences, or presentation order rather than genuine authorshipbased bias.Objective: We test whether LLM judges favor their own generated texts in blind pairwise comparison beyond what shared quality preferences would predict, and we quantify position bias as a competing explanation.Methods: We conduct a controlled, blind, multilingual experiment.Five near-frontier LLMs from five developers generate short texts in five European languages across three text types, with five generations per cell (375 texts).Each model then blindly compares its own text against each other model's text, in both presentation orders, yielding 3,000 evaluations.Self-selection is benchmarked against crossmodel selection of the same text as a quality-controlled baseline, and we apply equivalence testing to bound the size of small effects.Results: Across 3,000 evaluations, a correctly specified conditional-logit model (Bradley-Terry candidate strengths plus a self-bias term, controlling for both candidate quality and presentation order) revealed a modest but robust overall own-text preference (odds ratio or OR=1.26; p<.001).This was concentrated in two of the five judges (Gemini OR=2.09,Mistral OR=3.20; both p<.001); one judge showed the reverse (GPT OR=0.43; p<.001), and two showed no significant effect (Claude, Qwen).Model-specific position biases were strong (up to φ=0.62) and constitute the single largest distortion.Conclusion: Evaluation reveals a genuine own-text preference beyond shared quality, alongside strong position bias; both must be actively mitigated in LLM-as-judge pipelines.We do not attribute the owntext preference specifically to authorship recognition rather than familiarity, which our design does not separate.
No takes yet. Share an insight, caveat, or question.
Brückner et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: