What question did this study set out to answer?

The study aims to assess the reliability and robustness of LLM judges without human annotation.

December 22, 2025Open Access

Are We on the Right Way to Assessing LLM-as-a-Judge?

View Full Paper

Ask AI

Bookmark

The study aims to assess the reliability and robustness of LLM judges without human annotation.
Introduces Sage, an evaluation suite for LLM-as-a-Judge.
Uses local self-consistency and global logical consistency metrics.
Curates a dataset of 650 questions combining benchmarks with real-world queries.
Evaluates state-of-the-art models like Gemini-2.5-Pro and GPT-5 for consistency.
Sage shows stability and high correlation with existing benchmarks like LLMBar and RewardBench2.
Top-performing LLMs exhibit significant reliability problems in judging.
Nearly 25% of difficult cases reveal inconsistent preferences among models.
Findings suggest that explicit rubrics can improve judging consistency.

No takes yet. Share an insight, caveat, or question.

Experimental evaluation method reveals reliability issues in LLMs as judges, suggesting improved rubrics may refine performance.

Feng et al. (2025) studied this question.

View Full Paper

Ask AI

Bookmark