June 18, 2024Open Access

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

View Full Paper

Ask AI

Bookmark

Excellent human alignment seen in Llama-3 70B and GPT-4 Turbo, but they ranked lower than JudgeLM-7B.
Cohen's kappa metric indicates that high percent agreement doesn't ensure score similarity among models.
Various judge prompts and model families were assessed for alignment, offering insights on biases in evaluations with LLMs. Additionally, error analysis was employed to explore factors like instruction length and leniency bias.

Discussion

No takes yet. Share an insight, caveat, or question.

Comprehensive study evaluates LLMs aligning with human judgment, revealing significant biases and performance variances.

Thakur et al. (2024) studied this question.

View Full Paper

Ask AI

Bookmark