February 16, 2024Open Access

Humans or LLMs as the Judge? A Study on Judgement Biases

View Full Paper

Ask AI

Bookmark

Human and LLM judges show notable biases under various perturbations, impacting evaluation reliability.
Findings indicate significant vulnerabilities, suggesting even advanced judges are influenced by biases.
Analysis involves evaluating a novel framework across thousands of LLM and human judgments with a curated dataset of 142 samples on Bloom's Taxonomy standards, highlighting inherent biases in both judges' responses and evaluations of LLMs. This study emphasizes the urgent need for robust assessment frameworks to improve judgment accuracy and fairness in evaluations.

Discussion

No takes yet. Share an insight, caveat, or question.

Randomized trial reveals biases in evaluations by human and LLM judges, indicating vulnerabilities in assessing LLM performance.

Chen et al. (2024) studied this question.

View Full Paper

Ask AI

Bookmark