August 16, 2024Open Access

Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions

View Full Paper

Ask AI

Bookmark

LLMs-as-a-judge show minimal improvement from detailed prompts, impacting evaluation outcomes.
In tests, human judgment alignment occurs more effectively via perplexity than prompts, especially for textual quality.
Analysis utilized reinforcement learning from human feedback to assess quality across various LLMs-as-a-judge models and prompt structures. A taxonomy of common quality criteria was created to benchmark evaluations of models as judges considering human preferences.

Discussion

No takes yet. Share an insight, caveat, or question.

Randomized trial reveals limited benefits of prompts on LLM evaluation accuracy, indicating quality measures may align better with human judgments.

Murugadoss et al. (2024) studied this question.

View Full Paper

Ask AI

Bookmark

Randomized trial reveals limited benefits of prompts on LLM evaluation accuracy, indicating quality measures may align better with human judgments.