What question did this study set out to answer?

This study aims to replicate findings on correlated errors among LLM judges and measure panel independence using a deterministic method.

August 3, 2026Open Access

Six Nameplates, Two Effective Votes: A Pre-Registered Local Replication of Correlated LLM Judge Errors, with a Deterministic Instrument for Measuring Panel Independence

View Full Paper

Ask AI

Bookmark

This study aims to replicate findings on correlated errors among LLM judges and measure panel independence using a deterministic method.
Pre-registered replication using six open-weight judges (8B to 35B parameters) on consumer hardware.
Analysis involved 1,000 ChaosNLI-MNLI items and a deterministic instrument for measuring corroboration weights.
Four pre-registered predictions were tested and confirmed through a cross-fitted generalised covariance measure.
Effective votes recorded were 2.13 for six judges compared to 2.18 for nine judges in previous research.
Two judge pairs, sharing base weights, exhibited high error correlation, achieving identical labels on all items and receiving a corroboration weight of 0.
Mean pairwise error correlation was minimally explained by raw-text difficulty features (0%) but showed 13.6% correlation explained by human-annotation entropy.

No takes yet. Share an insight, caveat, or question.

Randomized trial examines correlated errors in large language model judges, suggesting implications for panel independence.

David Antonio Tomé (2026) studied this question.

View Full Paper

Ask AI

Bookmark