What question did this study set out to answer?

The aim is to replicate findings on correlated errors among LLM judges and examine the efficacy of a new measurement instrument.

August 3, 2026Open Access

Six Nameplates, Two Effective Votes: A Pre-Registered Local Replication of Correlated LLM Judge Errors, with a Deterministic Instrument for Measuring Panel Independence

View Full Paper

Ask AI

Bookmark

The aim is to replicate findings on correlated errors among LLM judges and examine the efficacy of a new measurement instrument.
Locally replicated using six open-weight judges with parameters ranging from 8B to 35B, on 1,000 ChaosNLI-MNLI items.
Measured judge correlation using a deterministic instrument that applies a Generalised Covariance Measure over input features.
Pre-registered analysis confirmed four predictions related to effective voting and error correlation.
Effective votes were 2.13 for six judges, matching the expected trend from previous research.
Error correlation observed between judges sharing base weights, with one pair producing identical results on all items.
Human-annotation entropy accounted for 13.6% of pairwise error correlation, while raw-text features had negligible impact.

No takes yet. Share an insight, caveat, or question.

Randomized trial confirms effective vote reduction due to correlated errors in LLM judges, suggesting new measurement tools are effective.

David Antonio Tomé (2026) studied this question.

View Full Paper

Ask AI

Bookmark

Randomized trial confirms effective vote reduction due to correlated errors in LLM judges, suggesting new measurement tools are effective.