What type of study is this?

This is a Quantitative Study study.

October 20, 2025Open Access

Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores

Puntos clave

Model consistency serves as a target metric for evaluating large language models effectively and cheaply.
The correlation between model consistency and human-generated Elo scores reaches 91%, confirming its predictive power.
This approach eliminates the need for human input in model evaluations, thus reducing associated costs.
The findings suggest that relying on model consistency can improve the efficiency of LLM assessments over traditional methods.

Resumen

New large language models (LLMs) are being released every day. Some perform significantly better or worse than expected given their parameter count. Therefore, there is a need for a method to independently evaluate models. The current best way to evaluate a model is to measure its Elo score by comparing it to other models in a series of contests - an expensive operation since humans are ideally required to compare LLM outputs. We observe that when an LLM is asked to judge such contests, the consistency with which it selects a model as the best in a matchup produces a metric that is 91% correlated with its own human-produced Elo score. This provides a simple proxy for Elo scores that can be computed cheaply, without any human data or prior knowledge.

Leer artículo completoexternamente

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo