What type of study is this?

This is a Quantitative Study study.

September 30, 2025Open Access

The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks

Key Points

Results show small LLMs exhibit 50%-80% answer consistency across multiple trials, indicating variability among models.
Among repeated questions in benchmarks MMLU-Redux and MedQA, accuracy correlates with consistent answers in small LLMs.
The study compares small and medium models, revealing that medium LLMs tend to have higher consistency rates than smaller counterparts.
A range of parameters, including inference temperatures and model types, significantly influence answer consistency and performance.

Abstract

This work explores the consistency of small LLMs (2B-8B parameters) in answering multiple times the same question. We present a study on known, open-source LLMs responding to 10 repetitions of questions from the multiple-choice benchmarks MMLU-Redux and MedQA, considering different inference temperatures, small vs. medium models (50B-80B), finetuned vs. base models, and other parameters. We also look into the effects of requiring multi-trial answer consistency on accuracy and the trade-offs involved in deciding which model best provides both of them. To support those studies, we propose some new analytical and graphical tools. Results show that the number of questions which can be answered consistently vary considerably among models but are typically in the 50%-80% range for small models at low inference temperatures. Also, accuracy among consistent answers seems to reasonably correlate with overall accuracy. Results for medium-sized models seem to indicate much higher levels of answer consistency.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper