What type of study is this?

This is a Quantitative Study study.

October 10, 2025Open Access

Can Linear Probes Measure LLM Uncertainty?

Key Points

Using Bayesian statistics enhances uncertainty quantification in large language models, showing significant improvements.
The method leverages training multiple Bayesian linear models to accurately capture uncertainty levels.
Results from numerical experiments demonstrate better performance compared to traditional softmax score approaches.
The proposed framework identifies key distributional features to inform a more efficient uncertainty quantification scheme.

Abstract

Effective Uncertainty Quantification (UQ) represents a key aspect for reliable deployment of Large Language Models (LLMs) in automated decision-making and beyond. Yet, for LLM generation with multiple choice structure, the state-of-the-art in UQ is still dominated by the naive baseline given by the maximum softmax score. To address this shortcoming, we demonstrate that taking a principled approach via Bayesian statistics leads to improved performance despite leveraging the simplest possible model, namely linear regression. More precisely, we propose to train multiple Bayesian linear models, each predicting the output of a layer given the output of the previous one. Based on the obtained layer-level posterior distributions, we infer the global uncertainty level of the LLM by identifying a sparse combination of distributional features, leading to an efficient UQ scheme. Numerical experiments on various LLMs show consistent improvement over state-of-the-art baselines.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper