This analysis demonstrates distinct token-level dynamics of certainty, hallucination, and refusal across language model families, indicating their reliability.
Key Points
The study aims to track epistemic dynamics during text generation in large language models to understand how certainty and hallucination manifest.
Introduced streaming epistemic geometry for token-by-token tracking in autoregressive generations.
Applied PCA-based subspace analysis on five independently trained model families.
Used a logistic classifier trained on first-token projection scores for evaluation.
Distinct dynamic signatures for hallucination, refusal, and certainty were identified from the first token.
Achieved an AUC of 0.991 for the logistic classifier on Llama-3.1-8B with successful zero-shot transfer to TruthfulQA.