Computational study demonstrates internal precision-weighting signals reliably detect and control confabulations in large language models, suggesting internal states can govern generation.
This preprint investigates whether large language models contain a latent analog of the brain's precision-weighting mechanism — an internal estimate of the reliability of their own knowledge — and whether it can be read, acted upon, and preserved. Four findings are reported on Qwen2.5-1.5B-Instruct. (1) Detection: a linear ridge probe on a single hidden state matches 10-sample semantic entropy at detecting confabulation (AUROC 0.819 vs 0.820), calibrates to ECE 0.031, transfers frozen across datasets with no loss of discrimination, reads at chance from the embedding layer, and peaks at three-quarters of network depth — before generation begins. (2) Action: routed into policy, the signal converts 99.7% of would-be confident errors into abstentions at a 5% risk target, and governs when to retrieve: precision-routed retrieval outperforms random routing by 7.1 points at a fixed 50% retrieval budget. (3) Attention: routed into attention as a bias toward evidence, the signal reliably re-routes processing (p<0.0001) with an inverted-U dose-response, but yields no significant accuracy gain at this model scale, where memory/evidence arbitration is rarely broken; a falsifiable scale-conditional prediction is stated. (4) Training: preference-tuning (DPO/LoRA) degrades calibration rapidly and chronically past the point of reward gain; an annealing curriculum inspired by the REBUS model of psychedelic therapy fails to rescue calibration at both tested doses, with an integration-timed protocol identified as the open question. Code, trained probe weights, and per-item experimental artifacts are available in the linked repository.
No takes yet. Share an insight, caveat, or question.
Caravella Chiron (2026) studied this question.