Answers are grounded in this paper and cite their sources.
Whether the hidden state probe can provide a false discovery rate (FDR) guarantee for the answer release of the small language model still lacks a cross-domain shift test. This article uses a multi-layer probe to identify the hallucinated answer, decides to release it with conformal selection, and tests it on HaluEval and TruthfulQA. The probe AUROC (area under the ROC curve) on HaluEval is 0.991, which is higher than 0.974 of the answer-length baseline; the FDR at the target level of 0.10 is 0.050, and the power is 0.992, which is higher than the baseline of 0.883. The exploratory AUROC with a single feature of punctuation at the end of the sentence reached 0.969, and the difference in the length and sentence style of HaluEval's answer contributed most of the AUROC. When cross-domain, AUROC dropped to 0.491, and FDR rose to 0.409, exceeding the target level; after recalibration, 0 answers were released. Exploratory in-domain retraining brings AUROC back to 0.904, FDR is 0.045, and power is 0.502. High AUROC cannot be directly interpreted as the model grasping the true and false information, and the probe must be retrained with the data of the target domain.
No takes yet. Share an insight, caveat, or question.
Si et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: