White-box hallucination detectors read a language model's internal states to flag hallucinated answers, and recent work reports AUROC above 98%. We re-evaluate two such detectors, MultiHaluDet and ICR Probe, on Mistral-7B and Llama-3.1-8B under one controlled protocol: shared folds, paired bootstrap confidence intervals, and pass/fail criteria fixed before each experiment. On HaluEval QA, answer length alone reaches 0.97 AUROC; when only answers of equal length are compared, MultiHaluDet falls from 0.985 to 0.805-0.815, and its six-classifier stacking stage adds nothing over averaging its deep models. When the models answer TriviaQA and Natural Questions themselves, MultiHaluDet scores below trained token log-probabilities in seven of eight comparisons and ties in the eighth, and ICR Probe scores below them in all eight. A logistic-regression probe on the full hidden state of the last answer token scores higher than MultiHaluDet in all twelve model-setting-label comparisons. Combined with token log-probabilities, it beats MultiHaluDet combined with the same log-probabilities in all eight realistic comparisons, with every confidence interval above zero. MultiHaluDet's training-seed variance (s.d. <=0.005) is far smaller than these gaps. We release our code and evaluation protocol.
No takes yet. Share an insight, caveat, or question.
Mohammed Abdul Jawad Ali Khan (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: