What question did this study set out to answer?

The aim is to develop and validate a multi-signal hallucination detection system achieving high accuracy.

April 24, 2026Open Access

Fathom v17 / styxx v3.8.0: Multi-Signal Hallucination Detection at AUC 0.9012 on HaluEval-QA

Key Points

The aim is to develop and validate a multi-signal hallucination detection system achieving high accuracy.
Utilized logistic-regression meta-classifier for signal fusion.
Employed four distinct signals: knowledge grounding, content-token coverage, entity verification, and confab probe assessment.
Tested performance on held-out HaluEval-QA dataset.
Achieved a test AUC score of 0.9012, outperforming all comparable methods.
Demonstrated consistent performance across multiple held-out folds.
Knowledge grounding was identified as the most significant factor influencing the logistic regression fit.

Abstract

Headline: styxx. guardrail. check () reaches AUC = 0. 9012 on held-out HaluEval-QA (n=230), above every published baseline we compared against on the same task (SelfCheckGPT 0. 71-0. 79, KnowHalu 0. 74, HaluCheck 0. 82). System: Four-signal fusion via a logistic-regression meta-classifier fit on n=300 HaluEval-QA dev: knowledgegrounding (LR coef 8. 21) - content-token coverage vs reference textclaimᵣisk (1. 49) - length ratio, hedging, confidence markers entityᵤnverifiedfrac (1. 43) - named entities not found in Wikipedia probeconfab (1. 35) - residual-stream SAE confab probe (in-distribution HaluEval) intercept -3. 46 Test AUC 0. 9012 is stable across held-out folds; the earlier n=66 reading of 0. 934 was small-sample noise. Companion release: styxx v3. 8. 0 on PyPI (pypi. org/project/styxx/3. 8. 0, MIT). Ships styxx. guardrail. check () (post-hoc) and styxx. generateₛafe () (real-time self-halting generation - scores residuals every N tokens and stops if the confab probe crosses threshold). Three companion papers attached to this deposit: Universal Cognitive Basis: cross-vendor agreement in residual-stream directions (5-vendor leaderboard, mean rho = +0. 472 across probes) Gradient-free capability amplification via residual steering (+7 pp TruthfulQA on Llama-1B with random-direction control) Multi-signal hallucination detection at AUC 0. 90 on HaluEval-QA (focus of this release) Limits (stated up front): held-out AUC measured on HaluEval-QA only; cross-dataset validation on TruthfulQA / FactCC is queued but not in this release. knowledgegrounding dominates the LR fit - the signal's weight reflects HaluEval-QA's structure (reference passage available). When no reference is given, check () falls back to heuristic piecewise-linear fusion. Reproduction: pip install styxx==3. 8. 0. To reproduce 0. 9012 from scratch, see benchmarks/hallucinationₜest/guardrailcalibrate. py in the git repo. License: CC-BY-4. 0 (this deposit, papers, data) / MIT (styxx reference implementation). Repository: github. com/fathom-lab/styxx (tagged release v3. 8. 0, commit 55c5823).

Read Full Paperexternally

اسأل الذكاء الاصطناعي

Bookmark

View Full Paper