Cognometry v0. 5 — extends v0 with two additional calibrated instruments (refusal, tool-call drift) and a phase-transition ablation result shipped in the same release window (2026-04-23). Three calibrated cognometric instruments: - Hallucination: 0. 998 AUC on HaluEval-QA, 8-benchmark cross-validated (v4) - Refusal: 0. 976 AUC on XSTest-v2 GPT-4 held-out, 18-feature LR trained on Llama-3. 2-1B apologetic refusals; empirical cross-substrate universality confirmation (v5. 1) - Tool-call drift: 0. 916 AUC on BFCL v3 5-fold CV, 22-feature LR, text-only. Beats the only published comparable baseline (Healy et al. 2026, AUC 0. 72 on Glaive, hidden-state MLP) while being black-box compatible (v6. 0) Phase-transition addendum: failure-class detectability does NOT rise smoothly with feature count. Each drift class has a critical feature that flips it from chance to near-perfect. argdrop 0. 50 -> 0. 998 at K=2 (+argcountᵦscore) ; argₛwap 0. 48 -> 0. 69 at K=6 (+typeₘismatchfrac) ; irrelevancecalled 0. 83 -> 0. 96 at K=10 (+promptcoverage). Inverse of emergent-capabilities literature for generative LLMs. Software: pip install styxx==6. 0. 0 Code: https: //github. com/fathom-lab/styxx (MIT on code, CC-BY-4. 0 on weights) Reproducers: 3 min CPU each, no API, no LLM inference Playgrounds: https: //fathom. darkflobi. com/cognometry (Pyodide, no install) OSF bundle: https: //osf. io/6syq4/ Tests: 655 passing, every AUC number backed by a committed script
Rodabaugh et al. (Fri,) studied this question.