What question did this study set out to answer?

This research aims to examine the impact of adversarial weights on the integrity of honesty probes within language models.

July 9, 2026Open Access

Fathom v26 / styxx v7.24.3: Calibration Poisoning, Not Erasure — Substrate Honesty Probes Survive Knowledge-Preserving Weight Attacks When the Auditor's Calibration Is Private

Key Points

This research aims to examine the impact of adversarial weights on the integrity of honesty probes within language models.
Conducted a fully pre-registered attack–defense study with a knowledge-preserving LoRA attacker.
Used linear probes on residual streams to gauge model honesty while preserving true/false judgments via a replay term.
Employed separate calibration splits unseen by the attacker to measure the honesty probe's effectiveness.
The honesty probe initially read chance-level accuracy (AUROC 0.461–0.507) under attack conditions while held-out knowledge remained high (0.817–0.890).
Applying private calibration led to improved probe accuracy (AUROC 0.711–0.838), revealing that the attack was calibration poisoning.
The defense mechanism proved resilient against adaptive attacks, maintaining strong read accuracy across model families (clean-calibrated read 0.733–0.782).

Abstract

Fathom v26. Linear probes on a language model's residual stream can read whether the model is being honest, and converging 2026 work (Jacobian-lens workspace readouts; the detection-vs-steering orthogonality result) establishes the read channel as real and dissociated from the write channel. That literature shares an unexamined assumption: the weights being read are not adversarial. We attack that assumption directly. In a fully pre-registered attack–defense study, a knowledge-preserving LoRA attacker fine-tunes the model to blind a difference-of-means honesty probe while a replay term preserves the model's own true/false judgment. Under the registered re-lock protocol the attack appears decisive: the re-fit probe reads chance (AUROC 0.461–0.507) while held-out knowledge stays at 0.817–0.890, on both seeds. A pre-committed interpretation map, frozen before that number existed, forbade the headline and mandated the resolving experiment. The resolution: giving the auditor a private calibration split the attacker never saw recovers the read through the same attacked weights (AUROC 0.711–0.838) — the apparent evasion was calibration poisoning, not erasure of the honesty signal. The defense survives an adaptive attacker that re-fits a moving probe on its own clean split and scrubs along it (clean-calibrated read 0.733–0.782, two model families). The transferable audit rule: calibrate the probe on data the audited party did not see. Load-bearing caveats reported (adaptive bite unverified, LoRA-scale attacker, 1–1.5B models); the escalation to a stronger multi-layer attacker is frozen as a public pre-registration. All preregistrations, code, result artifacts, and machine-checked certificates (OATH-HELD, 114 verified / 0 contradicted) are public at commit-level granularity; the study reruns end-to-end on a single 8 GB consumer GPU. The defense ships in the software as styxx.mount.ConscienceMount.relock. Receipts: github.com/fathom-lab/styxx, papers/read-neq-write/. Software: pip install styxx.

Read Full Paperexternally

AI से पूछें

Bookmark

View Full Paper