The Same Circuits, in {-1,0,+1}: A Mechanistic Circuit Anatomy of a Deployed 1.58-bit Language Model

Ternary large language models—weights restricted to {-1,0,+1}, ≈ 1.58 bits per parameter—now match full-precision (FP) models of the same size at a fraction of the memory, and open-weight checkpoints such as microsoft/bitnet-b1.58-2B-4T are being deployed. Yet mechanistic interpretability had reached ternary networks only at toy scale—small from-scratch transformers on modular addition—and it was not obvious that the standard toolkit (sparse autoencoders, attribution patching, integrated-gradient circuits) even applies to a deployed model when weights are {-1,0,+1} and activations pass through round(). We show that it does—establishing the deployed ternary LM as a tractable model organism that the standard mechanistic toolkit can read—and that the resulting structure is conventional. First, the straight-through estimator (STE) used to train BitNet makes gradient-based attribution principled on the deployed model: the inference forward carries the training-time backward, so ∂ L/∂(inputs) is finite and 99.9% non-zero (as measured in the pilot run) despite round(·) having zero gradient almost everywhere, integrated-gradient completeness is 1.01±0.11, and EAP-IG matches exact activation patching (Spearman 0.70–0.86). Second, we recover, to our knowledge, the first full causal circuit of a deployed ternary model: an indirect-object-identification (IOI) copy+suppress circuit read out by a sparse set of late attention heads—positive name-movers L22H19 and L27H13 that copy the indirect object, and an S-inhibition head L22H17 that suppresses the subject—matching the canonical FP IOI decomposition, now in 1.58-bit and hardened over three templates. Third, at the feature level—measured on a matched from-scratch toy twin, since the deployed 2B has no FP twin to diff against, and corroborated on the 2B by a cross-model crosscoder—ternarization shares the FP feature inventory along rotated directions, with no private ternary vocabulary. Ternarization changes the parameters, not the algorithm. We report the heterogeneous and negative results (the read-out is not a uniform name-mover population; "zero exclusive features" required a positive control to be meaningful) as part of the contribution.