Answers are grounded in this paper and cite their sources.
Benchmark evaluation measures rank-based fidelity of distributed Jacobian-lens checkpoints across language models, providing baseline calibration metrics for mechanistic interpretability tools.
The Jacobian lens ("jlens"), released 2026-07-02 as companion code to the *Verbalizable Workspace* paper, ships with pre-fitted checkpoints for ~36–38 model families. To our knowledge — a bounded-absence claim; see §6 for the search scope — no direct lens-versus-model fidelity measurement of these distributed checkpoints (rank agreement, KL divergence, top-1 agreement, or calibration) has been published by any party, including the upstream authors. This note contributes six fidelity readings of three distributed checkpoints under two position-domain bindings, plus two measured floors (chance and vanilla logit-lens), all under a frozen protocol: a rank-based fidelity statistic (wgrd) preregistered before each run, evaluated on a fixed 266-item prose set, with a positive/negative-control calibration battery passed in the same invocation as every measurement, and md5-pinned instruments, inputs, and outputs throughout. The distributed OLMo-3-7B lens reads 1.39 dex (median word-group rank deviation from the model it explains), the OLMo-3-32B lens 1.57 dex, and the Qwen3-8B lens 2.44 dex on its own tokenizer scale; the chance floor sits at 4.47 dex and the vanilla logit-lens baseline at 1.84 dex on the 7B ground. All values are robust to a position-domain correction aligned with the lens's own fit domain (largest shift: 0.10 dex). We report these as instrument characterization only: they license no claim about the models, about the upstream fitting objective, or about any downstream hypothesis. Instrument ineligibility is not refutation.
No takes yet. Share an insight, caveat, or question.
Manabu Higashida (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: