What question did this study set out to answer?

This research examines how a specific expert in a mixture-of-experts model processes language using an examination register.

July 1, 2026Open Access

Localizing an Examination Register to a Single MoE Expert, and Why the Expert Index Does Not Transfer

Key Points

This research examines how a specific expert in a mixture-of-experts model processes language using an examination register.
Characterized Expert 114 at layer 14 as a linear detector of the examination register.
Conducted held-out, bottom-up, and cross-model tests to evaluate the expert's performance.
Released comprehensive scripts, prompts, and analysis tools under the MIT license.
Recovered router direction distinguishes examination-register generation from controls (Cohen's d = 3.88).
An auto-interpreter achieved AUC 0.94, extending the register's applicability beyond self-reference.
The detector's functionality is model-specific and does not transfer to larger models.

Abstract

Localizing an Examination Register to a Single MoE Expert, and Why the Expert Index Does Not Transfer. Preprint, version 1.2.1 (June 2026). Not peer reviewed. Mixture-of-experts (MoE) routing emits a discrete, per-token record of which experts fire, a signal unusually legible for interpretability, yet single experts are rarely tied to a specific functional role. We study a generated examination register: language in which the model treats a target--itself, another entity, a natural object, or an abstract subject--as something to be examined from an ongoing interpretive stance. In Qwen3.5-35B-A3B and the refusal-reduced HauhauCS-Aggressive fine-tune, we characterize one routed expert, Expert 114 at layer 14, as a linear detector of this register, and bound what it does. Across held-out, bottom-up, and cross-model tests we show that (1) its recovered router direction separates examination-register generations from lexically matched controls with no overlap (Cohen's d = 3.88); (2) a blind, prompt-independent auto-interpreter recovers the same register at AUC 0.94, broadening it beyond self-reference to abstract examination and philosophical-worldview language; (3) the detector is a readout with only weak, conditional control: residual injection induces the register, yet gate suppression does not remove it, and it is invariant to whether the model affirms or denies an interior; and (4) the role is model-specific, failing to transfer to the larger Qwen3.5-122B-A10B. The result is not that E114 detects self-reference; self-reference was one discovery route into a broader examination register. We release the prompts, scripts, and provenance under the MIT license. This deposit is the named official paper-scoped artifact bundle: the paper (ACL-style LaTeX source and built PDF), per-run summaries, analysis scripts, prompt and class manifests, checksums, plots, and source journals. Contact: Jeffrey W. Shorthill (jws299792@icloud.com). Raw activation/router tensors are kept out of version control and archived separately (see ZENODO-TENSORS.md).

Read Full Paperexternally

AI에게 질문

Bookmark

View Full Paper