A symmetry-matching approach to blind-spot elimination in sparse autoencoders

AbstractLanguage models can treat numerically distinct clinical statements as interchangeable at the representation level, creating blind spots with patient-safety implications. We show that such blind spots arise from the orientation of the feature basis, not from insucient model capacity, and that they are eliminable. Using a vulnerability measure derived from algebraic error-detection theory, we add a dierentiable regularisation term to the sparse autoencoder training objective that penalises uneven perturbation sensitivity. Across three language models of dierent scale and architecture (GPT-2, Gemma 2, Qwen 2.5), the regularisation reduces blind-spot severity by 83100% on six perturbation families while alternative training objectives (JumpReLU, MDL) leave the blind spots unchanged. On the two larger models the numeric blind spot is fully eliminated (V ≤0.005). A single well-oriented feature basis suffices for all families simultaneously. No model retraining or additional capacity is required.