Silence in the Noise: Probing Attention Artifacts and Register Tokens in Audio Spectrogram Transformers

Vision Transformers and Language Transformers such as BERT develop outlier-token activations during training: a small fraction of tokens whose hidden-state norms grow an order of magnitude larger than the rest, acting as ad-hoc registers — and the dominant failure mode of post-training INT8 quantization, requiring engineering fixes to recover deployment-grade accuracy. The Audio Spectrogram Transformer (AST) inherits the ViT architecture almost wholesale, raising an interpretability question (do ASTs exhibit analogous artifacts in low-energy spectrogram regions?) and a deployment question (are ASTs subject to the same INT8 failure mode?). Using the public AudioSet-pretrained AST checkpoint and ESC-50: (i) Mechanistic — across all twelve encoder layers, AST's patch-token L2-norm distribution is tight and unimodal (final-layer max-to-median ratio 1.23, outlier rate 0.0%); the outlier pattern documented for deep ViTs and LLMs does not appear in AST at any depth. (ii) Register intervention — adding learnable register tokens reduces patch–patch attention norm monotonically (register attention mass 11.2% at n=16), with a small, non-significant accuracy improvement. (iii) Deployment — AST tolerates naive post-training INT8 quantization to deployment-grade fidelity on three platforms (token-state cosine 0.980–0.9988, top-5 rank agreement 0.73–0.95, 49–74% smaller) with no architectural modification — to our knowledge the first major transformer family shown to do so without activation-outlier mitigations. We propose the max-to-median final-layer norm ratio as a lightweight diagnostic for INT8-readiness.