Crosscoder Analysis of Circuit-Aware LoRA: Interpreting Security Model Fine-Tuning via Activation Difference Sparse Autoencoders

We apply crosscoder model diffing to analyze the circuit-level changes induced by Circuit-Aware LoRA fine-tuning. By training a sparse autoencoder (SAE) on the activation difference between base and fine-tuned models (\(Δ = hfine - hbase\)), we isolate the specific circuits modified by LoRA adaptation. Applied to security-domain fine-tuning of Qwen2.5-7B and Qwen3-8B, we show that Circuit-Aware LoRA modifies \(3.2×\) fewer features than standard LoRA for the same task performance, and that the modified features are concentrated in layers 11-26 (the semantic/reasoning workspace region). We further demonstrate that the diffing SAE encoder provides a principled initialization for future LoRA adapters, reducing training time by \(27%\).