Activation sparsity is widely used to accelerate large language model inference, and is often claimed to make neurons more interpretable, or monosemantic. We test this claim directly, comparing an efficiency-driven ReLU-sparsified model (MiniCPM-S-1B) against its dense SiLU sibling on per-layer neuron monosemanticity, measured by both a Jensen–Shannon concept-separability score and a variance-based η² score. While the sparse model appears more monosemantic in deeper layers, we show this advantage is fully explained by network depth: after regressing out depth, per-layer sparsity has no detectable correlation with the monosemanticity gain (residual r ≈ 0.12). This null result is robust across two orthogonal concept dimensions (topic and sentiment) and two methodologically distinct metrics. Our finding stands in tension with reports of sparsity-induced monosemanticity in mixture-of-experts models, suggesting that any link between sparsity and monosemanticity may depend on the specific form of sparsity rather than holding universally.
Ke Yufeng (Sun,) studied this question.