Answers are grounded in this paper and cite their sources.
Structural fragility analysis reveals pruning sensitivity phases in transformers, indicating how certain layers tolerate aggressive head removal.
We apply structural fragility analysis a domain-agnostic graph theoretic framework originally validated on molecular signalling networksto the internal topology of transformer attention mechanisms. We extract co-activation graphs from BERT-base (144 nodes, 12 lay ers × 12 heads) and compute per-edge triangle participation counts tri(e) and the Fragility Index (FI), which quanti es the proportion of connections with zero local redundancy. At a correlation thresh old of 0.8, BERT exhibits FI = 0.97%, con rming massive structuralover-redundancy that quantitatively explains the known tolerance of transformers to aggressive head pruning.Progressive layer-wise pruning (110 heads per layer, 3 trials, SST-2 validation) reveals three functional regimes: (i) immune lay ers (L1, L7, L10) with near-zero degradation even at 83% head removal, (ii) critical layers (L2, L3, L5, L6, L8, L9) exhibiting smooth monotonic decay, and (iii) bu er layers (L0, L4, L11) with delayed onset. Crucially, grouped multi-layer pruning reveals a structural phase transition in pruning sensitivity: at low perturbation budgets, topologically fragile layers degrade faster (∆ = −0.58% vs −0.45%), but at high budgets, topologically redundant layers collapse disproportionately (∆ = −4.18% vs −1.53% at 8 heads/layer). Normalised per head pruned, damage from robust heads doubles (0.075% vs 0.038%), with superlinear ampli cation characteristic of cooperative failure above a percolation threshold.This reverses the intuitive interpretation of redundancy: highly triangulated structures are not structurally safe but mark regions of concentrated functional dependency. The analysis is purely structural, requiring no gradient computation or ne-tuning, making it computationally efficient and inherently parallelisable for large-scale transformer architectures
No takes yet. Share an insight, caveat, or question.
David Martin Venti (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: