Examining the Universality Assumption of Attention Head Pruning across Datasets

Multi-head attention is a central component of modern transformer-based large language models (LLMs), allowing models to capture diverse patterns via multiple parallel attention heads. Empirical studies show that many heads are redundant, and pruning them can reduce computational and memory costs without substantially affecting output quality. Prior work has also suggested that head pruning may be universal, implying that the same heads can be pruned for different input datasets. Based on this assumption, much of the work on head pruning either considers only a single dataset or employs a fixed pruning strategy across different datasets. In this letter, we systematically evaluate this assumption across multiple datasets and models. Our results reveal that which attention heads to prune is largely dataset-specific rather than universal (i.e., dataset-agnostic), indicating that pruning strategies must be dynamically adapted to different datasets.