What question did this study set out to answer?

This research aims to analyze attention memory patterns in transformer models to understand their storage mechanisms.

March 21, 2026Open Access

Attention Memory Patterns — What Models Actually Store in KV-Cache

Key Points

This research aims to analyze attention memory patterns in transformer models to understand their storage mechanisms.
Conducted a systematic examination of attention memory patterns
Analyzed head specialization and attention sink phenomena
Investigated information density gradients across model layers
Assessed key-value redundancy patterns for cache compression
Identified distinct patterns of head specialization in attention mechanisms
Observed attention sink phenomena impacting information processing
Found varying information density across different layers
Highlighted key-value redundancy that can enhance cache compression strategies

Abstract

Systematic analysis of attention memory patterns in transformer-based large language models, examining head specialization, attention sink phenomena, information density gradients across layers, and key-value redundancy patterns that inform cache compression strategies.

Connected Papers

Building similarity graph...

Analyzing shared references across papers

Discussion

Authors

Oleh Ivchenko

Odessa National Polytechnic University

Actions

Institutions

Odessa National Polytechnic University

References and Citations

Connected Papers

Building similarity graph...

Analyzing shared references across papers

Attention Memory Patterns — What Models Actually Store in KV-Cache

Key Points

Abstract

Citation Network

Connected Papers

Discussion

Authors

Actions

Institutions

References and Citations

Citation Network

Connected Papers

Discussion

Cite this study