Attention Memory Patterns — What Models Actually Store in KV-Cache

Key Points

• This research aims to analyze attention memory patterns in transformer models to understand their storage mechanisms.
• Conducted a systematic examination of attention memory patterns
• Analyzed head specialization and attention sink phenomena
• Investigated information density gradients across model layers
• Assessed key-value redundancy patterns for cache compression
• Identified distinct patterns of head specialization in attention mechanisms
• Observed attention sink phenomena impacting information processing
• Found varying information density across different layers
• Highlighted key-value redundancy that can enhance cache compression strategies