Mercury Attention-Head Cartography and GQA/KV Sharing

Key Points

• The aim is to formalize the concept of attention head cartography to enhance understanding and management of attention mechanisms in models.
• Proposes addressing attention heads as ownership contracts with query and output components.
• Introduces a framework for understanding attention head roles in GQA and MQA models.
• Utilizes a formal model to analyze attention heads mapped to key/value interactions.
• Establishes a rigorous approach to managing attention heads beyond mere labeling.
• Demonstrates improvements in query routing efficiencies within models during attention processing.