Mercury Model Birthmarks: Structural Signatures of Token Cost, Latency, and Hidden Defects Across LLM Scales

Key Points

• This paper aims to define 'model birthmarks' as indicators of the health and function of language models.
• Introduces the concept of model birthmarks as stable structural signatures.
• Identifies various types of birthmarks related to tokenizer burden and latency.
• Proposes measurable vectors for effective language model inspection.
• Establishes that parameter count alone does not reflect model health.
• Validates a framework for the inspection of language models via defined birthmark signatures.
• Demonstrates the relevance of structural signatures in assessing model performance.