A mathematical perspective on transformers

Transformers play a central role in the inner workings of large language models.We develop a mathematical framework for analyzing transformers based on their interpretation as interacting particle systems, with a particular emphasis on long-time clustering behavior.Our study explores the underlying theory and offers new perspectives for mathematicians as well as computer scientists.Contents Outline 427 1. Modeling 429 2. Clustering 442 3. Further questions 453 Appendix A. Proof of Theorem 2.1 462 Appendix B. Proof of Theorem 2.4 464 Appendix C. Proof of Theorem 2.3 467 Appendix D. Proof of Theorem 2.13