Architectural Evolution of Large Language Models: A Systematic Survey on Attention Mechanisms, Efficiency, Adaptation, and Emerging Paradigms

Key Points

• The aim is to systematically review the development and efficiency enhancements of Large Language Models and their underlying architectures.
• Conducted a systematic survey of architectural advancements in Large Language Models.
• Analyzed various mechanisms including efficient attention and retrieval-augmented generation.
• Explored emerging paradigms like mixture-of-experts and multimodal architectures.
• Identified significant advancements in attention mechanisms that enhance processing efficiency.
• Found that multimodal LLMs improve performance across diverse data types.
• Noted the rise of new paradigms, such as Retrieval-Augmented Generation, that optimize inference.