Attention Mechanisms: From Theory to Transformers

Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs) have a well-documented structural limitation. When sequences get longer, the propagation of gradient signals becomes increasingly unreliable, known as the vanishing gradient problem. Such a weakness naturally limits their ability to model long-range token dependencies, which are crucial to tasks such as machine translation, document summarisation, and question answering. This bottleneck is directly addressed by attention mechanisms. Instead of encoding the whole input sequence into a single fixed-length hidden state, attention enables a model to dynamically retrieve and weight information from any position in the input at each decoding step. The resulting computational cost scales quadratically with sequence length but allows full parallelisation across token positions, making attention-based architectures tractable at scale. In this survey we investigate attention mechanisms from mathematical and empirical perspectives. We analyse the theoretical basis of scaled dot-product attention, the statistical motivation for the scaling factor, its extension to multi-head attention, and formally compare it to recurrent architectures along three axes: computational parallelism, information path length and gradient stability. It covers natural language processing, computer vision and speech systems, and attention-based models always beat sequential models. The methodology is a blend of systematic literature review, based on the ACL Anthology and arXiv (cs.LG, 2014-2024), and formal derivations and illustrative numerical analysis. We are not concerned with listing empirical findings, but with showing the structural reasons for them.