The remarkable success of autoregressive Large Language Models (LLMs) is predicated on the causal attention mechanism, which enforces a static and rigid form of informational asymmetry by permitting each token to attend only to its predecessors. While effective for sequential generation, this hard-coded unidirectional constraint fails to capture the more complex, dynamic, and nonlinear dependencies inherent in sophisticated reasoning, logical inference, and discourse. In this paper, we challenge this paradigm by introducing Dynamic Asymmetric Attention (DAA), a novel mechanism that replaces the static causal mask with a learnable context-aware guidance module. DAA dynamically generates a continuous-valued attention bias for each query–key pair, effectively learning a “soft” information flow policy that guides rather than merely restricts the model’s focus. Trained end-to-end, our DAA-augmented models demonstrate significant performance gains on a suite of benchmarks, including improvements in perplexity on language modeling and notable accuracy boosts on complex reasoning tasks such as code generation (HumanEval) and mathematical problem-solving (GSM8k). Crucially, DAA provides a new lens for model interpretability. By visualizing the learned asymmetric attention patterns, it is possible to uncover the implicit information flow graphs that the model constructs during inference. These visualizations reveal how the model dynamically prioritizes evidence and forges directed logical links in chain-of-thought reasoning, making its decision-making process more transparent. Our work demonstrates that transitioning from a static hard-wired asymmetry to a learned and dynamic one not only enhances model performance but also paves the way for a new class of more capable and profoundly more explainable LLMs.
Wen et al. (Tue,) studied this question.