GTR: Global Token Retrieval for Window-Based Vision Transformer

Vision Transformers have achieved remarkable success in visual recognition tasks due to their strong capability to model long-range dependencies through global self-attention. However, the inherent quadratic complexity of global attention severely limits their scalability when processing high-resolution inputs. To address this issue, window-based attention mechanisms have been proposed to reduce computational overhead, but they often struggle with limited receptive fields and insufficient global context modeling. In this work, we proposeGTR-ViT, a novel and efficient Vision Transformer framework that integrates a Global Token Retrieval (GTR) mechanism with an enhanced feedforward network in a synergistic manner. The GTR mechanism dynamically retrieves globally relevant tokens for each token, enriching the local features with comprehensive global context. These globally enhanced features are then fed into a gated feedforward network equipped with depthwise convolution, which further refines local representations through adaptive feature modulation. In turn, the improved local features serve as more representative queries for the GTR module, enabling more accurate global token retrieval in subsequent layers. This mutual reinforcement forms a virtuous cycle of local-global interaction, significantly boosting the models capability in both context understanding and feature discrimination. Extensive experiments on image classification and downstream tasks across multiple datasets demonstrate that GTR-ViT consistently surpasses prior state-of-the-art methods, highlighting the effectiveness of this collaborative design.