GLFormer: a hierarchical cross-layer transformer decoding framework with global–local feature collaboration for video action recognition

Abstract CLIP’s effectiveness across image recognition tasks stems from its large-scale multimodal pretraining, which equips it with powerful and transferable visual representations. However, when directly applied to video action recognition, the CLIP image encoder often falls short in modeling global temporal dependencies and fine-grained local motion dynamics across frames. Moreover, by treating all patch tokens uniformly, it lacks the ability to attend to motion-salient regions that are critical for recognizing human actions. To address these limitations, we propose GLFormer, a novel Transformer-based framework that integrates Global–Local collaborative features of the whole video with a hierarchical cross-layer transformer decoding strategy. Specifically, we employ a frozen CLIP-pretrained Vision Transformer (ViT) as the image encoder to extract multi-layer spatial features from video frames. We then decouple the modeling of global and local information for each-layer features: CLS tokens are temporally encoded to capture global video-level semantics, while patch tokens are refined via spatial attention and temporal residual modeling to highlight local motion cues. The resulting spatio-temporal features from multiple ViT layers are progressively aggregated through a hierarchical Transformer decoder, where a learnable video-level query token interacts with features across different layers in a cross-layer manner. Extensive experiments on benchmark datasets demonstrate that GLFormer achieves remarkable performance while significantly reducing computational overhead, striking a strong balance between accuracy and efficiency for video action recognition.