Cross-Scale Focal Interactive Attention Network for Skeleton-based Action Recognition

Skeleton-based action recognition has attracted much attention, benefiting from its succinctness and robustness. Previous methods often compute pairwise global self-attention for all joints in both spatial and temporal dimensions, which leads to computational redundancy. Besides, there is still a lack of a flexible strategy for effectively fusing multi-scale skeletal features. In this paper, we propose a Salient Feature Localization module that leverages the class activation map to guide the adaptive selection process of the most salient joint tokens in all frames and subject them to further processing. Then, Deformable Spatial-Temporal Focal Attention adopts the deformable sampling strategy to calculate focal attention on the selected salient features. Cross-Scale Adaptive Interactive Attention performs cross-scale attention in the spatial dimension and calculates the offset of salient feature reference points, thereby allowing for flexible integration of multi-scale information more dynamically and adaptively. Extensive experiments are conducted on the datasets NTU RGB+D and NTU RGBRGB+D. Our methods perform competitively compared to state-of-the-art methods.