Human behavior recognition based on graph neural network and cross-modal collaborative attention mechanism

Fusing visual, skeletal, and audio can enhance behavior recognition robustness. Yet existing methods struggle with dynamic spatiotemporal modeling and adaptive cross-modal collaboration, especially under occlusion or lighting variations. This paper proposes a novel Hierarchical Collaborative Attention Mechanism (HCAM) that consists of intra-modal spatiotemporal attention and cross-modal collaborative attention integrating multimodal spatiotemporal feature data, including visual, audio and skeletal information extracted by graph neural networks, to develop a model for human behavior recognition (Gnet_CAM).This method first constructs a dynamic spatiotemporal graph to model both physical relationships based on human joints and non-physical relationships mediated by audio attention. Subsequently, it employs the proposed HCAM to achieve adaptive extraction and alignment fusion of spatiotemporal features both within and across modalities. On the NTU RGB+D120 dataset, this method achieved a recognition accuracy of 93.1%. Outperforming the RGB+Skeleton-based modal HDMFNet by 1.8 percentage points, the skeleton-based modal 2s-AGCN by 5.1% and the RGB-based modal I3D by 10.6%. Ablation experiments demonstrated that removing either the virtual attention edges or the cross-modal collaborative attention results in a significant decline in model performance. The experimental results suggest that our proposed framework can effectively enhance the robustness of behavior recognition in complex scenarios. Not applicable.