MULTI-Stream Graph Convolutional Networks with Efficient spatial-temporal Attention for Skeleton-based Action Recognition

In skeleton-based action recognition, graph convolutional networks (GCN) based methods have achieved remarkable performance by building skeleton coordinates into spatial-temporal graphs and explored the relationship between body joints. ST-GCN 19 proposed by Yan et al is regarded as a heuristic method, which firstly introduced GCN to skeleton-based action recognition. However, it applied graph convolution on joints of each frame equally. Less contribution joints caused interference in generating intermediate feature maps. We designed a spatial-temporal attention module to capture significant feature in spatial and temporal dimension simultaneously. Moreover, we adopted inverted bottleneck temporal convolutional networks to decrease computational amount and learned more feature with residual construction. Besides useful message in joints, bones and their movement also contain learnable information for analyzing action categories. We input data to a multi-stream framework. Finally, we demonstrated the efficiency of our proposed MSEA-GCN on NTU RGB+D datasets.