Multi-Modal Human Action Recognition in videos using LSTM with Feature Fusion

Abstract Over past few decades, Human action recognition (HAR) in RGB-Depth videos has been extensively studied on the availability of economical depth sensors. Presently, unimodal methods such as skeleton-based and RGB video-based have achieved significant advancements by utilizing progressively larger amounts of data. Nevertheless, there has been a challenge of extracting the temporal feature in RGB video, spatial feature in skeleton sequence. To solve this challenge, we propose Skeleton-RGB Feature Fusion LSTM (SRF-LSTM) network to combine the skeleton and RGB stream modalities and to recognize actions. The fusion of RGB video and skeleton sequences capture both temporal and spatial information effectively. The proposed SRF-LSTM model performs better with an average accuracy of 90.4% than state-of-the-art techniques on NTU RGB-D dataset.