Video-based Sign Language Recognition with R(2+1)D and LSTM Networks

Sign language, serving as a crucial means of communication for individuals with hearing impairments, plays a pivotal role in achieving barrier-free communication with the broader population. Despite advancements in utilizing deep learning for automatic recognition of dynamic sign language, challenges persist, especially in the context of video-based sign language recognition. This requires careful consideration of spatial and temporal features, along with the inherent long-term characteristics of the sequence. In this paper, we introduce a novel sign language recognition model framework that integrates the R(2+1)D and LSTM networks. Initially, the R(2+1)D network is employed to extract spatial and temporal features from sign language sequences. Subsequently, these features are input into the LSTM network to capture long-term features and eliminate redundant information. The recognition of the video sequence is accomplished through a fully connected layer. We evaluate the proposed network on two distinct Sign Language datasets, which achieved 96.21% on CSL and 99.69% on LSA64. Our method demonstrates significant improvements over existing methods.