Enhanced by Visual and Semantic Consistency for Continuous Sign Language Recognition

Abstract Camera-based interface enables simple human-computer interaction with intuitive sign language for hearing-impaired users. Sign language, as a visual language, utilizes changes in hand shape, body movements, and facial expressions to collaboratively convey information. Most of the current continuous sign language recognition (CSLR) models focus their attention on the extraction of information from each frame of the image and ignore the dynamically changing characteristics of the signer across multiple frames. The contrasts with the essence of sign language recognition: which aims to learn the most essential feature representations of changes in the hand-controlled part and the non-hand-controlled part, and convert them into language. In this paper, we first use the feature alignment method to explicitly capture the spatial position offset and motion direction information between neighboring frames, direct a dynamic attention mechanism to focus on the subtle change region, enhance visual representation extraction. And we propose a dynamic decoding method based on maximum backtracking probability to decode word-level features and achieve word consistency constraints without increasing computational resources, enhance semantic consistency. We propose a comprehensive CSLR model utilizing a Dynamic Attention Mechanism and Maximum Backtracking Probability Dynamic Decoding (DAM-MCD), enhancing the model's inference capability and robustness. Experiments were conducted on two publicly accessible datasets, PHOENIX14 (Koller et al. Comput Vis Image Underst 141:108–125, 2015), PHOENIX14-T (Camgoz et al., in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7784–7793, 2018), demonstrating that the DAM-MCD model achieves higher accuracy compared to methods employing multi-cue input. The results further show that the DAM-MCD model effectively captures sign language motion information in videos. Models will be made public on: https://github.com/smellno/Continuous-Sign-Language-Recognition-.