Semantic-aware multimodal learning with dynamic semantic importance fusion for robust sign language recognition

Sign language recognition (SLR) remains challenging due to signer variability, viewpoint changes, visual ambiguity, and complex temporal gesture dynamics. Existing methods often rely on single-modality representations or static fusion strategies, limiting their ability to exploit the complementary nature of skeletal structure, visual appearance, and motion evolution. To address these limitations, this paper proposes SASLR, a semantic-aware multimodal framework for robust sign language recognition. The framework learns three complementary representations through a Structural Semantic Stream for articulated body relationships, an Appearance Semantic Stream for hand-shape and upper-body visual cues, and a Dynamic Semantic Stream for global motion evolution. The central contribution of this work is the Dynamic Semantic Importance Fusion (DSIF) mechanism, which reformulates multimodal integration as an adaptive semantic importance estimation problem and dynamically weights modality contributions according to each sign instance. Extensive experiments on the dynamic datasets ChaLearn 2013, ChaLearn 2016 , and AUTSL demonstrate the effectiveness of the proposed multimodal framework with DSIF, achieving accuracies of 98.65%, 88.00%, and 98.40%, respectively. On the static ISL dataset, where only the Appearance Semantic Stream is applicable, the appearance-based pathway achieves 80.23% accuracy, demonstrating the effectiveness of the visual semantic component in isolation. Modality contribution analysis, fusion comparison, ablation studies, computational efficiency evaluation, explainability analysis, and failure-case investigation further confirm the robustness, interpretability, and practical feasibility of the proposed approach. The results demonstrate that semantic-aware multimodal representation learning with adaptive importance fusion provides an effective and interpretable solution for sign language recognition.