A General Safety-Aware Hybrid Multimodal Architecture for Sign Language Understanding in Automated Vehicle Interaction

Sign language understanding for automated vehicles sits at the intersection of accessibility, intelligent transportation, and safety-critical human–machine interaction. The existing sign-language recognition systems are largely confined to controlled environments, limiting their utility in mobility scenarios characterized by lighting variation, motion blur, and partial occlusion. This paper proposes STCM-HVNet, a safety-aware hybrid multimodal architecture integrating four components: a spatial visual encoder, a MediaPipe-based pose encoder, a bidirectional LSTM temporal encoder, and a context-aware fusion and safety decision module. The architecture is formulated as a multi-task system that jointly predicts sign category, interaction intent, and urgency level, and incorporates confidence-aware rejection and fail-safe action mapping. Experiments are conducted on two Arabic sign-language resources. On the RGBArS image benchmark (31 classes, 7856 images), the proposed pipeline achieves a Top-1 accuracy of 45.38%, Top-3 accuracy of 75.15%, and Macro-F1 of 0.4479, outperforming LinearECOC, kNN-5, and Bagged Trees baselines. On the Arabic sign-language video benchmark (12 classes, 479 clips), the BiLSTM temporal encoder achieves a Top-1 accuracy of 93.15% and Macro-F1 of 0.9383, outperforming frame-aggregation (87.67%) and CNN-LSTM (89.04%) baselines. Ablation results confirm complementary contributions from the visual and pose branches. A safety-threshold analysis and a Monte Carlo dropout comparison demonstrate that the proposed safety decision/gating layer provides a controllable trade-off between prediction coverage and reliability.