E-Signspeak: An Emotion-Integrated Indian Sign Language Translation System Using Deep Learning

Sign language serves as the primary mode of communication for over 18 million hearing-impaired individuals in India, yet real-time translation systems that capture both linguistic content and emotional context remain limited. This paper presents E-SignSpeak, a dual-modality deep learning framework that jointly recognizes Indian Sign Language (ISL) gestures and facial emotional expressions from video input, and generates context-aware natural language output with speech synthesis. The ISL recognition module employs a CNN-augmented stacked Bidirectional LSTM with multi-head self-attention, trained on the INCLUDE-262 dataset, achieving 90.86% Top-1 accuracy across 262 classes. The Facial Emotion Recognition module fine-tunes MobileNetV2 on FER2013, achieving performance consistent with human-level benchmarks (~65%). The outputs are fused through a lightweight API pipeline, where few-shot FLAN-T5 prompting generates emotion-aware sentences. To the best of our knowledge, this work represents one of the first web-deployable systems integrating ISL recognition and emotion understanding into a unified assistive pipeline. The results demonstrate the feasibility of affect-aware sign language translation for real-world applications.