Voice and Sign Language to Intent Model for the Deaf: A Multimodal Approach to Assistive Communication

Communication barriers for hearing-impaired individuals remain a critical challenge in India, affecting over 63 million people who struggle to access essential services in healthcare, banking, and government institutions. This paper presents a zero-cost multimodal assistive communication system combining Indian Sign Language gesture recognition with dysarthric speech processing to generate accurate intent classification. The system leverages MediaPipe for hand tracking, achieving 95% hand detection accuracy, Wav2Vec2 and Google Speech Recognition API for processing broken speech patterns, and Random Forest classifiers for intent classification. The proposed open-source solution demonstrates expected accuracy of 85% or higher with sub-2-second response times, offering a 96% improvement in communication speed compared to traditional interpreter-based methods. The modular architecture enables deployment across diverse environments including homes, hospitals, educational institutions, and public spaces.