Real-time Sign Language Translation using Computer Vision and Machine Learning

Indian Sign Language is a visual-gestural mode of communication used by the deaf community in India. It serves as a means of communication by utilizing various gestures, eye movements, and body movements. ISL is distinct from spoken language and has its own grammar and syntax. This research paper presents a solution to bridge the communication gap faced by the deaf and mute communities who use sign language as a mode of communication and the rest of the population. This paper proposes an application that can translate sign language, specifically Indian Sign Language to text in real-time. The text generated will be available in two languages, Hindi and Kannada to help the non-English speaking population. This application employs modern computer vision and machine learning techniques. Mediapipe has been used for preprocessing the input videos to generate features. Three models: Convolutional Neural Network, Recurrent Neural Network - LSTMs and Transformer-Encoders have been implemented and compared to evaluate the best working model for the given task. The results indicated a similar level of accuracy for both LSTMs and Transformers models at 95-97%, thus were combined to create an ensemble model. The sequence of words that is predicted as output is then passed through a translation architecture to translate this text into coherent Hindi/Kannada sentences. Overall, this research provides a faster and more affordable method to bridge the communication gap present.