Machine Learning and Mediapipe Assisted Sign Gesture-based Smart Keyboard for Deaf and Blind People

There are 1.1 billion deaf and 2.2 billion blind people, and many people have speech impairments. These disabilities cause them to be unable to communicate with others and use modern technologies. This paper presents a real-time smart touchless keypad by which people with blindness, deafness, and speech impairments can interact with the computer with the help of 37 hand gestures using a single camera. The presented solution uses a hand tracking module from the mediapipe library for feature extraction, a random forest algorithm for the classification of gestures, and a PyAutoGUI library for controlling the mouse and keyboard according to the classified hand gestures. The presented solution has achieved an accuracy of 99.66% on the test set, indicating that this system can also detect gestures for communication with deaf or speech-impaired people. Overall, this system employs computer vision and machine learning techniques to improve the lives of the deaf, blind, and mute people.