Major communication barriers are often encountered among deaf and/or speech-impaired people, which is particularly serious during emergencies. This study proposes a novel vision-based sign-to-voice conversion that recognizes hand gestures through a convolutional neural network incorporated with advanced preprocessing and transformer-based classification. A custom gesture dataset was created by capturing labeled images and video frames under different backgrounds, lighting conditions, and hand orientations for proper training. Skin filtering, background subtraction, noise removal, and extraction of regions of interest are incorporated to preprocess the raw gestures for better representation before feature learning. Experimental results show 99% accuracy, 97% recall, and an F1-score of 97%, while low error measures (PSNR 33.1 dB, and MSE 0.03) are indicative of the system’s high reliability with minimal sensitivity to noise. Comparison demonstrates that the proposed CNN-transformer pipeline significantly outperforms classic template-based or color-based techniques. Given its robustness and generalizability, this system can hence support practical communication in real time and thus act as a promising assistive technology to increase accessibility, especially for people with disabilities in urgent communication situations.
Gajjala et al. (Wed,) studied this question.