The demand for assistive technologies and smart home automation had increased the necessity of developing a system with effective and real-time human- computer interaction. This paper proposes an AI-based sign language translator for smart home automation, where the user can use hand gestures to control home appliances. We used a computer vision-based, deep learning IoT embedded system to design a simple yet effective system for accurate and low-latency gestural identifications. Unlike raw image-based approaches, the system uses vision-based input acquired by a camera, from which MediaPipe extracts 21 hand landmarks for every frame. For identifying dynamic gestures, one approach is to use a hybrid CNN–LSTM model that learns spatial and temporal features from sequences of gesture data. Trained on gesture sequences of 30 frames per sample, the model obtains an average recognition accuracy ~95% on the test dataset. The system operates on an edge device which runs on Raspberry Pi technology to deliver its real-time function while achieving cost- effective and energy-saving results. The system operates in real-time environments because it processes each gesture with an average inference time of one to two seconds. The system uses relay modules for gesture recognition to create control commands which are sent to IoT devices to perform actions such as activating buzzers and controlling fans and switching lights on and off. The suggested method achieves precise results through landmark-based processing which decreases computational needs by 70 to 80 percent compared to standard image-based deep learning methods. The system performs well in a variety of backgrounds and lighting situations, making it appropriate for real-world implementation. The proposed method provides a flexible solution which operates with short delays and low costs to create smart home automation systems that assist disabled users to achieve independent living.
Katkar et al. (Thu,) studied this question.