Natural Human-Machine Interaction Using Static Hand Gestures for a Gestural Calculator System with DNN

Hand gesture recognition (HGR) represents a real challenge for natural human-computer interaction, which aims to revolutionize the naturalness of traditional interfaces, allowing intuitive control of various devices without using a keyboard or mouse. Despite the availability of frameworks such as MediaPipe, which enable better detection and tracking, the major challenge remains interpreting gestures made with both hands in a natural operational setting. In this regard, this study presents a real-time gesture calculator that combines gestures made with both hands (see using one hand) and aims to address the problem of interpretation in arithmetic operations. By leveraging MediaPipe to classify the 21 hand landmarks, an optimized dense neural network (DNN) was developed capable of recognizing 13 distinct static gestures. The latter includes six gestures for each hand (ranging from 0 to 5) to represent all digits from 0 to 9, five mathematical symbols, and two specialized commands designed explicitly for control management. Even with a standard webcam, this model achieved 91% accuracy on a reduced dataset of gestures from both hands. Beyond gesture recognition, this work demonstrates how these gestures can be integrated into a fluid sequence for arithmetic operations.