ABSTRACT Existing gesture‐controlled robotic systems often decouple perception from control, resulting in delayed, discrete, and noncontinuous motion execution. To address these limitations, this study introduces a novel real‐time human‐robot interaction framework‐ AI‐RTGM (Artificial Intelligence‐based Real‐Time Gesture Mapping)‐that unifies gesture recognition, trajectory generation, and robot control within a single adaptive pipeline. Unlike traditional models that depend on pre‐scripted commands or sensor‐bound gestures, AI‐RTGM integrates a CNN‐based vision model for continuous gesture tracking (GESTID) with a dynamic trajectory generation and execution module (TAGEM) capable of producing smooth, coordinated 7‐DoF motions in real time. The framework facilitates bidirectional communication between gesture perception and robotic actuation via ROS and libfranka , achieving an end‐to‐end latency below 70 ms. Experimental validation on tasks such as pick‐and‐place and surface cleaning demonstrates precise, stable, and compliant motion replication directly driven by human intent. The proposed AI‐RTGM model fundamentally differs from existing approaches by integrating real‐time gesture perception with continuous trajectory‐level control, enabling a more natural, intuitive, and scalable paradigm for human‐robot collaboration.
Hussain et al. (Sun,) studied this question.