With the development of human–computer interaction technology, non-contact interaction based on gaze tracking and facial movements has become a research hotspot. Traditional mouse-and-keyboard methods pose challenges for people with disabilities or limited hand movements, while existing gaze-tracking systems often rely on expensive hardware or lack sufficient accuracy. This paper designs and implements a real-time system using ordinary cameras, achieving natural, efficient interaction via multimodal input combination. The system uses an improved MobileNetV2 backbone to construct GazeTrackNet for gaze estimation. It adopts MediaPipe Face Mesh to detect facial landmarks. Meanwhile, it applies geometric feature analysis, including eye aspect ratio and mouth aspect ratio, to identify actions such as blinking and mouth opening. It adopts a hybrid control strategy that combines gaze jumping and head fine-tuning, using mouth state as the main control switch. Key contributions include a lightweight gaze-tracking algorithm that enables stable and efficient gaze detection on consumer-grade hardware, a multimodal interaction strategy based on facial movement that improves system stability and ease of use, and a complete prototype system that achieves real-time performance on standard laptops. Experimental results show an average gaze average angle error of 3.0°, 97% eye state recognition accuracy, and end-to-end latency below 70 ms. The system can satisfy the requirements of daily desktop interaction under normal indoor lighting, and shows potential for future barrier-free interaction applications after further validation with target users. Existing gaze-tracking methods either suffer from low precision on lightweight devices or bring heavy computational overhead. Common facial recognition approaches also face frequent false trigger interference. Compared with them, our scheme achieves balanced accuracy and real-time performance via an attention-enhanced structure, and the designed dual anti-shake mechanism effectively suppresses misjudgment, delivering a more stable hands-free interaction experience.
Liu et al. (Thu,) studied this question.