Gaze Tracking with Multi-Modal Neural Network in Desktop Environments Using Generic Web Camera

Gaze tracking is the process of estimating where a person is looking on the screen using only information from eye movement without additional input from the user, it contributes greatly in understanding and improving human-computer interactions. This paper addresses an improved gaze tracking technique building upon established approaches that are using generic web cameras instead of requiring high-cost specific hardware like infrared cameras and sensors. In the proposed method, a pipeline was constructed of various high performing AI models such as Google's MediaPipe face localization and face landmarks solutions to acquire both Iris metrics and localized eye image data. The data obtained is then passed to the proposed model that uses both image data and location metrics to predict the user's gaze position on the screen using a multi-modal architecture. Experiments done on realistic desktop environments show 8% mean distance error of the screen dimensions with a standard deviation of +/-6% from the actual gaze location. The proposed method is not affected by most logistical limitations affecting other similar approaches.