A facial gesture-based interaction framework for hands-free game control

Facial gesture interaction has gained increasing attention as a hands-free input modality for interactive systems. However, existing solutions still face limitations in adaptability, personalization, and short-term interaction design, often due to restricted gesture vocabularies, hardware dependencies, and limited adaptation to individual users. To tackle these problems, this paper presents a modular framework for integrating facial gesture recognition into gamified interactive applications, enabling intuitive hands-free control for accessibility-oriented contexts. The system supports real-time facial pose tracking using both standard webcam or virtual reality-based configurations. A lightweight calibration procedure enables user-specific gesture definition, while selective landmark masking and pose-based analysis enhance gesture discrimination and interaction stability. The framework also supports sequence-based gesture interaction, allowing the recognition of ordered pose patterns. To assess technical feasibility and interaction performance, five gesture-controlled mini-games and a Simon-says training task were implemented. A study with sixteen participants under controlled conditions showed stable gesture recognition across both hardware configurations. The webcam provided lower latency, while both configurations showed small, gesture-dependent differences in performance. Feedback collected from the short User Experience Questionnaire indicated positive usability across pragmatic and hedonic dimensions although further validation in real-world settings is needed to assess user experience beyond short-term controlled conditions. This work focuses on technical feasibility and interaction validation rather than application-specific performance.