Gaze and Language in Action: Multimodal Interfaces for Cognitive Assistance

In this paper, we present a novel software framework for portable, gaze-aware assistance, aimed at supporting people with dementia during daily kitchen tasks. Unlike traditional systems that rely on fixed infrastructure, our approach enables real-time, context-sensitive guidanceusingahostdevice(e.g.,a laptop)together with Meta’s Project Aria smart glasses. The core contribution lies in a multimodal sensor fusion architecture that combines egocentric vision (YOLOv8n-based object detection), binocular gaze tracking, and inertial spatial localisation. This fusion supports persistent object awareness without requiring environmental instrumentation. Detected objects and their spatial context are structured into a knowledge graph (KG) that maps feasible user actions. A compact language model (GPT-4o-mini) leverages this representation to generate voice-based task prompts,such as reminders to turn off the stove, instructions to make tea, or explanations about kitchen items. All assistance is delivered through non-intrusive voice feedback. Preliminary evaluations indicate the feasibility of this approach for enabling lightweight, wearable, and adaptive support in unstructured indoor environments for cognitively impaired users.