Decoding Visual Attention: from 3D Gaze to Social Gaze inference in Everyday Scenes

Beyond words, non-verbal behaviors (NVB) are known to play important roles in face-to-face interactions. However, decoding non-verbal behaviors is a challenging problem that involves both extracting subtle physical NVB cues and mapping them to higher-level communication behaviors or social constructs. This is particularly the case for gaze, one of the most important non-verbal behaviors with functions related to communication and social signaling. Indeed, as a display of attention and interest, gaze is a fundamental cue in understanding people's activities, behaviors, and state of mind, and plays an important role in many applications and research fields, such as the design of intuitive human-computer or robot interfaces, or for medical diagnosis, like assessing Autism Spectrum Disorders (ASD) in children. However, decoding the visual attention of others —that is, estimating their gaze (3D line of sight) and Visual Focus of Attention (VFOA)— is a challenging task, even for humans. It often requires not only inferring an accurate 3D gaze direction from the person's face and eyes but also understanding the global context of the scene (what is going on, where are people, are people interacting, where are salient items), including the person’s activity and situation as well as the scene structure to detect obstructions in the line of sight or apply attention priors that humans typically have when observing others. Due to this duality, two lines of research have been followed in recent years. The first one focused on improving appearance-based 3D gaze estimation from images and videos, while the second investigated gaze following —the task of estimating the 2D pixel location of where a person looks in an image. In this presentation, I will discuss different methods that address the two cases mentioned above. I will first focus on several methodological ideas on how to improve 3D gaze estimation, including approaches to build personalized models through few-shot learning and gaze redirection eye synthesis, differential gaze estimation, or taking advantage of priors on social interactions to obtain weak labels for model adaptation. In the second part, I will introduce recent models aiming at estimating gaze targets in the wild, the first one taking advantage of different modalities and applicable to privacy-sensitive settings, and the second one leveraging new depth estimation methods yielding geometry-preserving point clouds and avoiding distorted reconstructed 3D scenes. I will discuss current limitations and directions for future research. This research was suported by the Swiss National Science Foundation.