STEP-FACE: Sequential TExtual–Visual Prompting for multi-task face analysis

Jointly analyzing stable facial attributes (gender, age) and transient affect (emotion) is a useful functionality for applications such as adaptive human–computer interaction, wellbeing screening and security. However, the design of such a solution remains challenging due to data scarcity, label imbalance, missing annotations and task interference in multi-task learning. In this paper, we propose STEP-FACE, a parameter-efficient framework that adapts a Vision-Language Model (VLM) to simultaneous gender, emotion and age recognition. It is based on Sequential TExtual-visual Prompting (STEP), the proposed parameter-efficient procedure which first learns task-specific textual prompts and then uses them to optimize visual prompts, injecting relevant cues for face analysis into the visual encoder. The inference is performed with the visual branch only, reducing memory and latency with respect to standard VLMs. The multi-task learning procedure designed for STEP-FACE further enforces task-representative batching, dealing with missing labels, balancing the task-specific losses and treating age estimation as a classification problem by adopting an ordinal-ranking formulation. The effectiveness of the proposed solution is confirmed by the experimental results obtained on widely used face analysis benchmarks: STEP-FACE achieves state-of-the-art or competitive performance, with 97.8% and 98.3% accuracy for gender recognition on FairFace and VGGFace2, 86.1% for emotion recognition on RAF-DB, and 63.8% and 63.5% for age classification on FairFace and UTKFace, while consistently improving over the considered baselines and remaining competitive with existing single-task and multi-task solutions in a parameter-efficient multi-task VLM setting.