Answers are grounded in this paper and cite their sources.
Facial expressions alone are often insufficient to infer emotion: human observers routinely use scene, body and social context to disambiguate facial cues. Recent multimodal frameworks built on vision-language and large language models (VLMs/LLMs) reason jointly over facial and contextual evidence, but they typically process entire videos. This is computationally costly, and the few emotionally expressive frames are diluted among many uninformative ones. We introduce Reasoning from Facial Evidence for Emotion Recognition (ReFER), which focuses multimodal reasoning on the most relevant moments. A Multiple Instance Learning (MIL) module predicts the emotion from an Action Unit (AU) sequence and scores each frame's relevance. A VLM then describes the visual context of the top-scoring frames. Finally, an instruction-tuned LLM fuses the MIL prediction, the AU measurements and the context description into a final label and a natural-language explanation. Beyond accuracy, we analyze which inputs actually drive the fused prediction. Across four in-the-wild video facial expression recognition (FER) benchmarks and five open-weight fusion LLMs, the best fusion models outperform the AU-only MIL baseline on seven of eight dataset-metric pairs, with the largest gains on AFEW and DFEW. Component ablations show that the initial MIL prediction, rather than the contextual description, is the dominant input to the fusion stage. A label-corruption analysis further reveals that all fusion models anchor strongly on the supplied initial prediction, adopting corrupted labels in 82.6%–99.9% of cases. These results show that an explanation consistent with the available evidence does not imply that the evidence faithfully determines the prediction and points to the need for fusion mechanisms explicitly designed to reconcile conflicting multimodal evidence.
No takes yet. Share an insight, caveat, or question.
Gaudio et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: