Key result
GPT-4o outperforms humans at reading mental states from upright faces but underperforms when inverted.
Why the study?
Whether large language models' ability to reproduce human-like inferences about emotions and mental states extends beyond text to other modalities remains unclear.
GPT-4o demonstrates advanced mental state inference abilities from upright faces but exhibits a highly structured, orientation-dependent error pattern for inverted faces that qualitatively differs from humans.
GPT-4o’s upright advantage with inversion deficits suggests non-human processing; leaves open validation for clinical mental-state inference.
Large Language Models (LLMs) are capable of reproducing human-like inferences, including inferences about emotions and mental states, from text. Whether this capability extends beyond text to other modalities remains unclear. Humans possess a sophisticated ability to read the mind in the eyes of other people. Here we tested whether this ability is also present in GPT-4o, a multimodal LLM. Using two versions of a widely used theory of mind test, the Reading the Mind in Eyes Test and the Multiracial Reading the Mind in the Eyes Test, we found that GPT-4o outperformed humans in interpreting mental states from upright faces but underperformed humans when faces were inverted. While humans in our sample showed no difference between White and Non-white faces, GPT-4o's accuracy was higher for White than for Non-white faces. GPT-4o's errors were not random but revealed a highly consistent, yet incorrect, processing of mental-state information across trials, with an orientation-dependent error structure that qualitatively differed from that of humans for inverted faces but not for upright faces. These findings highlight how advanced mental state inference abilities and human-like face processing signatures, such as inversion effects, coexist in GPT-4o alongside substantial differences in information processing compared to humans.
No takes yet. Share an insight, caveat, or question.
Strachan et al. (2024) studied this question. GPT-4o vs. Human participants was evaluated on Accuracy (fraction correct) on the Reading the Mind in the Eyes Test (RMET) and Multiracial RMET. GPT-4o outperformed humans in interpreting mental states from upright faces on the Reading the Mind in the Eyes Test, but underperformed humans when faces were inverted.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: