Plural Emotion Readings in Short Social Video Clips: A Descriptive Study

Emotion recognition datasets used to train affective computing systems typically assign each stimulus a single "correct" emotion. This convention rests on a testable assumption: that when two thoughtful readers watch the same social moment and arrive at different emotion labels, one of them is wrong. In a preliminary study with 43 adult participants across seven countries, 442 non-excluded responses were collected on 33 short social video clips (10–30 seconds each). Participants selected among 14 emotions or a "mixed" option to indicate multiple simultaneous emotions, attributed each reading to specific perceptual cues, and rated their confidence. The observations point to two levels of plurality in the human response data. Across readers: on the median clip, the most-picked emotion was chosen by 70% of readers (IQR: 50–76%), with roughly five distinct emotions mentioned per clip; eight of 33 clips (24%) reached the ≥ 80% agreement level used here to define a consensus tier. Within individual responses: 24% of responses used the "mixed" option to select more than one simultaneous emotion, with a mean of 1.42 emotions selected per response, and mixed responses carried nearly the same confidence rating as single-emotion responses (4.24 vs 4.44 on a 5-point scale). Breaking mixed responses down by confidence level, the mixed rate did fall as confidence rose (from 37% at confidence 3 to 20% at confidence 5), but roughly one response in five at the highest confidence level was still mixed. Together the observations indicate that the human response distribution on this corpus is not well represented by a single canonical label, and that single-label representation under-specifies the observed human response distribution on this corpus. The response-level dataset (adult participants) is released to enable replication and reuse. Companion paper (Kim, 2026b — "Where One AI Diverges from Plural Human Readings on Short Social Video Clips") is being deposited separately on Zenodo. Open dataset: https://www.mindlenslab.org/live-findingsPre-registration: https://www.mindlenslab.org/research/hypothesesAnalysis code: https://github.com/evelyn-mindlens/preprint-analysis (MIT license)Project site: https://mindlenslab.org