Formulating emotion perception as a probabilistic model with application to categorical emotion classification

Automatic recognition of emotions is an important part of affect-sensitive human-computer interaction (HCI). Expressive behaviors tend to be ambiguous with blended emotions during natural spontaneous conversations. Therefore, evaluators disagree on the perceived emotion, assigning multiple emotional classes to the same stimuli (e.g., sadness, anger, surprise). These observations have clear implications on emotion classification, where assigning a single descriptor per stimuli oversimplifies the intrinsic subjectivity in emotion perception. This study proposes a new formulation, where the emotional perception of a stimuli is a multidimensional Gaussian random variable with an unobserved distribution. Each dimension corresponds to an emotion characterized by a numerical scale. The covariance matrix of this distribution captures the intrinsic dependencies between different emotional categories. The process where an evaluator judges the stimuli is equivalent to sampling a point from this distribution, reporting the class with the highest value. The proposed approach recursively estimates this multimodal distribution using numerical methods. The mean of the Gaussian distribution is used as a soft label to train a deep neural network (DNN). Our experimental results show that the proposed training method leads to improvements in F-score over training with (1) hard-labels based on majority vote, and (2) soft-label framework proposed by other studies.