Evaluation of Multimodal Face and Voice Classification of Emotion-Derived Negative-Affect Labels

This paper presents an end-to-end prototype that classifies emotion-derived negative-affect labels from facial video and speech audio. Clips from two acted-emotion corpora, CREMA-D and RAVDESS, are combined and mapped to a binary target: angry, disgust, and fearful clips are labeled 1, and happy, neutral, and calm clips are labeled 0. Because both corpora annotate acted expression rather than physiological or self-reported stress, the target is a proxy, and the results describe classification of that proxy. A facial branch (ResNet18 embeddings with a multilayer perceptron) and a voice branch (240 statistics derived from 40 MFCCs and their deltas, with logistic regression) are evaluated separately and under weighted late fusion (0.40 face, 0.60 voice). Splits are grouped by actor. On a held-out test set of 743 clips, late fusion achieved 78.33% accuracy, 82.85% F1-score, 86.44% recall, and 0.8522 ROC-AUC, compared with 76.04% accuracy for the face branch and 67.29% for the voice branch. The fusion gain over the face branch is clear on the CREMA-D test subset but reverses on the RAVDESS subset, and the accuracy difference between fusion and face-only is not established as significant by the analyses reported here. The system also displays outputs from two pretrained visible-expression models. These outputs are not stress evidence, and the application’s expression-priority rule is not evaluated in this paper.