Detecting Moral Emotions with Facial and Vocal Expressions: A Multimodal Emotion Recognition Approach

Altruism is an innate aspect of human virtue and represents a capability that humans have evolutionarily developed as social beings. Studies have identified moral emotions, such as compassion and elevation, as the mechanisms behind altruistic motivation and behavior. While research on moral emotions has drawn much attention, the difficulty in measuring these subtle emotions has limited both research and real-world applications. Advances in multimodal emotion recognition technology (MMER) have shed light on these challenges. This study aims to bridge the gap in the literature by proposing the use of facial and vocal expressions to detect moral emotions. We conducted an experiment with one hundred and four participants and randomly assigned them into groups to investigate how feelings of compassion and being moved can influence altruistic behavior. This study employed the Random Forest (RF) model to differentiate people with different emotional responses and used SHapley Additive exPlanations (SHAP) analysis to provide better insights into the results. Our data showed that using facial and vocal expressions to detect hidden moral emotions is feasible; however, the prediction accuracies were below 0.7 and much lower than expected. Our results highlight the difficulty in detecting moral emotions and the limitation of using a simple data fusion strategy to boost prediction accuracy. The possible causes for the failure to achieve satisfactory results are discussed before concluding this article.