Commonalities and variations in emotion representation across modalities and brain regions

Abstract Humans express emotions through various communication modalities such as facial expressions and natural language. However, there are still many uncertainties regarding the relationships between multiple emotions expressed through different modalities and their correlations with neural activities in diverse brain regions. First, we calculated the representations of various emotions as multi-dimensional vectors of data from three different modalities (visual, language, and visio-linguistic modalities) and used Representational Similarity Analysis to compare similarities between modalities. Second, we examined the similarity between the emotion representation of each modality and representations derived from brain activities across 360 regions. Third, we investigated the linear transferability of emotion representation from vision to other modalities. Our results revealed: 1) the representational structures of emotions expressed in different modalities share commonalities with modality-specific variations; 2) the emotion representations in different solo modalities showed relatively higher similarity with representations in different partial brain regions, while multi-modal emotion representation was most similar to representations across the entire brain region; and 3) emotion representations can be linearly mapped onto each other. These findings suggest that emotional experiences are represented differently in each brain region with varying degrees of similarity across different modalities, and that they may be cross-modally conveyable through different modalities.