Emotional Expression in Different Languages: Cross-Lingual Model Transfer

The task of Speech Emotion Recognition has recently gained in popularity, mainly due to the increase of the available data and the development of the new machine learning algorithms. Currently, the most common approach to solving this task is to use transformer-based speech models, with the goal to determine whether the phrase belongs to one of 4-9 basic emotions. However, the way people express even these basic emotions can differ based on their native language and cultural background. This paper analyses the capabilities of modern machine learning models, trained on emotional speech data from different languages, with the main goal to examine the effect of language’s innate linguistic features on a model’s behavior. For that, we compared the results of a few models, trained on the emotional data of different languages and transferred to the Russian language. We trained the HuBERT speech model on the task of speech emotion recognition, using training data from nEMO, ESD and JVNV emotional speech datasets, which correspond to Polish, Chinese and Japanese languages respectively. The trained models then were tested on Russian emotional speech dataset Dusha. The obtained results show prosodic features being the main contributor to the model’s performance, while differences in syntax generally not affecting it.