Emotion Recognition in Adaptive Virtual Reality Settings: Challenges and Opportunities

Recently, there has been a notable surge of interest in Emotion Recognition (ER) systems, primarily due to their potential in improving interactions between humans and computers. Meanwhile, Virtual Reality (VR) has emerged as a groundbreaking technology that is also capable of transforming humancomputer interaction through the simulation of immersive and flexible environments. The integration of ER into VR holds great promise for further advancing human-computer interaction by allowing the virtual environment to adapt to the user’s emotional state. This adaptive VR setting is particularly relevant in fields such as education and gaming, where there is often the need to adapt the content to a person’s emotions. However, applying traditional ER systems to adaptive VR settings comes with several challenges. In this paper, we identify the key differences between traditional ER and ER performed in VR environments. Specifically, we argue that the two scenarios primarily differ in terms of data collection methodologies and handling multimodality. After reviewing the main modalities considered in ER, and describing existing datasets, we delve into the challenges associated with these factors, highlighting the limitations of using traditional datasets in adaptive VR settings, and the fact that traditional ER models are not designed to effectively handle the multiple modalities arising from the VR setting. In addition to discussing these challenges, we also explore unique opportunities that arise from overcoming them. These opportunities include acquiring diverse datasets, eliciting genuine emotional responses, and exploiting multiple data modalities.