Multi-emotion and intensity-driven response generation for richer multimodal dialogue

The combination of well-chosen words becomes a thought, although the thought attains meaningful significance only when spoken with emotion. The significance of every sentence could vary depending on the emotions behind it. As AI speech agents evolve, understanding and integrating human emotions is key to fostering natural and impactful interactions. Recently, researchers have given preference to the development of communication systems that are tuned to emotions. In recent work, our focus is on generating multi-emotional responses with precisely controlled intensity by integrating essential insights from text, audio, and visual elements in multimodal dialogues. In this regard, the authors designed a Multimodal Multi Emotion and Intensity-leaded Deliberation Decoder (MMEI-DD) framework that encodes the multimodal knowledge using the Transformer network and captures the inter-modality representation. The desired emotion and corresponding intensity are provided during the two-step decoding mechanism to the deliberation decoder to generate multi-emotion and intensity guided responses. The authors expanded the multimodal feature set (MEIMD + +) based on the recently introduced MEIMD (Multiple Emotion Intensity-Aware Multi-party Dialogue) dataset and demonstrate that integrating multimodal knowledge enhances the performance and reliability of our proposed approach compared to the existing method. In this paper our proposed model leverage to support further research, the necessary resources and code will be made publicly available.