What question did this study set out to answer?

The central aim is to develop an effective framework for recognizing emotions from multiple modalities, addressing limitations in existing methods.

July 21, 2026Open Access

Emotion-BIND for multimodal emotion recognition and reasoning

QTQunli TangBeijing University of Posts and Telecommunications QWQi WangBeijing University of Posts and Telecommunications SZShouhao ZhangTsinghua University

Key Points

The central aim is to develop an effective framework for recognizing emotions from multiple modalities, addressing limitations in existing methods.
Introduced Emotion-BIND, integrating ImageBind-based representations and a differentiated positional encoding strategy.
Mapped features from text, audio, images, and video into a shared embedding space.
Applied 2D and 3D RoPE for spatial and temporal modeling across modalities.
Emotion-BIND achieved an Unweighted Average Recall (UAR) of 49.44%, Weighted Average Recall (WAR) of 71.00%, and F1 score of 90.05% on the DFEW dataset.
On the MELD dataset, it recorded a UAR of 55.67%, WAR of 67.88%, and F1 score of 91.02%.
While Emotion-BIND outperformed others in WAR and F1, its UAR was lower than the best comparison model.

Abstract

Constructing Multimodal Emotion Recognition in Conversation (MERC) models is important for understanding affective states from text, audio, images, and video. Existing approaches often rely on linear layers for cross-modal feature alignment and freeze encoder parameters during training, which can introduce feature degradation and modality misalignment. Conventional one-dimensional positional encoding is also limited when modeling visual and temporal information. To address these issues, this study presents Emotion-BIND, a multimodal framework that integrates ImageBind-based representations with a differentiated positional encoding strategy, termed m-ROPE. The framework maps features from multiple modalities into a shared embedding space and applies 2D RoPE to image representations and 3D RoPE to video representations, supporting multimodal spatial and temporal modeling. Experimental results show that Emotion-BIND achieves an Unweighted Average Recall (UAR) of 49.44%, a Weighted Average Recall (WAR) of 71.00%, and an F1 score of 90.05% on the DFEW dataset, with the highest WAR and F1 score among the compared methods, although its UAR remains lower than the best comparison model. On the MELD dataset, Emotion-BIND achieves a UAR of 55.67%, a WAR of 67.88%, and an F1 score of 91.02%. The source code is archived at https://doi.org/10.5281/zenodo.21186484 and is also available on GitHub at https://github.com/qunlitang/Emotion-BIND.

Ask AI

Helpful

Bookmark

View Full Paper