What question did this study set out to answer?

To enhance the detection of sarcasm in multimodal contexts by addressing inconsistencies and contradictions across different modalities.

April 22, 2026Computers, materials & continua/Computers, materials & continua (Print)0 citationsOpen Access

Quantum-Inspired Complex-Valued Fusion Framework: Optimizing Intra-Modal Semantics and Inter-Modal Fusion in Multimodal Sarcasm Detection

DZDong ZhangChinese Academy of Sciences LSLianhe ShaoXi'an Polytechnic University WXWeijie Xu

Puntos clave

To enhance the detection of sarcasm in multimodal contexts by addressing inconsistencies and contradictions across different modalities.
Proposed a quantum-inspired complex-valued multimodal feature representation method.
Developed an asymmetric quantum interference fusion mechanism to model interactions between text, visual, and audio modalities.
Conducted experiments on Mustard and Memotion datasets to validate the framework's effectiveness.
Achieved a 3.71% and 2.74% increase in F1-score compared to M2Seq2Seq and SRLM on the Mustard dataset.
Noted performance improvements of 0.28% and 0.83% on the Memotion dataset relative to the same benchmarks.
Confirmed the model's key modular effectiveness through ablation experiments.

Resumen

With the popularization of multimodal content on social media, accurately identifying sarcastic intent is of great significance for understanding public attitudes and grasping public opinion trends. However, sarcastic expressions rely on context, exhibit inconsistencies in multimodal information, and have implicitly contradictory semantics. These characteristics pose challenges to traditional single-text modality methods. Existing multimodal methods, due to their default assumption of symmetric modal interactions and difficulty in capturing the subtlety of sarcasm and modal contradictions, yield limited detection performance. Therefore, this paper proposes a quantum-inspired complex-valued fusion framework to optimize the intra-modal semantics and inter-modal fusion in multimodal sarcasm detection. Firstly, this framework constructs a quantum-inspired complex-valued multimodal feature representation method. It embeds the text, visual, and audio modalities into the complex-valued Hilbert space, and models the feature intensity and directional information, respectively, through the two dimensions of “amplitude-phase”, providing highly expressive basic features for fusion. Secondly, an asymmetric quantum interference fusion mechanism is designed. Based on the principle of quantum interference, a directional interference term and trainable parameters are introduced to accurately capture the asymmetric interaction relationship between modalities, where “text dominates semantic interpretation and vision supplements detailed evidence”, effectively mining the modal contradictions on which sarcasm depends. Experimental results show that the F1-score of the proposed model has increased by 3.71% and 2.74% compared with M2Seq2Seq and SRLM, respectively, on the Mustard dataset. On the Memotion dataset, it also achieves performance improvements of 0.28% and 0.83% relative to M2Seq2Seq and SRLM. The effectiveness of the key modules in the model is also verified through ablation experiments.

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo