A Novel Cross Modal Fusion Based Cascaded Capsule Autoencoder Network for Multi‐Modal Sentiment Analysis

Multi‐modal sentiment analysis (MSA) enhances human‐computer interaction by allowing machines to know and respond to human emotions. However, processing text, audio and visual data is challenging due to the high computational demands of recent deep learning approaches. The recent multi‐modal deep learning algorithms have faced various issues, such as being resource‐intensive, struggling to capture complex relationships between each modality and having limited contextual understanding. To resolve these issues, this study proposes a novel Cross modal Fusion based Cascaded Capsule Autoencoder Network (CMF‐C2AEN) for effectively categorizing sentiments from the multi‐modal inputs. Initially, the input multi‐modalities like text, audio and video are gathered from the publicly accessible CMU‐MOSI dataset. Then, text pre‐processing, text feature extraction, audio and video feature extraction are enabled to improve the quality and significance of multi‐modal feature representations for obtaining accurate outcomes. Finally, the fetched features are fused using cross modal feature fusion network, and the sentiments are analyzed by using a cascaded capsule autoencoder model. The experimental result evaluation illustrates that the proposed model obtains higher performance than other existing methods in varied metrics like accuracy 98.30%, precision 93.90%, recall 98.00%, and F1‐score 95.90%. Also, the attained results prove that the proposed study mitigates the complexity issues and enhances the feature learning abilities.