Multi-modal sentiment analysis lies at the intersection of natural-language processing and multimedia analysis, aiming to unravel complex emotional expressions in multimedia content. This article presents a novel approach to Urdu multi-modal sentiment analysis, focusing on the integration of textual, acoustic, and visual cues to predict sentiment. Our methodology involves a systematic approach to feature extraction from each modality, followed by individual and fused modality sentiment classification. We employ a convolutional neural network (CNN) model integrated with 300-dimensional Embedding FastText to capture meaningful text representations in the textual modality. The acoustic modality utilizes the Librosa library for audio feature extraction, encompassing Mel-frequency cepstral coefficients (MFCCs), intensity, pitch, and loudness. We utilize three-dimensional convolutional neural networks (3D-CNNs) to extract spatial and temporal features from videos for the visual modality. We explore feature- and decision-level fusion techniques to combine the strengths of individual modalities. The results highlight the effectiveness of the fused approach, achieving an accuracy of 91.18%. Our findings underscore the importance of leveraging multiple modalities for comprehensive sentiment analysis, opening avenues for applications in social media sentiment assessment, content recommendation, and market sentiment evaluation. The proposed framework not only contributes to the advancement of Urdu sentiment analysis but also serves as a stepping stone for further research in multilingual and cross-modal sentiment analysis, thereby enriching our understanding of emotions expressed in multimedia content.
Butt et al. (Wed,) studied this question.