Textual Context guided Vision Transformer with Rotated Multi-Head Attention for Sentiment Analysis

Social media multimodal sentiment analysis has proliferated the research attention of the research community, as it opens up various paradigms for social issues such as cyberbullying, hate speech, healthcare, politics, business analysis and many more. At the same time, it is an open problem to learn the intrinsic representation of multiple modalities in order to identify correlated patterns. In the proposed work a sentiment analysis framework is presented that defines Textual Context guided Vision Transformer with Rotated Multi-Head Attention, in order to exploit correlation between image-text pair and mine rich discriminatory features for multimodal sentiment analysis. A novel Rotated-Multi-head attention mechanism is defined that translates the visual or text embeddings in distinct feature space resulting in Adaptively Rotated Refined Embedding (ARREmb). To exhibit the performance of the proposed work, extensive experiments are carried out on three publicly available datasets-BG, Twitter and MVSA-single dataset, in terms of Precision, Recall, F1-score and accuracy. The experiments support superior performance of the proposed approach by laying out comparison with SOTA followed by ablation study.