Fuzzy logic-driven confidence aggregation for multimodal sentiment classification

Computational intelligence focuses on intelligent computer systems that mimic human nature and linguistic reasoning. Sentiment analysis is an area of considerable relevance within computational intelligence. Multimodal sentiment analysis is an extension of textual sentiment analysis, where the sentiments of people’s opinions are analysed by including multimedia content in addition to textual content. This mode of sentiment analysis faces multiple problems, as the sentiments of text and multimedia content may be contradictory. In addition, another added factor is the imbalance of the data that these problems suffer from in certain topics, which causes a problem when generating intelligent models. In this paper, we design a novel approach for multimodal sentiment analysis, proposing a new way of labelling tweets, not always prioritising polarized classes but using annotator confidence. Moreover, during this design, an information integration and fusion methodology is proposed for the construction of a metamodel that includes fuzzy logic to perform information weighting according to the confidence of the annotator. This proposal has been applied a public unbalanced dataset of tweets with text and images, with a large unbalance towards the negative class label. Applying the proposed fuzzy methodology, we reached a macro-F1 score of 0.493 for the negative class, 0.681 for the neutral class, and 0.832 for the positive class. The model obtains satisfactory performance since the individual image and text sentiment analysis results are worse, especially the negative class, which in initial image classification achieves an F1 score of 0.08.