A comprehensive survey on multimodal sentiment analysis: Approaches, challenges and open issues

Multimodal sentiment analysis (MSA) aims to extract affective knowledge from various types of input media, including image, audio, and text. Given the significant impact of leveraging the vast amount of multimodal data from social networks and of fusing and integrating information extracted from different modalities on enhancing the validity and accuracy of the sentiment analysis process, MSA has become an important, fascinating, and widespread topic. The MSA methods presented in the literature exploit the latest advances in deep learning and transformers for multimodal feature extraction, fusion, classification, and polarity identification to enhance efficiency and performance. Despite various research having been conducted and recent articles presented with diverse approaches for MSA, the lack of a comprehensive study that examines several aspects of the problem addressed in these studies, including approaches, paradigms, methods, tools, and techniques, and evaluates them by considering related issues such as the datasets and benchmarks used, criteria and parameters for measuring results, and the extent to which intermediate and final objectives are addressed and achieved, is noticeable. In this paper, MSA and its various processes and phases are addressed within a comprehensive study framework. In this regard, techniques, tools, and datasets for feature extraction, multimodal fusion, and sentiment classification are thoroughly reviewed, categorized, summarized, compared, and evaluated, and for better comparison, some evaluation factors are proposed at each stage. Furthermore, a comprehensive overview of existing challenges, open research topics, and future research directions is provided along with a set of guidelines for conducting future studies.