A multimodal sentiment analysis system for recognizing person aggressiveness in pain based on textual and visual information

This article proposes a multimodal sentiment analysis system for recognizing a person’s aggressiveness in pain. The implementation has been divided into five components. The first three steps are related to a text-based sentiment analysis system to perform classification tasks such as predicting the classes into non-aggressive, covertly aggressive, and overtly aggressive classes. The remaining two components are related to an image-based sentiment analysis system. A deep learning-based approach has been employed to do feature learning and predict the three types of pain classes. An aggression dataset for the text-based system and the UNBC-McMaster database for an image-based system has been employed, respectively. Experimental results have been compared with the state-of-the-art methods, showing the superiority of the proposed approach. Finally, the scores due to text-based and image-based sentiment analysis systems are fused to obtain the performance for the proposed multimodal sentiment analysis system.