A Multi-modal Approach for Emotion Recognition of TV Drama Characters Using Image and Text

Research on facial emotion recognition has long been popular for various purposes. This paper investigates the recognition of the character emotions, to assist in understanding the story. The goal of this research is to classify the facial images of the characters in the Korean TV series `Misaeng: The Incomplete'1 into 7 emotions: Angry, Disgust, Fear, Happy, Neutral, Sad, and Surprise. We built a multi-modal deep learning model which utilizes facial images as well as textual information describing the situations, to classify the facial images. Our experiments indicate that employing multi-modality enhances the performance of facial emotion recognition of story characters.We concludes with discussions and future work.