Social Signal Processing for Evaluating Conversations Using Emotion Analysis and Sentiment Detection

Social Signal Processing is vital in discovering the intensity of sentiments and emotions expressed by individuals based on their facial expressions and gestures, phrases and tone of usage. Continuous monitoring of customer satisfaction is an utmost important aspect in industries such as banking and finance. Detecting dissatisfied customers among large data captured during conversations is a challenging task. In this paper, we present a multi-modality framework to analyze customer satisfaction levels, especially in determining dissatisfied customers, using image, speech and text analysis. We employ a two-level synthesis: (a) emotion analysis of speech signals and sentiment detection from text data, which is converted from the same speech, and (b) facial emotion analysis from image data. Our approach helps in better understanding of customers and identification of their exact perspective with respect to services provided by the industries.