Emotion Sensing for Mobile Computing

Emotion is a complex mental state that guides people's thoughts and behaviors. The ability to recognize one's own emotions and to understand others' emotions helps people manage their personal lives and social relations more successfully. Meanwhile, mobile and wearable devices have become indispensable tools for most of the population. They provide great opportunities for building emotional intelligent applications. In this article, we introduce how mobile and wearable devices work as “emotion sensors” by leveraging their sensing, computing, and communication capabilities, which help monitor people's mental health, facilitate social interactions, and improve user experience. We first describe a general emotion sensing framework that consists of sensing, inferring, and responding steps. After reviewing the state-of-the-art systems on this topic, we summarize and discuss various sensing modalities enabled by mobile and wearable devices. We then describe the widely used inferring procedures and methods. Additionally, we present three solutions of facial expression recognition on smartphones as a case of emotion inference. Finally, we introduce emerging emotion sensing applications, and discuss challenges and opportunities.