How Is Your User Feeling? Inferring Emotion Through Human-Computer Interaction Devices1

Emotion can influence important user behaviors, including purchasing decisions, technology use, and customer loyalty. The ability to easily assess users’ emotion during live system use therefore has practical significance for the design and improvement of information systems. In this paper, we discuss using human–computer interaction input devices to infer emotion. Specifically, we utilize attentional control theory to explain how movement captured via a computer mouse (i.e., mouse cursor movements) can be a real-time indicator of negative emotion. We report three studies. In Study 1, an experiment with 65 participants from Amazon’s Mechanical Turk, we randomly manipulated negative emotion and then monitored participants’ mouse cursor movements as they completed a number-ordering task. We found that negative emotion increases the distance and reduces the speed of mouse cursor movements during the task. In Study 2, an experiment with 126 participants from a U.S. university, we randomly manipulated negative emotion and then monitored participants’ mouse cursor movements while they interacted with a mock e-commerce site. We found that mouse cursor distance and speed can be used to infer the presence of negative emotion with an overall accuracy rate of 81.7 percent. In Study 3, an observational study with 80 participants from universities in Germany and Hong Kong, we monitored mouse cursor movements while participants interacted with an online product configurator. Participants reported their level of emotion after each step in the configuration process. We found that mouse cursor distance and speed can be used to infer the level of negative emotion with an out-of-sample R2 of 0.17. The results enable researchers to assess negative emotional reactions during live system use, examine emotional reactions with more temporal precision, conduct multimethod emotion research, and create more unobtrusive affective and adaptive systems.