EmotionVibe: Human Emotion Recognition Through Footstep-Induced Floor Vibrations

Emotion recognition is critical for various applications, including the early detection of mental health disorders and emotion-based smart home systems. Previous studies utilized various sensing methods for emotion recognition, such as wearable sensors, cameras, and microphones. However, these methods are often intrusive or raise significant privacy concerns, which may reduce user acceptance for continuous, long-term deployment. This paper introduces a non-intrusive and privacy-friendly personalized emotion recognition system, EmotionVibe, which leverages footstep-induced floor vibrations for emotion recognition. The main idea of EmotionVibe is that individuals' emotional states influence their gait patterns, subsequently affecting the floor vibrations induced by their footsteps. However, there are two main research challenges: 1) the complex and indirect relationship between human emotions and footstep-induced floor vibrations and 2) the large between-person variations within the relationship between emotions and gait patterns. To address these challenges, we first empirically characterize this complex relationship and develop an emotion-sensitive feature set including gait-related and vibration-related features from footstep-induced floor vibrations. Furthermore, we personalize the emotion recognition system for each user by calculating gait similarities between the target person (i.e., the person whose emotions we aim to recognize) and those in the training dataset and assigning greater weights to training people with similar gait patterns in the loss function. We evaluated our system in human walking experiments with 20 participants, summing up to 37,001 footstep samples. EmotionVibe achieved the mean absolute error (MAE) of 1.11 and 1.07 for valence (unpleasant to pleasant) and arousal (calm to excited) score estimations, respectively, reflecting 19.0% and 25.7% error reduction compared to the baseline method (using only gait-related features without personalization).