Dynamic Stress and Emotional Health Management Platform Based on the Fusion of Multimodal Music Emotion Perception and Wearable Physiological Signals

This is the authors' abstract. We don't add key points for this paper.

In order to solve the problems that existing wearable emotion monitoring methods have in continuously describing the stress assessment process and lack the ability to perform dynamic emotion regulation and feedback, this study proposes the Multimodal Stress-aware Cross-modal Emotion Network (MS-CMENet). A multimodal continuous emotion dataset, including Electroencephalography (EEG), Electrocardiography (ECG), Electromyography (EMG), and music stimulation, is constructed from data collected from 240 participants. Then, the models performance is verified in a 40-min stress-induction and music-intervention experiment. This study uses a unified time axis to dynamically model multimodal physiological signals; it realizes continuous tracking of pressure index [Formula: see text], valence state vector [Formula: see text], heart rate variability [Formula: see text], arousal state vector [Formula: see text], and frontal EEG Beta relative power [Formula: see text]. The results reveal that in the stress induction stage, [Formula: see text] increases from 0.25 to 0.68, while it decreases to 0.42 in the music intervention stage; [Formula: see text] drops from 50.20 to 35.64[Formula: see text]ms and then recovers to 40.80[Formula: see text]ms. Frontal [Formula: see text] adds from 0.21 to 0.37 and then declines to 0.28. Repeated measures analysis of variance indicates that all indicators have significant stage main effects ([Formula: see text]); the pressure index shows the highest effect size (partial [Formula: see text]). The findings show that MS-CMENet can stably achieve multimodal emotion perception, dynamic stress assessment, and tracking of music intervention effects.