Music therapy is also an efficient non-pharmacological therapeutic approach to emotional regulation and mental health rehabilitation, and yet the clinical practice is still based on subjective judgments of therapists. Objective, automated emotion recognition could enhance the accuracy, flexibility, and scalability of therapy sessions by supplementing clinical judgment with data-driven affective feedback. This paper presents a multimodal deep learning framework for music-induced emotion recognition that integrates music-induced affect modeling with physiological affect representation learning in a staged, cross-dataset design. The structure includes two branches: an audio emotion encoder, trained on the DEAM and PMEmo 2018 datasets to continuously predict valence–arousal from musical stimuli, and a physiological affect encoder, trained on the DEAP and AMIGOS datasets using TorchEEG to classify emotions from EEG and peripheral signals. A cross-modal attention fusion module aligns and combines the learned representations from both branches into a shared embedding space for joint emotion inference. To be methodologically transparent, the framework employs a two-stage experimental approach: Stage 1 focuses on learning music emotion representations from audio datasets, and Stage 2 focuses on physiological affect learning and cross-dataset fusion validation. Importantly, since there is no publicly available synchronized clinical music therapy dataset, the two branches, audio and physiological, are trained on separate public proxy datasets and do not use audio from real clinical music therapy sessions. Therefore, the reported cross-dataset transfer validation should be considered to be a cross-platform level validation of multimodal integration in proxy conditions rather than direct transfer from proxy datasets to real clinical music therapy conditions. Experimental findings indicate that the proposed attention-based intermediate fusion method outperforms unimodal baselines and early- and late-fusion approaches in terms of accuracy, F1-score, and concordance correlation coefficient. Ablation and cross-dataset studies also testify to the strength of the suggested multimodal design.
Feng Liu (Wed,) studied this question.