Multi-modal data fusion using deep Convolutional Neural Networks estimated positive affect with a correlation of 0.69 (P<0.05) compared to 0.59 with traditional machine learning.
Does deep learning with multi-modal physiological data fusion improve the estimation of positive and negative affects compared to traditional machine learning?
Deep learning applied to raw physiological signals from wearable sensors can estimate positive and negative affects with higher correlation than traditional machine learning methods using hand-crafted features.
Effect estimate: Correlation
Absolute Event Rate: 0.69% vs 0.59%
p-value: p=<0.05
Automated momentary estimation of positive and negative affects (PA and NA), the basic sense of feeling, can play an essential role in detecting the early signs of mood disorders. Physiological wearable sensors and machine learning have a potential to make such automated and continuous measurements. However, the physiological signals' features that are associated with the subject-reported PA or NA may not be known. In this work, we use data-driven feature extraction based on deep learning to investigate the application of raw physiological signals for estimating PA and NA. Specifically, we propose two multi-modal data fusion methods with deep Convolutional Neural Networks. We use the proposed architecture to estimate PA and NA and also classify baseline, stress, and amusement emotions. The training and evaluation of the methods are performed using four physiological and one chest motion signal modalities collected using a chest sensing unit from 15 subjects. Overall, our proposed model performed better than traditional machine learning on hand-crafted features. Utilizing only two modalities, our proposed model estimated PA with a correlation of 0.69 (P<; 0.05) vs. 0.59 (P<; 0.05) with traditional machine learning. These correlations were 0.79 (P<; 0.05) vs. 0.73 (P<; 0.05) for NA estimation. The best emotion classification was achieved by the traditional method with 79% F1-score and 80% accuracy when all the four physiological modalities are used. This is while with only two modalities, the deep learning achieved 78% F1-score and 79% accuracy.
Hssayeni et al. (Fri,) conducted a other in Affect estimation and emotion classification (n=15). Multi-modal data fusion with deep Convolutional Neural Networks vs. Traditional machine learning on hand-crafted features was evaluated on Estimation of positive affect (PA) (Correlation, p=<0.05). Multi-modal data fusion using deep Convolutional Neural Networks estimated positive affect with a correlation of 0.69 (P<0.05) compared to 0.59 with traditional machine learning.