August 14, 2020SensorsOpen Access

CNN and LSTM-Based Emotion Charting Using Physiological Signals

View Full Paper

Ask AI

Bookmark

Why the study?

Using physiological signals like EEG, ECG, and GSR poses challenges for performance improvement across broader emotion classes in less constrained real-world environments.

Structured PICO

Population

Two publicly available datasets (DREAMER and AMIGOS) containing physiological signals (EEG, ECG, GSR) from low-cost, wearable sensors

Intervention

Computational framework using 2D Convolutional Neural Network (CNN) for 14 channels of EEG, and a combination of Long Short-Term Memory (LSTM) and 1D-CNN for ECG and GSR

Comparator

State-of-the-art approaches

Outcome

Emotion classification accuracy into four classes (High Valence-High Arousal, High Valence-Low Arousal, Low Valence-High Arousal, and Low Valence-Low Arousal)surrogate

A multi-modal deep learning framework combining CNN and LSTM architectures on physiological signals (including ECG) achieves high accuracy for emotion classification.

Implication

Enables real-world multimodal emotion classification; leaves open prospective clinical validation before adoption.

Abstract

Novel trends in affective computing are based on reliable sources of physiological signals such as Electroencephalogram (EEG), Electrocardiogram (ECG), and Galvanic Skin Response (GSR). The use of these signals provides challenges of performance improvement within a broader set of emotion classes in a less constrained real-world environment. To overcome these challenges, we propose a computational framework of 2D Convolutional Neural Network (CNN) architecture for the arrangement of 14 channels of EEG, and a combination of Long Short-Term Memory (LSTM) and 1D-CNN architecture for ECG and GSR. Our approach is subject-independent and incorporates two publicly available datasets of DREAMER and AMIGOS with low-cost, wearable sensors to extract physiological signals suitable for real-world environments. The results outperform state-of-the-art approaches for classification into four classes, namely High Valence-High Arousal, High Valence-Low Arousal, Low Valence-High Arousal, and Low Valence-Low Arousal. Emotion elicitation average accuracy of 98.73% is achieved with ECG right-channel modality, 76.65% with EEG modality, and 63.67% with GSR modality for AMIGOS. The overall highest accuracy of 99.0% for the AMIGOS dataset and 90.8% for the DREAMER dataset is achieved with multi-modal fusion. A strong correlation between spectral- and hidden-layer feature analysis with classification performance suggests the efficacy of the proposed method for significant feature extraction and higher emotion elicitation performance to a broader context for less constrained environments.