Deep learning framework for subject-independent emotion detection using wireless signals

Figures

• 

  Fig 1. Emotion detection process in which each participant is asked to watch emotion evoking videos on the monitor while being exposed with radio waves. The Tx antenna is used to transmit RF signals towards the participant, whereas Rx antenna is used to receive RF reflections off the body. The ECG monitor is also connected to a participant’s chest for recording heart beats. The data received from ECG is used to correlate heart beats variations with emotion evoking videos.

• 

  Fig 2. Proposed deep neural network architecture for emotion classification. Time domain RF signal is processed through two convolutional-1D layers and an additional LSTM cell that captures the time dependency (section 1 in S1 File). The CW transformation is processed by two convolutional-2D layers (section 2.1 in S1 File). Each feature map in convolutional layers represents a unique extracted feature from the layer input. The features extracted from two distinct inputs of the model are then concatenated, leading to a broad learning capability. The detailed visualization of 32 and 64 features maps is presented in section 2.2 of S1 File.

• 

  Fig 3. Confusion matrices obtained by LOOCV for DL and ML models.

• 

  Fig 4. t-SNE plots representing the full RF and ECG databases. The plots were obtained by reducing the dimensions of the continuous wavelet images of each signal. It can be observed that the wavelet images of RF signals (panel (a)) demonstrate a better separability between emotions than that of ECG signals (panel (b)).

• 

  Fig 5. ROC-AUC representing the degree of separability between classes. The emotions ‘Disgust’ and ‘Relax’ are highly separable from the rest. Micro-average aggregates the contribution from all classes to compute the average ROC curve. Macro-average computes the ROC metric for each class independently and takes the average, hence treating all classes equally.

• 

  Fig 6. Variations of probability of different emotions over time, predicted for a randomly selected subject from the test set. Smooth probability curves are generated by interpolating the discrete probability values.

• 

  Fig 7. CW transformed ECG inputs fed to the CNN model and the confusion matrices representing the ECG classification performances of CNN and SVM ML models.