Evaluating Multimodal Wearable Sensors for Quantifying Affective States and Depression With Neural Networks

With the increasing proliferation of embedded sensors in wearable devices, there is potential for modeling individual emotional and mental state variations. The popular measure for the quantification of emotions outlines the affective states of arousal and valences, with high and low being the discrete categories of interest. Recent works explore the discernability of digital behavior differences between groups with and without mental disorders. However, the interaction between physiological states and affective states within a predominantly depressive population remains to be studied with the aid of wearables. Despite the pervasiveness of emotional state inference through the tracking of ubiquitous physiological trackers such as heart rate, blood volume pulse, skin conductance, and motion, a dearth of work is noted in the exploration of physiological markers in single and multimodal settings. This work provides an extensive evaluation of a convolutional neural network with an attention mechanism ensembled with a random forest algorithm to effectively leverage multiple raw signal-to-image transformations as feature inputs to predict depression severity and affective state. The proposed models are assessed on the Daily Ambulatory Psychological and Physiological recording for Emotion Research (DAPPER) dataset, and achieve the sensitivity: specificity scores of 58.75%:45.59%, 62.34%:43.41%, 49.43%:51.70% for predicting depression, valence, and arousal with a mixture of uni- and bi- modality applying Continuous Wavelet Transforms and Short-time Fourier Transform to motion skin-conductance readings, respectively. This work is envisioned as a preliminary study to contribute towards the monitoring of affective states among a depressed population by utilizing low-frequency sensor recordings with the DAPPER dataset.