Arousal‐valence recognition using CNN with STFT feature‐combined image

A novel ocular‐features‐combining method, called short‐time Fourier transform (STFT) feature‐combined image, and a simple convolutional neural networks (CNNs) model are proposed for arousal‐valence recognition. The STFT feature‐combined image aims to represent information on two ocular features (pupil size and eye movements) as a single image. The CNN model consists of two convolutional layers and uses STFT feature‐combined image as an input. The experimental results demonstrate the effectiveness of the proposed method, and show that CNN model is not only effective for emotion‐recognition methods based on other modalities, but also effective for ocular‐feature‐based emotion recognition.