Dynamic FERNet: Deep learning with optimal feature selection for face expression recognition in video

Summary This work concentrates on the recognition of facial emotion from video sequences with deep learning. Once the input video is converted into frames, the face detection is performed on each frame using the viola–jones face detection algorithm. Then, the feature extraction is performed by three well‐performing feature extraction techniques like modified local directional pattern, spatio‐temporal features, and scale‐invariant feature transforms. The extracted features from all the frames of the video are concatenated. To reduce the feature‐length for decreasing the training complexity, and enhance the recognition performance, the optimal feature selection is accomplished with the distance‐based tunicate swarm algorithm. These selected features are processed to an innovative deep learning model termed a heuristically modified recurrent neural network. The same D‐TSA improves the performance of RNN by optimally tuning its hidden neurons. Experimental results on a widely used benchmark dataset and manually collected dataset show that the classification performance is improved using spatio‐temporal features, SIFT, M‐LDP, and optimal feature selection, and thus, the proposed model with HM‐RNN outperforms the other existing models.