ABSTRACT Emotion recognition through multi‐modal techniques uses multiple formats of input data in a significant area of research. However, existing studies face challenges related to the extraction of high‐level emotional features and increased model complexity. Thus, this paper introduces Multi‐modal data. Initially, text, video, and audio data are collected from BAUM 1 and Enterface05 datasets to classify emotions. Audio files undergo preprocessing with the Pass Gaussian Filter (PGF), while video clips are transformed into key frames using Spherical Interpolation based Q‐learning (SIQ). The text data are preprocessed through tokenization and stemming. The features are extracted using a casual neural network for audio data, geometric feature calculations (GFC) for video data, and an improved term frequency‐inverse document frequency (ITF‐IDF) model for text data. The features are selected by the enhanced genetic gray lag goose optimization (EGG‐LGO) method for audio data, the parrot optimization algorithm (POA) for video data, and the Adapted Firefly Optimization Algorithm (AFOA) for text data. The densely connected recurrent network with dual attention (D‐RNA) model classifies emotions from audio data. Text emotions are classified using the self‐attention based capsule‐bi‐directional long short term memory (SA‐CBiLSTM) model, and emotions from visual data are classified using the gated attention enclosed residual context aware transformer (GRCAT). Finally, text, visual, and audio modality outputs are fused by a decision‐level strategy to obtain the final output. The BAUM‐1 dataset achieves accuracies of 99% for video, 99.3% for audio, and 98.4% for text data. The Enterface05 attains 98.18% accuracy for video, 98.75% for audio, and 97.7% for text.
Tarik et al. (Sun,) studied this question.