Enhancing Emotion Recognition through Federated Learning: A Dual Neural Network Approach and Firefly Algorithm Optimization

Emotion recognition using multimodal methods, i.e., integrating speech and facial expressions, has garnered considerable interest due to its potential to enhance human-computer interaction, healthcare, and mental health monitoring. Accurate emotion recognition from both auditory and visual inputs is essential for creating intelligent systems that can empathetically interact with users. Furthermore, Federated Learning has also proven to be an important method for decentralized model training, providing privacy-preserving abilities at the cost of slightly lower accuracy. The goal of this study is to build a privacy-preserving, multimodal emotion recognition system combining speech and facial expressions with Federated Learning to enhance emotion detection performance with data privacy protection across distributed clients. This research employs the RAVDESS dataset in speech emotion recognition with a Long Short-Term Memory (LSTM) model for interpreting spoken patterns. Facial expression recognition makes use of the 3D Facial Expression Recognition Data dataset by utilizing a 3D Convolutional Neural Network (3D-CNN) for different facial emotions' classification. These models are developed with Federated Learning, distributed form, so that data is stored locally in the devices. For hyperparameter tuning, Firefly Algorithm (FA) is used to optimize both models' performances. A multimodal approach, where LSTM is used for speech recognition and 3D-CNN is used for facial expression recognition, achieved 99.6% accuracy on the RAVDESS dataset and 99.4% accuracy on the 3D Facial Expression Recognition Data in the proposed framework. The federated model attained a F1-score of 96.8%, a precision of 96.3%, and recall of 95.4%. This demonstrates that we can combine speech and facial expression recognition within a federated learning framework in an E-Learning Application, and future work can be applied to real world applications in healthcare and interactive applications.