CamWave-Emo: Advanced Contactless Emotion Detection from Facial Expression Analysis Using Camera Labeling and mm-wave Sensing

This study introduces a novel dual-modality emotion recognition system that combines mm-wave radar with camera-based labeling to provide accurate and privacy-preserving emotion detection. The mm-wave radar captures subtle physiological signals through micro-Doppler and time-frequency characteristics, while the camera assists in labeling facial expressions. The radar data is transformed into spectrograms, which are then fused with camera datasets to train deep learning models, employing convolutional layers for feature extraction and recurrent layers for temporal pattern recognition. Performance evaluation, conducted across a wide range of real-world occlusion and interference scenarios, shows that the system achieves 98.5% accuracy, 0.98 F1-score, and 0.98 recall, significantly outperforming traditional systems. Other experiments, including those for multi-person interference, hand-held paper occlusion, and industrial goggles, achieved accuracy rates of 92%, 91%, and 92%, respectively. The system’s latency for real-time processing is 60.5 ms on edge devices like the NVIDIA Jetson, making it suitable for applications requiring low-latency emotion recognition. Additionally, radar parameter optimization, such as adjusting the ADC sample rate and chirp size, has been shown to improve classification accuracy. These findings highlight the system’s robustness and adaptability to varying environmental conditions and its potential use in privacy-sensitive applications, including healthcare, security, and interactive media. Future work will explore radar-only systems, further reducing dependence on visual data, and investigate more advanced deep learning techniques to improve performance, scalability, and real-time deployment.