Abstract— This paper proposes a real-time emotion recognition system that integrates facial expression analysis and heart-rate monitoring within a Virtual Reality (VR) environment for mental health assessment. Facial cues are captured via webcam or HMD camera, while heart rate is measured using camera-based photoplethysmography or wearable sensors. These inputs are processed in a closed-loop framework that adapts the VR environment—altering visual or auditory stimuli based on the user’s emotional state. Simultaneously, emotional and physiological data are visualized on a clinician-facing dashboard. The system was developed using JavaScript (TensorFlow.js, face-api.js), Python (Flask, WebSockets), and VR environments styled in Unity. Test sessions showed sub-second feedback latency and approximately 88% accuracy in recognizing core emotions under optimal lighting. Notably, heart rate spikes correlated with stress-labeled emotions like fear and surprise, indicating accurate arousal detection. The results highlight the potential of emotion-aware VR systems for responsive, data-driven therapeutic applications in mental health contexts. Index Terms—Virtual Reality, Affective Computing, Emotion Recognition, Heart Rate, Real-Time Systems, Mental Health Assessment.
Kumar et al. (Mon,) studied this question.