Development of a Multimodal Mental Health Monitoring System Using Emotion Recognition and HRV with AI Chatbot

Despite the increasing focus on the mental health of older adults and active senior populations, assessment tools still lag behind those for physical health monitoring. To bridge this gap, this study introduces an AI chatbot-based multimodal stress monitoring system that utilizes emotion recognition and heart rate variability (HRV). The system analyzes chatbot conversations, video, audio, and heart rate signals to assess facial expressions, speech emotions, and HRV, allowing for stress evaluation and user stratification into risk groups. Negative emotions are quantified and combined with HRV data to generate a stress score. Facial and speech emotion models were trained on the RAVDESS, CREMA, and TESS datasets, yielding 21,000 augmented samples through a BiLSTM network. Additionally, a deep learning-based HRV model utilized data from smartwatches to predict stress levels. By integrating facial, vocal, and HRV features through weighted fusion, the system produces a comprehensive stress index that categorizes users Healthy, Caution, Risk. This approach facilitates continuous monitoring at home, supporting early detection for preventive care and informed clinical decision-making.