Emotional chain reasoning-driven multimodal fusion for empathic humanoid robot interaction

As humanoid robots enter companion service, educational assistance, rehabilitation support, and public interaction, the central challenge is not only to recognize emotion but to convert multimodal human cues into coordinated language and action. The proposed Emotional Chain Reasoning-Driven Multimodal Fusion Network (ECDT-MFN) is not a separate alternative to vision-language-action (VLA) architectures. At the task and pipeline level, it is an emotion-aware VLA extension: visual observations and language context are mapped to social actions, while voice prosody, facial expression, body movement, and haptic feedback provide an auxiliary biometric-affective observation stream. The contribution lies in adding dynamic affective conditioning, explicit-to-implicit emotional chain reasoning, and empathy-conditioned action planning to this VLA-style perception-language-action loop. The model jointly produces empathic verbal responses and executable social behaviors such as gaze, head posture, interaction distance, waiting, assistance, and conflict avoidance. Evaluation is therefore organized at two levels. Participant-centered, end-to-end HRI outcomes—including perceived empathy, behavioral appropriateness, user satisfaction, interaction continuity, and long-term consistency—serve as the primary evidence of effectiveness; automatic emotion-recognition, semantic-similarity, module-ablation, and noise-retention measures are retained only as diagnostic evidence. Results show that the complete emotion-aware VLA pipeline improves the tested whole-interaction outcomes while maintaining the measured software-inference latency of the reference implementation. These findings support the value of affective conditioning for socially responsive VLA control, rather than claiming a new non-VLA control paradigm.