Bidirectional coupling of physiology and emotions reveals unique speaker-listener dynamics

Abstract Conversational interactions extend beyond verbal exchange, involving dynamic synchronisation across physiological, emotional, and acoustic domains. This study aimed to characterise multimodal speaker-listener coupling during structured debates to examine how bidirectional dynamics relate to autonomic regulation, signal complexity, and emotional states. We analysed data from the K-EmoCon database, which included 32 participants in 10-minute debates. Heart rate variability (HRV) features, speech features, and multi-perspective emotion annotations were synchronised and analysed in segmented speak-listen phases. Cross-correlation and bidirectional coupling quantified coupling strength and directionality, and their relationships with emotional states were assessed. Negative lag segments showed significantly higher HRV features, including AVNN (0.655 0.631–0.749, p = 0.003), HF power (0.620 0.338–0.781, p 0.001), and SD1 (1.30x10−3 1.19–1.41x10−3, p = 0.003). Positive lag segments were associated with higher sample entropy (0.058 0.046–0.065, p 0.001) and diffusion entropy (μr: 1.209 1.201–1.221, p 0.001). Lower emotion states (1–2) tended to exhibit negative-lag dynamics, whereas moderate to high states (3 and 5) showed a modest positive-lag predominance. Multimodal coupling reveals distinct physiological and emotional signatures linked to leadership and responsiveness in conversation, providing insights for therapeutic, educational, and collaborative applications.