Designing engagement-aware agents for multiparty conversations

Recognizing users' engagement state and intentions is a pressing task for computational agents to facilitate fluid conversations in situated interactions. We investigate how to quantitatively evaluate high-level user engagement and intentions based on low-level visual cues, and how to design engagement-aware behaviors for the conversational agents to behave in a sociable manner. Drawing on machine learning techniques, we propose two computational models to quantify users' attention saliency and engagement intentions. Their performances are validated by a close match between the predicted values and the ground truth annotation data. Next, we design a novel engagement-aware behavior model for the agent to adjust its direction of attention and manage the conversational floor based on the estimated users' engagement. In a user study, we evaluated the agent's behaviors in a multiparty dialog scenario. The results show that the agent's engagement-aware behaviors significantly improved the effectiveness of communication and positively affected users' experience.