A First Step Towards Considering Nonverbal Communicative Cues in Dialog Systems

Currently, spoken dialogue systems mostly consider only voice interactions and do not consider nonverbal cues. This work aims to explore the domain of interaction with conversational assistants, taking into account not only verbal cues, but also the importance of integrating nonverbal aspects to make interactions between humans and agents more natural, richer, and more efficient. This work presents an initial proof-of-concept for the modules needed for dialogue systems to take into account verbal and nonverbal inputs, as well as their articulation, demonstrating their viability. To this end, we use a fusion module that can directly or indirectly affect the communicative acts, depending on the verbal and nonverbal communication inputs. Preliminary results demonstrate the implemented system behaviour of how verbal content is affected by nonverbal cues.