Modeling degree of dialogue breakdown in open-domain conversations

With the advancement of large language models (LLMs), the response accuracy of dialogue systems has greatly improved. However, dialogue breakdowns still occur, causing user discomfort and potentially leading to disengagement from the interaction. Therefore, it is essential to either prevent breakdowns in advance or appropriately repair them when they arise. Previous studies have primarily focused on binary classification—determining whether a breakdown has occurred or not. As dialogue breakdowns can occur to different degrees, we hypothesize that identifying the extent to which a system utterance is broken allows for the design of suitable repair strategies corresponding to the degree of breakdown. To address this, we introduce the concept of the degree of dialogue breakdown, which reflects the extent to which users perceive a system utterance as broken. We then develop a model to estimate this degree. We first asked crowd workers to rate the degree of breakdown in a set of author-prepared example utterances. Based on these annotations, we built a predictive model to estimate the perceived degree of breakdowns. Furthermore, since users may differ in how strongly they perceive breakdowns, we also analyze the relationship between users’ personality traits and their sensitivity to the degree of dialogue breakdown.