A Method for Determining Utterance-Response Relationships between Distant Utterances for Improving the Quality of Dialogue Corpora from Novels

With the advancement of various dialogue systems based on large language models, the demand for language resources has significantly increased.However, there are not enough Japanese dialogue corpora for developing and evaluating chat-oriented dialogue systems.In addition, it is difficult to construct a corpus easily, because most of the conventional corpus construction methods are based on manual processes, which are both time-consuming and costly.Therefore, we focus on a method for constructing a large Japanese chat-oriented daily dialogue corpus using characters' lines from novels.A large number of novels are produced every day, containing a wide variety of characters' dialogues.When a corpus is constructed by extracting only consecutive lines from novels, the number of utterances per dialogue becomes quite small.Since novels also contain distant lines, it is necessary to consider the utterance-response relationships between distant lines when constructing a dialogue corpus.In this paper, we propose a method for determining the utterance-response relationship between distant lines using a fine-tuned language-based model.From the evaluation results, we confirmed the validity of the proposed method and the feasibility of constructing a dialogue corpus from novels.