Randomly Wired Network Based on RoBERTa and Dialog History Attention for Response Selection

Recently, chatbot research for conversation has been actively conducted in the field of natural language processing. Among the dialogue systems, a retrieval-based system has emerged as an important one. Because, it showed better performance than the response generation models and it actually seemed applicable. The response selection task is used to develop a retrieval-based system. Thus, we did research to improve the performance of retrieval-based dialogue systems for response selection. We used Google's bidirectional encoder representations from transformers (BERT) natural-language processor with the robustly optimized BERT pre-training model (RoBERTa) as the base model to improve performance. The Recall@N of our model was improved by 1.7–4.2% using two methods: First, instead of using a simple feed-forward network at the end of the model, we employed a randomly wired neural network that contained multiple wiring paths, thus achieving better performance than that of feed-forward networks. The experimental results of this study demonstrated that feed-forward networks can be replaced by randomly wired neural networks. Second, we calculated the correlation between dialog history (i.e., context) and the last utterance of the context, reflecting this correlation as an attention network that yielded an answer prediction. Using first and second method achieved higher performance than using only the first method in dialog response selection tasks.