Developing the Ensemble of Models to Find the Most Relevant Text to the Question

The wide application of intelligent assistants and chatbots makes information search a relevant task. Large language models are increasingly used to generate answers to user questions. The task of searching the Russian-language texts that contain information necessary to generate an answer to a user's question is considered in the article. Authors have developed text datasets in Russian, which differ both in the number of characters included in each text fragment and in the number of fragments themselves. The largest text dataset contains 1,000,000 texts with 5,000 characters each. Also, authors developed 6 classes of questions, which differ in the way the answers to them are presented in the text. The ensemble of models that uses the vector representation of words obtained using neural networks with transformer architecture and the values of the ranking function is developed and tested on different text datasets. The ensemble has proven its effectiveness in finding the most relevant text. Various filterings of text fragments are proposed and tested. The developed double filtering made it possible to remove from consideration chunks that do not contain an answer to the question. The obtained high metrics of accuracy and speed of applying the developed software that implements the ensemble of models and filtering determine the possibility of use in various intelligent assistants.