Creating a Multimodal Dataset for the SciLibRu Semantic Library Using a Language Model

This paper is devoted to the creation of an annotated dataset of multimodal mathematical expressions, in which formal expressions in the Lean4 language are supplemented with representations in natural language (Russian and English), LaTex symbolic notation, and formula visualizations, and the development of a neural network model based on this dataset for constructing vector representations of mathematical objects. The dataset is constructed by integrating complementary modalities. In the paper, a multimodal learning architecture with adaptive loss function balancing is presented. In the proposed approach, it is possible to solve the problem of conflicting gradients that arises when training on modalities with different noise levels using a two-level optimization scheme: for each modality, loss functions with local components are determined. The model was trained on the created dataset. The resulting model of vector representation of mathematical expressions is intended to implement the process of automatic analysis and reading of scientific publications in the SciLibRu semantic library of subject areas. In this paper, automatic reading is understood as the process of automatic extraction of texts, formulas, and formal expressions for the purpose of subsequent semantic processing and analysis.