Analyzing Image Patterns and Generating Text: Advances in Multilingual Vision-Language Transformers

Creating an artificial neural network capable of generating textual and audio descriptions of graphical images in multiple languages holds significant value from technological, social, economic, and humanitarian perspectives. This paper discusses neural networks and machine learning models focused on processing graphical images (pictures) and producing textual descriptions that can be used to solve multilingual tasks. These include convolutional neural networks, recurrent neural networks, its improved version called long short-term memory, as well as transformer-based models, bidirectional encoder representations from transformers, its multilingual version, and bootstrapped language-image pretraining. The paper presents a comparative analysis of these models, allowing the identification of their advantages and weaknesses when performing multilingual and visual tasks. Special attention is given to the bootstrapped language-image pretraining model, which is designed for simultaneous processing of text and images within a unified framework. Its main limitation is its focus on the English language, which poses a challenge for non-English language tasks. In the present paper is explored several approaches to overcoming this limitation: (1) integrating a translator at the output stage; (2) replacing the encoder/decoder with implementations that support the target language; (3) retraining the existing model using multilingual data. Each method comes with specific technical challenges, the analysis of which and corresponding recommendations are presented in the conclusion of the paper. To address this issue, we developed the model with name “Marta.” This model is based on the language model, which is a variant of the text-to-text transfer transformer model. To extract image features, the model uses base-sized vision transformer model that processes images as 16 × 16 patches (ViT-B-16), which is already trained for image feature recognition. Over 200 000 “image/description” pairs for training are already prepared in Georgian, so that the language model can learn how to handle the tensors output by the vision transformer model, that is, to correctly transform them into verbal interpretations and learn the logical process.