Do you understand Italian? Evaluating LVLMs on Italian visual question-answering

Abstract Large Vision-Language Models are an extension of Large Language Models to process signals from the vision-language domain. Their performance is remarkable since they exploit the cutting-edge capabilities of pre-trained Large Language Models. However, most works using these models focus on English language data for training and evaluation. In light of this, several works have been released to extend current training mixtures to consider non-English data. As a matter of fact, several works also considered cultural aspects to decrease bias related to American and Chinese culture. However, evaluation in non-English languages is still limited w.r.t. English. Specifically, there are no datasets that focus on text-centric non-English images, since most available multimodal benchmarks focus on natural images. This limitation makes it unclear whether Large Vision-Language Models can be used for non-English text-centric tasks. In this work, we aim to understand the capabilities of these models for the Italian language and introduce a benchmark dataset focusing on text-centric Italian images. Therefore, we introduce our benchmark dataset ETCII ( Evaluation of Text-Centric Italian Images ), to evaluate Large Vision-Language Models on three diverse task categories for text-centric images.