Analyzing the Multilingual Ability of Vision-Language Models to Generate Explanations for Artworks

Key Points

• The aim is to evaluate the multilingual ability of vision-language models (VLMs) in generating explanations for artworks and assess the impact of English instruction on other languages.
• Proposed a multilingual image understanding dataset without relying on machine translation.
• Conducted experiments with various models to evaluate the explanation generation abilities of VLMs.
• Investigated whether English training improves performance in other languages.
• VLMs demonstrated decreased performance when tasked with generating explanations in languages other than English.
• Identified challenges in effectively leveraging knowledge from English data for other languages.
• Highlighting the necessity for multilingual training in both image encoder learning and integrated learning processes.