Cross-Task Evaluation and Empirical Analysis of Japanese Visual Language Models

Key Points

• The aim is to develop a comprehensive evaluation framework for Japanese visual language models that enables consistent assessment across tasks.
• Constructed llm-jp-eval-mm framework integrating 10 Japanese and 9 English datasets.
• Defined capability axes and designed evaluation protocols for systematic assessment.
• Evaluated 32 existing visual language models under uniform conditions.
• Identified relative weaknesses in specific capability areas of domestically developed visual language models.
• Reported structure of performance correlation between Japanese and English models.