Evaluating the Multilingual Differences of ChatGPT and Google Gemini on the MMLU Dataset Translated into Thai

This study examines the multilingual capabilities of two leading Large Language Models (LLMs), ChatGPT and Google Gemini, using the English and Thai versions of the Massive Multitask Language Understanding (MMLU) dataset. Aimed at assessing model performance across different linguistic and cultural contexts, our research sheds light on the inherent challenges of cross-lingual language processing. Through a series of comparative analyses, we uncover significant performance disparities between the models across the two languages, with both models demonstrating superior efficacy in English over Thai. These findings highlight the current limitations of LLMs in achieving truly universal language understanding, emphasizing the impact of linguistic diversity on model accuracy and responsiveness. The study contributes to the broader discourse on the need for more inclusive and representative model training approaches, advocating for the integration of diverse linguistic inputs and cultural nuances. Our research shows the importance of advancing multilingual capabilities in LLMs to ensure equitable access to AI technologies across global communities, proposing pathways for future work aimed at enhancing the universality and applicability of language models.