Saudi Dialects to MSA Machine Translation: A Systematic Evaluation of LLMs

Converting Dialectal Arabic (DA) texts into Modern Standard Arabic (MSA) is considered an important step in downstream applications. This is because the huge diversity of Arabic dialects leads to limited available resources of such data. Besides, dealing directly with DA involves multiple complexities, as there are no specialized preprocessing tools designed for this form of the language. The recent advances in Large Language Models (LLMs) sound like a promising technology to ease the process of translating DA texts to MSA, particularly for such a low-resource task. In this paper, we evaluate the efficacy of LLMs on the task of translating fine-grained Saudi Dialectal Arabic to MSA. This was done using two LLM-based models, namely ALLaM (a multidialectal model) and GPT-3.5 (a multilingual model). This process involved designing a specific prompt framework for the LLM models. The experiments were conducted on two different Saudi dialect datasets: SauDial (a newly developed dataset) and MADAR (a benchmark dataset). The results were evaluated using four metrics: BLEU, TER, METEOR, and COMET. Our findings indicate that ALLaM significantly outperforms GPT-3.5 on both datasets across different Saudi dialects, with an average BLEU score of 53.17% and 35.02% on SauDial and MADAR, respectively.