Evaluating Performance and Accuracy of Large Language Models in Translating Code-Mixed Hindi to English: A Comparative Study

This is the authors' abstract. We don't add key points for this paper.

Translating code-mixed Indian languages, where more than two languages are blended in a single sentence, poses significant difficulties and challenges in translation systems. Given the wide variety of languages in India, evaluating the performance of large language models on these tasks has become increasingly important. This study conducts a comparative analysis of advanced LLMs, including BLOOMZ, GEMMA, GPT, and Gemini, focusing on their ability to translate Hindi-English text which is code-mixed. The models are evaluated and assessed based on translation accuracy and contextual consistency. Among the models evaluated, Gemini achieved a BLEU score of 20.82, demonstrating superior performance compared to the others. The analysis reveals the strengths and weaknesses of each model, highlighting the ongoing challenges in translating code-mixed content. These findings provide useful insights for model selection in real-world applications and highlight areas for future research to improve translation quality in code-mixed scenarios. As the prevalence of code-mixing continues to rise, improving the capabilities of LLMs in handling such complexities is essential for better communication and understanding in multilingual societies.