Assessing the Quality of Large Language Models in Machine Translation Tasks

Key Points

• The aim is to compare the quality of translations produced by various large language models against established benchmarks.
• Comparative evaluation of translations from six large language models: DeepSeek, Grok, Mistral, Qwen, GigaChat, and Yandex.
• Assessment of translation quality using both quantitative metrics (BLEU, METEOR, chrF) and qualitative expert analysis.
• Comparisons made against Google Translate and based on criteria of adequacy, equivalence, and harmony.
• Modern large language models outperform classical machine translation, addressing traditional challenges effectively.
• LLMs demonstrate significant capability in translating expressive linguistic elements such as phraseologisms and puns.
• Expert evaluation highlights improved adequacy and equivalence in translations by LLMs compared to reference translations.