Comparative Evaluation of Machine Translation Models Using Human-Translated Social Media Posts as References: Human-Translated Datasets

Machine translation (MT) of social media text presents unique challenges due to its informal nature, linguistic variations, and rapid evolution of language trends.In this paper, we propose a human-translated English dataset to Arabic, Italian, and Spanish, and a humantranslated Arabic dataset to Modern Standard Arabic (MSA) and English.We also perform a comprehensive analysis of three publicly accessible MT models using human translations as a reference.We investigate the impact of social media informality on translation quality by translating the MSA version of the text and comparing BLEU and METEOR scores with the direct translation of the original social media posts.Our findings reveal that Mari-anMT provides the closest translations to human for Italian and Spanish among the three models, with METEOR scores of 0.583 and 0.640, respectively, while Google Translate provides the closest translations for Arabic, with a METEOR score of 0.354.By comparing the translation of the original social media posts with the MSA version, we confirm that the informality of social media text significantly impacts translation quality, with an increase of 12 percentage points in METEOR scores over the original posts.Additionally, we investigate inter-model alignment and the degree to which the output of these MT models align.