Human Quality Evaluation of Machine-Translated Poetry

The quality of literary translation was from the beginning of literacy an important factor in publishing and, as a consequence, in research and education. The quality of literary text translation is of utmost significance for researchers and students, especially in higher education. Only complete and high-standard translations are believed to be necessary for the use in the evaluation and study of style and concepts of a given author or a literary genre. This quality verification applies even more to machine translation in general, due to the fact that such translations are deemed subpar and unsuitable for further dissemination and examination. The need for human quality evaluation of machine-translated text is therefore highly emphasised, since human translations are considered to be the “gold standard” and reference translations in the machine translation process. The aim of this paper is to explore, on the example of a data set consisting of poems written by a relevant contemporary Croatian poet, the effectiveness of applying machine translation on the Croatian-German language pair in the domain of poetry, with regard to human judgment of machine translation quality. Human evaluation in this paper is conducted by taking into account two machine translation quality criteria - adequacy and fluency, after which an inter-rater agreement analysis is performed.