Evaluating COMET-Kiwi for Korean–Spanish bidirectional translation: a translation studies and linguistic analysis

This study evaluates COMET-Kiwi, a reference-free machine translation evaluation metric, for bidirectional Korean–Spanish translation assessment. Experiments use a linguistically controlled dataset with 1846 pairs across lexicology, morphology, syntax, pragmatics, and information structure. Each item is scored twice (Korean→Spanish; Spanish→Korean) and interpreted through a functionalist lens, where adequacy depends on communicative purpose, register, politeness, and cultural fit. COMET-Kiwi assigns high scores to many semantically well preserved translations involving clause structure, established terminology, tense/mood, deixis, and word formation, suggesting partial sensitivity to typologically motivated reformulations. However, systematic gaps emerge for pragmatically loaded or structurally divergent phenomena (slang, polarity shifts, experiencer constructions, honorifics), and score–label alignment is less stable when Korean is the target. The discussion also flags English-mediated translation conventions as a potential source of directional asymmetry and metric bias. Overall, the results delineate where reference-free scoring tracks functional adequacy and where it diverges for a low-resource, typologically distant pair.