Phenomenon-wise Evaluation Dataset Towards Analyzing Robustness of Machine Translation Models

Though its importance has been recognized, it is still not clear as to what creates the large performance gap between the translation of clean input and that of UGC. To answer the question, we present a new dataset, PheMT , for evaluating robustness of MT systems against speci(cid:12)c linguistic phenomena in Japanese-English translation. We provide more (cid:12)ne-grained error analysis about the behavior of the models with the accuracy and relative drop in translation quality on the contrastive dataset specif-ically designed for each phenomenon. Our experiments with the dataset revealed that not only our in-house models but even widely used o(cid:11)-the-shelf systems are greatly disturbed by the presence of certain phenomena.