This study aims to empirically examine how the Japanese colloquial translation performance of ChatGPT 4.0, a conversational AI model, has evolved over time. It also seeks to explore the potential and limitations of machine translation (MT) for educational and practical applications. The analysis was conducted using a dataset of 1,290 Korean dialogue sentences from three Netflix original dramas—Kingdom, Queenmaker, and The Silent Sea. These sentences were translated into Japanese using ChatGPT under identical conditions in 2023 and 2025, and the results were compared with human translations (Netflix subtitles). The findings show that while 374 translation errors (29.0%) were identified in the 2023 output, the number dropped significantly to 40 (3.1%) in the 2025 output, indicating an 89.3% reduction. Notably, typical MT issues such as incomplete sentences, word or clause order errors, and typographical errors were absent in both periods, and grammatical and honorific-related errors nearly disappeared by 2025. These results suggest a remarkable improvement in ChatGPT’s grammatical stability and structural completeness. However, certain limitations remain. In particular, Kingdom, a historical drama, revealed persistent mistranslations of honorifics and titles, over-generation of interjections and figurative expressions, and insertions of words not present in the source text. These issues indicate challenges in pragmatic accuracy and context-sensitive interpretation. This study empirically demonstrates the rapid advancement of LLM-based MT systems and offers practical implications for their use in translation education and content production. Moreover, by providing concrete examples of colloquial translation across diverse genres, the study contributes to the development of pedagogical frameworks and training materials for future translation practice.
Sim et al. (Thu,) studied this question.