Using AI in Translation Quality Assessment: A Case Study of ChatGPT and Legal Translation Texts

The use of artificial intelligence (AI) in Translation Quality Assessment (TQA) has emerged as an exciting new line of research hoping to explore the potential of this revolutionary technology within the field of translation studies in general and its effect on translator training ecosystem. The aim of this study is to explore how AI’s evaluation of students’ legal translations aligns with instructors’ evaluations and to look at the potential benefits and challenges of using AI in evaluating legal translations tasks. Ten anonymous copies of instructor-graded English-to-Arabic mid-term exam translations were collected from an undergraduate legal translation course at a Saudi university and evaluated using ChatGPT-4o. The system was prompted to detect the translation errors and score the exam using the same rubric that was used by the instructors. A manual segment-by-segment comparison of ChatGPT-4o and human evaluations was conducted, categorizing errors by type and assessing alignment by comparing the scores statistically to determine if there were significant differences. The results indicated a high level of agreement between ChatGPT-4o and the instructors’ evaluation. In addition, paired sample t-test comparisons of instructor and ChatGPT-4o scores indicated no statistically significant differences (p > 0.05). Feedback provided by ChatGPT-4o was clear and detailed, offering error explanations and suggested corrections. Although such results encourage effective integration of AI tools in TQA in translator training settings, strategic implementation that balances automation with human insight is essential. With proper design, training, and oversight, AI can play a meaningful role in supporting modern translation pedagogy.