Quality estimation of AI-driven translations of ancient Buddhist scriptures: a multi-model automated translation and evaluation framework

Abstract We evaluate six flagship Large Language Models (LLMs) (spanning prior- and current-generation systems across the ChatGPT, Claude, and Gemini families) on Pali-to-English translation for selected suttas from the Majjhima Nikaya. We introduce an automated pipeline that segments scripture into context-linked JSON, enforces format-faithful outputs via prompt engineering, and closes the loop with artificial intelligence (AI)-based quality assessment using the Generative Pretrained Transformer (GPT) Estimation Metric Based Assessment (GEMBA) in both no-reference (NR) and human-reference modes, alongside METEOR, TER, and BERTScore. Results show that modern LLMs can produce high-quality translations of ancient Pali: GPT-5 is the most consistent on NR-GEMBA (fewest sub-80 rated lines), while Claude 4 Sonnet aligns most closely with the human reference on traditional metrics; a compact head-to-head matrix further reveals a line-wise edge for Claude 3.5 Sonnet in pair-wise wins. Inter-evaluator agreement is moderate-to-high. Overall, the workflow can broaden access to Buddhist scripture while leaving final interpretive authority with human scholars, and it provides a scalable template for evaluating AI translations of ancient texts.