Evaluating translation performance of Large Language Models in maritime legal translation

The generic ‘intertextuality’ in maritime laws across languages and the bidirectional translation characteristics in maritime legal translation practice provide a unique research perspective for evaluating translation performance of Large Language Models (LLMs). Terminology in maritime legislation presents features that are technically universal but legally-specific to languages; legal concepts and the generic structure of legislation have the ‘lingua franca’ nature owing to the international shipping industry. This study constructed a Chinese-English bilingual maritime legal dataset comprising 2,000 sentence pairs. Using three automatic translation evaluation metrics (BERTScore, BLEURT, and COMET), this study evaluated translation performance of eight LLMs. Results show a consistent performance decline from English-to-Chinese (E2C) to Chinese-to-English (C2E) across all models, with divergent directional preferences among individual models. Claude-3.5 Sonnet and ERNIE Bot (4.0) share the highest performance in the E2C direction, while TowerInstruct-7B shows the weakest performance in both directions. Qualitative case analysis reveals that LLMs maintain intertextual correspondence at the surface level of standardized terminology but struggle with legal concept reconstruction where cross-jurisdictional institutional knowledge is required. These findings provide empirical evidence for determining LLM application boundaries in legal translation and inform domain-specific optimization strategies.