Evaluation reveals finetuned language models predict English-Chinese interpreting scores with direction-dependent accuracy, highlighting the need for speech-sensitive scoring approaches.
Key Points
To evaluate the feasibility and accuracy of finetuning multilingual pre-trained language models for the automatic assessment of English-Chinese interpreting.
Finetuned two pre-trained language models (GTE-multilingual and mmBERT) on over 1,700 human-scored English-Chinese interpreting renditions rated via a four-band, eight-point analytic rubric.
Trained models via five-fold cross-validation for band classification and score regression, evaluating held-out test performance using Accuracy, Quadratic Weighted Kappa, Mean Absolute Error, and Pearson's correlation.
Conducted ablation studies testing the effects of reference interpretations, analytic rubrics, and interpreting direction on automated scoring behavior.
mmBERT consistently surpassed GTE-multilingual in score prediction accuracy, yet both models struggled to evaluate speech fluency from textual transcripts alone.
Analytical rubrics and reference interpretations provided conditional benefits as supervisory signals during model finetuning.
Systematic performance discrepancies emerged between interpreting directions, with directional effects most pronounced in fluency scoring.