What question did this study set out to answer?

The research aims to develop automated evaluation methods for large language models (LLMs) that mimic human judgment.

April 1, 2026Open Access

KNUIR at the NTCIR-18 AEOLLM: Automatic Evaluation of LLMs

Key Points

The research aims to develop automated evaluation methods for large language models (LLMs) that mimic human judgment.
Proposed two evaluation approaches: LLM-based scoring and feature-based machine learning.
Employed prompt engineering with GPT models in the LLM-based method.
Utilized transformer-based metrics such as BERTScore and semantic similarity in the feature-based evaluation.
GPT-4o Mini with updated prompts achieved the highest performance in evaluations.
The feature-based approach outperformed GPT-3.5 Turbo but was close to GPT-4o Mini.
LLM-based methods were scalable but lacked explainability, while feature-based offered better interpretability but needed extensive tuning.

Abstract

In this study, we aim to propose automated evaluation methods of LLMs that approximate human judgment by exploring and comparing two distinct approaches: (1) LLM-based scoring, which utilizes GPT models with prompt engineering, and (2) feature-based machine learning, using transformer-based metrics such as BERTScore, semantic similarity, and keyword coverage. As part of this research, we participated in the NTCIR-18 Automatic Evaluation of LLMs (AEOLLM) task. We submitted the results of the test data set and the reserved data set to NTCIR-18 and analyzed the results obtained. The results show that GPT-4o Mini (with the updated prompt) achieved the highest performance, while the feature-based approach performed competitively, surpassing GPT-3.5 Turbo and showing a small gap with GPT-4o Mini. LLM-based methods offered scalability but lacked explainability, whereas feature-based approaches provided better interpretability but required extensive tuning, highlighting the trade-offs between the two strategies. Throughout the analysis, We expect that the findings of our work will provide insights into the understanding of human judgment and automated evaluation of LLMs.

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper