AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Assessing foundation models' abilities for human-level tasks is crucial for Artificial General Intelligence (AGI) development.Traditional benchmarks, which rely on artificial datasets, may not accurately represent these capabilities.In this paper, we introduce AGIEval, a novel bilingual benchmark designed to assess foundation models in the context of humancentric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests.We evaluate several state-of-the-art foundation models on our benchmark.Impressively, we show that GPT-4 exceeds the average human performance in SAT, LSAT, and math contests, with 95% accuracy on SAT Math and 92.5% on the Chinese college entrance English exam.This demonstrates the exceptional performance of contemporary foundation models.In contrast, we also find that GPT-4 is less proficient in tasks requiring complex reasoning or specific domain knowledge.Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal their strengths and limitations, providing valuable insights into future directions for enhancing general capabilities.By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a meaningful and robust evaluation of foundation models' performance in real-world scenarios 1 .Recently, large foundation models, such as the large language models (LLMs) ChatGPT(OpenAI, 2022) and GPT-4 (OpenAI, 2023), exhibited remarkable versatility and adaptability, with plethora of applications spanning various domains as a decision-making assistant, from processing daily events to assisting in specialized fields such as law * indicates equal contribution.Yaobo Liang and Nan Duan are the corresponding authors.1 The data, code, and all model outputs are released in