A study on the task list of Korean benchmarks: based on the case studies for the different organizations of the task lists in domestic and foreign NLP benchmarks

Through examining the successful building experience of the well-known Korean and foreign language NLP benchmarks and closely comparing their task list compositions, the study intends to provide a novel organizing scheme for the task list in the future-oriented Korean benchmarks. When creating benchmarks for a low-resource language like Korean, as opposed to English, a worldwide language, there may be a variety of obstacles, from procuring language resources to identifying important demands in a real-world setting. To figure out a more efficient and economical construction plan for the next-generation Korean benchmark, it is of utmost necessity to pay special attention to the organization of the task list, which serves as the basis of the benchmark, Consequently, in this study, we propose a new organization plan for the task list of the Korean benchmarks based on an insightful investigation of the various task list organization methods in the previous NLP benchmarks. Additionally, we suggest a step-by-step development roadmap for Korean benchmarks through the prioritization of the various tasks, incorporating the comprehensive consideration of the commonality of the evaluation system for the AI’s language ability and the specificity of the development of the Korean AI model.