An Exploratory Multi-Model Comparative Design and Validation of AI Preliminary Evaluation Models for Holistic Admissions Document Review

Document review in holistic admissions in Korean universities is based on evaluators’ qualitative judgment; therefore, differences in evaluators’ experience and institutional evaluation environments may directly affect fairness among applicants. To mitigate such variation, a support tool is needed to apply each institution’s evaluation criteria consistently. This study examined, through an exploratory multi-model comparison, whether an AI-based evaluation support tool can implement institution-specific evaluation criteria, secure construct validity, and contribute to reducing inter-evaluator variance. Based on the evaluation items from a joint study by five Korean universities (Lim et al., 2022), four AI preliminary evaluation models were constructed: Model A (LLM self-generated criteria), Model B (fact extraction layer), Model C (no-rubric control), and Model D (expert criteria encoding). Thirty mock student records were independently evaluated by the four models and by two expert evaluators from different universities. The results showed that generic models achieved good rank agreement (Kendall W=.859) but low categorical agreement (Fleiss κ=.165). Agreement between the two cross-institutional experts was κ=.397, and all disagreements were confined to adjacent grades, indicating differences in evaluation criteria across institutions. Model D reproduced Expert A’s judgment at κ=.648, while its agreement with Expert B was κ=.261. However, in the 18 cases where the two experts agreed, Model D showed the highest hit rate among the four models at 77.8%. The findings suggest that encoding evaluation criteria is effective when applied to institution-specific criteria, and that the fact extraction layer effectively detects factuality deficits in records with excessive AI-generated wording. AI-based evaluation support tools may contribute to fairness by protecting applicants from evaluator- and year-based variance and by preventing students’ genuine effort and growth from being obscured.