GPT-based Creativity Assessment: Focusing on Comparison with Human Experts

The development of generative artificial intelligence (AI) is significantly reshaping creativity research, particularly regarding its potential for effectively assessing creative products. Evaluations of creative outputs have traditionally relied on expert-based Consensual Assessment Technique (CAT); however, CAT demands substantial time and resources to achieve high reliability. Thus, it has become necessary to investigate AI-driven evaluation methods capable of complementing or substituting expert assessments. This study compared creativity evaluations conducted using GPT-4.1 and GPT-4o—multi-modal large language models (LLMs)—on creative title-generation tasks by 99 middle school students with CAT evaluations provided by six creativity experts. Specifically, evaluations were repeatedly conducted under varying conditions of model, prompt type, and temperature. Agreement among GPT evaluations was measured by percentage agreement and intraclass correlation (ICC), whereas consistency with CAT evaluations was examined using Pearson’s r, Spearman’s rho, and root mean square error (RMSE). The analyses revealed optimal GPT model, prompt, and temperature settings consistent with CAT, providing practical guidelines for GPT-based creativity assessments. This study contributes foundational insights for designing and implementing AI-based evaluations that align with CAT principles.