This study aims to explore the potential application of automated scoring in educational settings by empirically analyzing the accuracy and rater effects associated with an essay-type assessment conducted by a large language model (LLM), specifically ChatGPT. To this end, the agreement between human raters and ChatGPT was examined. Furthermore, the Many-Facet Rasch Model was applied to compare rater characteristics. The key findings are as follows. First, ChatGPT tended to concentrate its scores on specific values in certain criteria. Second, the level of agreement between human and automated ratings was relatively high for criteria with structured scoring rubrics such as “task completion” and “validity of evidence,” but generally low in other criteria. Third, while some human raters demonstrated misfit or overfit behaviors, ChatGPT displayed a slightly lenient scoring pattern but remained within an acceptable range of model fit. These findings indicate that ChatGPT-based automated scoring may contribute to improving scoring consistency. However, further technical refinement is needed for criteria that require nuanced judgment and contextual interpretation.
Oh et al. (Tue,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: