Best of both worlds: combining LLMs and traditional ML for automated scoring of an open-response situational judgment test

Introduction This study presents a novel dual-architecture automated scoring system combining large language models (LLMs) with traditional machine learning (ML) regression techniques. Using the Casper open-response situational judgment test as a test case, we demonstrate the effectiveness of this system for scoring tests of complex constructs in an accurate and interpretable way. Methods The first component of the dual architecture system relies on LLMs-as-Judges for extracting construct-relevant features from text. The second component of the scoring system uses traditional regression techniques to aggregate features extracted by the first component. We evaluated the effectiveness of each component using human ground truth data. Results Firstly, we found that the LLM judges we constructed by applying a prompt optimization framework performed at levels comparable or superior to trained human evaluators on the same task (average QWK across features for both human-AI and human-human agreement). Secondly, we were able to implement feature engineering and weighting techniques characteristic of regression modeling to achieve scoring performance that is again comparable or superior to trained human evaluators (mean absolute error (MAE), Exact Agreement, and Adjacent Agreement). Discussion By separating the feature extraction and scoring components we were able to achieve promising results for a novel interpretable and reliable scoring system as applied to an open response situational judgment test. The approach demonstrated here also serves to establish a standard for future development and evaluation of similar dual-model automated scoring systems for other assessments.