Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems

In large‐scale assessments, constructed response items are often scored using hybrid scoring systems, which combine human and automated scores. In this study, we augment automated scoring with confidence modeling to strategically route difficult‐to‐score responses for human review. We utilize hybrid performance curves to visualize the impact of routing on performance. Additionally, we propose several hybrid scoring policies for selecting optimal routing thresholds given practical constraints. Our findings reveal that hybrid scoring systems can achieve an overall performance that exceeds that of human‐ and automated‐only systems. Moreover, the superior performance of the hybrid system is less expensive than a human‐only system. These findings highlight the complementarity of human raters and automated scoring engines. Although current standards focus on the performance of human raters and automated scoring engines in isolation, we recommend that practitioners also report on the performance of the hybrid scoring system as a whole.