Evaluating an Automatically Scorable, Open‐Ended Response Type for Measuring Mathematical Reasoning in Computer‐Adaptive Tests

The first generation of computer‐based tests depends largely on multiple‐choice items and constructed‐response questions that can be scored through literal matches with a key. This study evaluated scoring accuracy and item functioning for an open‐ended response type where correct answers, posed as mathematical expressions, can take many different surface forms. Items were administered to 1,864 participants in field trials of a new admissions test for quantitatively oriented graduate programs. Results showed automatic scoring to approximate the accuracy of multiple‐choice scanning, with all processing errors stemming from examinees improperly entering responses. In addition, the items functioned similarly in difficulty, item‐total relations, and male‐female performance differences to other response types being considered for the measure.