Contrasting State-of-the-Art in the Machine Scoring of Short-Form Constructed Responses

AbstractThis study compared short-form constructed responses evaluated by both human raters and machine scoring algorithms. The context was a public competition on which both public competitors and commercial vendors vied to develop machine scoring algorithms that would match or exceed the performance of operational human raters in a summative high-stakes testing environment. Data (N = 25,683) were drawn from three different states, employed 10 different prompts, and were drawn from two different secondary grade levels. Samples ranging in size from 2,130 to 2,999 were randomly selected from the data sets provided by the states and then randomly divided into three sets: a training set, a test set, and a validation set. Machine performance on all of the agreement measures failed to match that of the human raters. The current study concluded with recommendations on steps that might improve machine-scoring algorithms before they can be used in any operational way. Notes1 Error rate was a combination of two components that were equally weighted: (a) the error rate of words over the total number of words and (b) the error rate of characters over the total number of characters. The latter component takes into consideration characteristics like punctuation.