SPRAG: a Short Programming Related Answer Grading Dataset

Automated Short Answer Grading (ASAG) is one of the most stud-ied applications of NLP in the education domain. Most of the work in this area focuses on natural language responses. The main contribution of this work is the building and benchmarking of the Short Program-ming Related Answer Grading dataset. The corpus contains questions and answers taken from python programming subjects involving symbols, keywords, and no grammar. We propose data collection, preprocessing, and annotating methods that enable the creation of the corpus. This paper also provides an initial analysis of the corpus. A range of pre-trained sentence-transformer models are evaluated on the dataset for both binary and multi-class classification. Our best fine-tuned model yields an accuracy of 86.16% on binary classification and 56.11% on multiclass classification. We find that fine-tuning the pre-trained models results in the best performance for binary classification. The dataset is freely available to the public.