Objectives : This work explores the application of two advanced state-of-the-art models, BERT (Bidirectional Encoder Representations from Transformers) and USE (Universal Sentence Encoder), to automate the grading of short answers. Methods: This work investigates the use of BERT and USE models for automatically grading short answers. The research utilizes HP:SAS dataset containing manually graded responses by two human evaluators. The student responses as well as model answers responses of question 1 and question set 6 are then processed using the BERT and USE models, with scores generated based on cosine similarity measures between student answers and predefined model answers. Findings: The work demonstrates that BERT and USE embeddings can effectively capture contextual and semantic similarity, their performance is heavily dependent on the function which generates the score. Our finding reveal that a non-linear mapping function mimics the human grading more than a linear mapping function. Such a function enhances accuracy (0.67) and reduces the error (0.617) by computing Pearson correlation coefficient and RMSE respectively. Notably, longer responses achieved higher Pearson correlations (0.67) than shorted answers (0.59). The results bring out usability and choice aspects of BERT and USE in relation to ASAG, contributing to the understanding of their application across various answers. We conclude with a weighted ensemble method combining BERT and USE with subject- specific strictness parameter (k) provides a robust framework for automated assessment. Novelty: Evaluates and compares two deep learning models for automatic short answer grading, a scarcely explored area. A novel contribution is the granular analysis across different scoring ranges across two question sets of the dataset. The novelty of this work lies in the transition from linear scoring to non-linear mapping framework. This approach introduces a tunable sigmoid- based ensemble that would replicate human assessment. Finally, a comparison with existing studies demonstrates very limited research. Keywords: Bidirectional Encoder Representations from Transformers (BERT), Universal Sentence Encoder (USE), Transformer, word embedding, Non-linear mapping, deep learning, short answer grading
Chakraborty et al. (Sun,) studied this question.