Auto-marking short answer questions in science: The foundational years of transformer-based models from BERT to GPT-4

Auto-marking, also referred to as auto-scoring or auto-grading, is the automated assessment of student responses using computational methods. Applying auto-marking to short answer questions in science education has long posed significant challenges due to the domain-specific knowledge required and the diverse ways students express understanding. The emergence of transformer-based models from 2017 onwards represents a potential breakthrough in addressing these challenges. This scoping review explored the performance of early transformer-based auto-markers during this foundational period (2017-2024), providing insights that remain relevant as the technology continues to evolve. We followed a systematic process, adhering to relevant PRISMA guidelines. Our review included literature from 2017 to early 2024, focusing on English natural language responses on science content in an educational assessment context. A final set of 21 articles was reviewed and coded in depth to answer our research questions which explored the types of auto-marking models being used, the datasets used to fine-tune and test them, and their performance. The most commonly used models in this context were BERT models and BERT variants, which increased in frequency reaching a peak in 2021. After 2021, papers using GPT models started to appear. The SciEntsBank dataset was the most commonly used to test auto-markers but several other datasets (e.g., ASAP SAS, Beetle) also featured in our review. BERT models generally performed better than previous models on the SciEntsBank dataset. As of yet, GPT models have not been evaluated on SciEntsBank but there was one study in the review that directly compared GPT-3.5 and BERT base and found that GPT-3.5 outperformed BERT base across different items and item types. The review also shows that models that utilise additional forms of data like textbooks and marking rubrics seem to consistently outperform models without these, and that auto-markers may present issues in terms of low reliability, lack of explainability and bias. By examining auto-markers in this foundational period, this review identifies enduring patterns, critical gaps, and methodological challenges that will inform the development of future auto-marking systems regardless of specific model architectures.