Protocol for Conducting a Scoping Review on The Use of AI in Automated Scoring of Short-Answer Questions in Medical Education

Assessment is a central component of medical education, requiring methods that can assess learners’ knowledge and reasoning while maintaining acceptable reliability and feasibility. Short-answer questions (SAQs) require learners to construct responses rather than select from predetermined options and can support assessment of applied knowledge and reasoning. However, their use is constrained by the time and resources required for scoring and by potential variability among human graders. These limitations have increased interest in automated approaches to scoring written responses. Artificial intelligence (AI)-based systems using machine learning, deep learning, transformer-based models, large language models, and related natural language processing techniques are increasingly being applied to the automated scoring of SAQs. Such systems may reduce grading workload, improve scoring consistency, and facilitate timely feedback. However, questions remain regarding their scoring performance, reliability, validity, fairness, transparency, and feasibility for use in educational settings. The approaches and evaluation methods reported across the literature are heterogeneous, creating a need to systematically map the available evidence. This scoping review aims to map the literature on AI-based automated scoring of SAQs in medical education. Specifically, it will identify and characterize the AI-based models used; map the measures and metrics used to assess scoring performance, reliability, and validity; describe reported advantages, limitations, and implementation challenges; and examine how studies have compared the scoring performance, reliability, validity, and fairness of AI-based systems with human graders. The review will follow the Joanna Briggs Institute methodology for scoping reviews and will use the Population–Concept–Context framework to guide study selection and data extraction. English-language original empirical studies published from 2015 onward involving medical students and AI-based automated scoring of SAQs will be eligible. The findings will provide a structured overview of current approaches and evaluation practices, identify gaps in the evidence, and inform future research and implementation of AI-based automated scoring in medical education.