A realist synthesis review of algorithmic approaches to developing multiple-choice questions in the natural sciences

This review study aimed to identify factors influencing the effectiveness of algorithmically generated test items, the quality of the tasks generated, and their broader impact on education when algorithmic procedures are implemented. The applied method of realist synthesis enabled the integration of evidence from previous studies to identify successful and unsuccessful configurations within the system Mechanism – Context – Outcomes. The mechanism encompasses diverse components of the algorithmized MCQ development, from the theoretical basis to workflow procedures. The context is understood as the specific conditions in which these mechanisms operate, while the outcomes are the results of the interaction between mechanisms and context. The findings obtained from 19 selected articles suggested that in creating test items, the design of algorithmic definitions plays a more critical role than their technical implementation. Of the measured outcomes, productivity was the only area in which automated multiple-choice question (MCQ) generation outperformed manual creation. However, it struggled to produce test items with sufficient complexity. The least favourable conditions arise when developers pursue “universality,” producing algorithms that are not tailored to specific assessments, disciplines, or item banks. The most effective approach involves a team of developers with diverse expertise, equipped with well-defined algorithms, collaboratively creating MCQs for structured item banks under state or university regulations. Algorithmizing improves assessment techniques. It contributes to the reliability, transparency, and trustworthiness of examinations, enhancing their societal value, and also teachers’ expertise in creating educational materials for more differentiated, personalized, and targeted learning.