Comparative Analysis of Large Language Models for Automated Question Generation From Video-Based Learning Content

This is the authors' abstract. We don't add key points for this paper.

The increasing adoption of video-based learning has presented a significant challenge in effectively assessing learner’s comprehension and engagement. This paper addresses this gap by conducting a comparative analysis of ten state-of-the-art large language models (LLMs) for Automated Question Generation (AQG) from video transcripts. Our evaluation centres on assessing the quality of four question types: short-answer, multiple-choice, boolean, and gap-fill. Our major contributions are three-fold. First, we propose an evaluation framework for comparing multiple LLMs in their ability to generate different question types, identifying the models best suited for each based on five key evaluation metrics–relevance, reading comprehension, question difficulty, clarity, and context utilisation–that holistically assess question quality. Second, we release the prompts used for both question generation and evaluation, providing a valuable resource for future AQG research. Third, we uncover the strengths and weaknesses of each LLM, identifying which question types require further improvement, and uncovering statistically significant differences in AQG performance across the LLMs. By providing an in-depth comparative analysis, this study provides important insight into the development of more effective AQG systems, contributing to the broader integration of AI in automated assessment in video-based learning environments.