Formative and summative use of automatic conversational assessment- in higher education

The increasing use of generative AI in education raises concerns about misuse, particularly regarding students’ work originality. Conversational assessments offer a promising alternative for measuring students’ subject mastery but are difficult to scale. Building upon a prototype for automatic conversational assessment (ACA) using large language models, technology, this study examines whether the ACA-method can summatively assess high-order cognitive skills and whether its formative use supports learning ethically and responsibly. A mixed-methods design combines system logs, assessment results, survey data on usefulness, trust, and learning perspectives and qualitative analysis of student-bot interaction. Eleven batchelor level statistics students participated in a formative assessment, and six in a subsequent summative assessment. Results showed that students who used the bot performed significantly better on the final exam, indicating the bot’s potential educational value. Overall, perceptions were positive, though feedback qualityneeds improvement. The findings provide empirical support for the ACA-bot’s potential in this educational context.