Designing and Evaluating a Socratic-Scaffolded Conversational AI Assistant for Educational Tasks: Cross-Cohort Effectiveness, User Perception, and the Effort Signature

Conversational artificial intelligence (AI) assistants are proliferating, yet direct-answer interaction can yield short-term task gains alongside skill regression on later unaided work. We report the design and three-semester evaluation of a Socratic-scaffolded conversational AI assistant in an introductory programming course at a northern Taiwanese university. The system layers retrieval grounding to instructor materials, Socratic-by-default response prompting, and metacognitive interaction tracking using Bloom's taxonomy. A cross-cohort task-effectiveness study (pooled N = 82) found large pre-to-post gains (d = 1.20) replicated across cohorts, although the absence of a non-scaffolded control precludes causal attribution. A perception study (N = 203) showed positive usability, acceptance, and satisfaction (normalized means 75–79%), with no decay across semesters. An interaction-pattern study (N = 90) revealed the predicted effort signature: users judged the Socratic mode more effortful yet endorsed it for perceived depth of understanding. We discuss implications for designing assistants for skill-building rather than task completion.