Evaluating AI tutor feedback in medical education: a case study of computer basics and applications course for undergraduates

Owing to a restricted network and limited time on the computer course, medical undergraduates in the curricula offered by our university demanded timely and high-quality feedback. With the rapid advancement of artificial intelligence, intelligent tutoring systems can provide a solution. This article explored the main factors of AI feedback on teaching outcomes by employing a fine-tuned local language model. First-year medical students (n = 200) communicated with an AI tutor before and after the computer course this semester. After communicating with an AI tutor fine-tuned by specialized course learning content and five-year feedback dialogue, these students marked the AI tutor by filling in a structured questionnaire. Once the data were recorded, the PLS-SEM model explored the relations among these features. The results presented an apparent gap between AI and human tutor (W = 17.894, p < 0.005). The model in PLS-SEM illustrated learning outcome was not highly consistent with AI tutor performance and student features (HTMT 0.314, 0.228, respectively), while AI tutor was rated higher in accuracy at 3.40 (p < 0.001). In conclusion, the AI tutor demonstrated potential to answer questions once the knowledge can be trained well. In the short learning term, an AI tutor can present much clearer feedback in a special course, and can be used as a quick intelligent notebook.