Evaluation of a Problem-Based Learning Program’s Effect on Artificial Intelligence Ethics among Japanese Medical Students: Mixed-Methods Study (Preprint)

BACKGROUND The impact of the rapid advancement of artificial intelligence (AI) on the medical domain is significant. Accordingly, it is necessary to integrate AI education into medical school curricula. However, the integration of AI education remains limited. A key challenge to this integration is the discrepancy between medical students' positive perceptions of AI and their actual competencies, with research in Japan identifying specific gaps in the students' competencies in understanding regulations and discussing ethical issues. OBJECTIVE This study evaluates the effectiveness of an educational program designed to improve medical students' competencies in understanding legal and ethical AI-related issues. It addresses the following research questions: (1) Does this educational program improve students' knowledge of AI and its legal and ethical issues, and what is each program element’s contribution to this knowledge? (2) How does this educational program qualitatively change medical students' thoughts on these issues from an abstract understanding to a concrete and structured thought process? METHODS This study adopted a mixed-methods, convergent parallel design using a single-group pretest/posttest framework involving 118 fourth-year medical students. The one-day intervention comprised a lecture and Problem-Based Learning (PBL) session centered on a clinical case. Further, a 24-item multiple-choice questionnaire (MCQ) was administered at three time points (pretest, mid-test, and posttest), and descriptive essays were collected before and after the intervention. Data were analyzed using linear mixed-effects models; the Wilcoxon signed-rank test; and text mining, including cooccurrence network analysis with Jaccard coefficients. An optional survey on student perceptions (ARCS model) was conducted (N=76, response rate = 64.4%), as well. RESULTS The program caused a significant increase in objective knowledge from the pretest (median 17(15-18)) to posttest (median 19(17-21); β = 1.42, P < 0.001) periods. However, no statistically significant difference was noted between the score gains of the lecture phase and PBL phase (P = .54). Further, qualitative analysis revealed the significant transformation of cooccurrence network structures (Jaccard coefficients: 0.116 and 0.121) from fragmented clusters to integrated networks. Word frequency analysis revealed the increased use of professional and ethical terms (e.g., bias and personal information). The optional survey indicated that students' confidence (mean = 3.78, standard deviation (SD) = 0.87) was significantly lower than their perception of the program's relevance (mean = 4.20, SD = 0.71; P < 0.001). CONCLUSIONS The PBL program effectively improved overall knowledge and, more importantly, qualitatively transformed students' thinking about AI ethics from an abstract level to a concrete and structured thought process based on clinical context. The discrepancy between the quantitative and qualitative results of the study suggests MCQs’ insufficiency in assessing the higher-order skills fostered by PBL. Finally, this mixed-methods study indicates that PBL is an effective pedagogical method for AI ethics education.