The performance of large language models (LLMs)-DeepSeek in the generation of in-training examination questions in radiology resident education is understudied. In this study, multiple-choice questions (MCQs) were generated by DeepSeek and written by radiology experts according to Chinese educational standards. Fourteen MCQs (A1, A2, A3/A4 types) from each source were randomly intermixed into a 28-item online test, administered to 40 radiology residents (17 second-year, 23 third-year residents). Participants completed the closed-book exam by answering each question, identifying its origin (DeepSeek or expert), and rating it on perceived difficulty, curriculum relevance, overall quality, and clinical realism using standardized scales. There were no significant differences between DeepSeek-generated and expert-written questions in overall correct response rates or source attribution accuracy. Furthermore, item-type stratification revealed that residents had similar correct response rates for DeepSeek-generated and expert-written A1 questions, while lower correct response rates for DeepSeek-generated A3/A4 questions than expert-written ones. Subjective evaluations showed lower ratings of clinical scenario realism for DeepSeek-generated A2 questions compared to expert-written ones, with a more pronounced difference among third-year residents. DeepSeek demonstrates potential in the generation of in-training examination questions in radiology resident education. However, it is less effective than human experts in creating higher-order questions requiring clinical scenarios.
Qian et al. (Tue,) studied this question.