What question did this study set out to answer?

To compare student performance, question difficulty, and discriminatory power between medical examination questions generated by ChatGPT-3.5 and those created by experienced human educators.

December 28, 2023Academic Medicine

Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions

View Full Paper

Ask AI

Bookmark

Authors

MLMatthias Carl LaupichlerUniversity Hospital Bonn JRJohanna Flora RotherUniversity Hospital Bonn IKIlona C Grunwald KadowUniversity of Bonn

Discussion

Member takes

Implication

Comparative study finds lower discriminatory power in ChatGPT-generated exam questions than human-written items in medical students, highlighting challenges in automated assessment creation.

Key Points

To compare student performance, question difficulty, and discriminatory power between medical examination questions generated by ChatGPT-3.5 and those created by experienced human educators.
Assessed N=161 medical students in a formative neurophysiology paper-and-pencil test before their summative exam in summer 2023.
Administered 46 multiple-choice questions (25 created by an experienced medical educator and 21 created by ChatGPT-3.5 using specified learning objectives) in random order.
Collected student responses to exam items and asked participants to classify each question as either human- or ChatGPT-generated.
Item difficulty showed no statistically significant difference between human-generated and LLM-generated questions.
Discriminatory power was statistically significantly higher for human-written questions than LLM-written questions (mean = 0.36, SD = 0.09 vs. mean = 0.24, SD = 0.14; P = .001).
Students correctly identified the authoring source (human vs. LLM) for an average of 57% of the questions.

Cite This Study

Laupichler et al. (2023) studied this question.

synapsesocial.com/papers/6a0fd1975725bbd5cc601ff0 https://doi.org/10.1097/acm.0000000000005626

View Full Paper

Ask AI

Bookmark

Implication

Comparative study finds lower discriminatory power in ChatGPT-generated exam questions than human-written items in medical students, highlighting challenges in automated assessment creation.

Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions

Discussion

Member takes

Implication

Key Points

Cite This Study

Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider