Improving non-expert performance in musculoskeletal MRI protocoling through a large language model

MRI protocoling, the process of selecting appropriate sequences, is especially complex in musculoskeletal (MSK) cases. This study evaluates whether a large language model (LLM) can effectively assist non-experts in MRI protocoling. We collected two retrospective datasets of musculoskeletal (MSK) MRI orders: (1) a prompt-development dataset (Jan–Dec 2023; 538 cases) and (2) a testing dataset (Jan–Jun 2024; 107 cases). A prompt was refined on GPT-4o via the web-based ChatGPT interface to use patient information and complete an MRI protocol worksheet. For the testing dataset, 6 non-experts (three radiology residents [RD] and three radiographers [RG]) completed the worksheet with and without LLM-assistance. All worksheets were scored using a clinical pass system, categorizing responses as “excellent (4 points),” “acceptable (3 points),” “insufficient (2 points),” or “failure (1 point)” based on potential outcomes. We compared clinical pass scores between LLM-assisted and human-only worksheets using the Wilcoxon signed-rank test and a mixed linear model. Fleiss’ kappa across five iterations was measured for stochasticity. LLM-assisted worksheets had higher average clinical pass scores than human-only worksheets for both RD (3.42 vs. 3.18, p < 0.001) and RG (3.06 vs. 2.9, p < 0.001). LLM-assistance decreased the proportion of “insufficient” and “failure” cases, potentially decreasing MRI re-checks, with an average reduction of 12.2% (13 out of 107) for RD and 8% (8.7 out of 107) for RG. This improvement was consistent across professions (p = 0.43). LLM outputs showed strong reliability across five iterations (κ = 0.66). LLM-assistance improves the performance of non-experts in MSK MRI protocoling and has the potential to reduce repeat scans.