Streamlining endometriosis MRI reporting: Automated extraction of #Enzian scores from pelvic MRI reports using local and online LLMs

Background The #Enzian classification standardizes endometriosis reporting but requires specialized training. This study evaluated whether large language models (LLMs) can accurately extract #Enzian scores from MRI reports to support novice radiologists and clinicians. Methods This retrospective study included 186 pelvic MRI reports (2022–2025). Two uroradiologists established report-based reference #Enzian scores for all cases via consensus. Twelve LLMs—six online (Gemini 2.5 Pro, Claude Sonnet 4.0, GPT-5, o3, Grok 4, Claude Opus 4.1) and six local (Llama 3.1 8B, Aya Expanse 8B, MedGemma 27B, DeepSeek R1 32B, Llama 3.3 70B, GPT-OSS 120B)—were tested with one-shot prompting. A 50-case subset was additionally scored by 12 trainees without #Enzian experience as novice benchmark. Accuracy for exact #Enzian score matching was assessed against the expert reference; 95% bootstrap confidence intervals were calculated for accuracies, and logistic regression with Dunnett-adjusted simultaneous inference and cluster bootstrapping was used to compare LLM and trainee performance. Results In the full 186-report cohort, online LLMs achieved 93.8% [95% CI: 92.4–95.1] to 95.8% [95% CI: 94.7–96.8] accuracy vs. expert reference, while local models reached 72.7% [95% CI: 70.0–75.5] to 92.5% [95% CI: 91.1–93.8]. In the 50-case subset, trainees achieved 87.4% [95% CI: 84.4–89.9] accuracy. Online LLMs reached 88.9% [95% CI: 86.1–91.3] to 92.4% [95% CI: 90.1–94.7], with three models significantly outperforming trainees: Gemini 2.5 Pro (OR 1.81, p<0.001), Grok 4 (OR 1.60, p=0.012), and o3 (OR 1.57, p=0.034). Local models achieved 61.7% [95% CI: 56.4–67.0] to 87.1% [95% CI: 83.9–90.0] and were significantly inferior to trainees (p<0.001) except GPT-OSS 120B (OR 0.98, p=1.00). Online LLM API costs for the full cohort ranged from $3.82 to $42.26. Conclusions Online LLMs accurately extracted #Enzian scores from MRI reports with performance exceeding that of novice radiologists, suggesting potential as training aids and for standardized reporting in surgical planning. Out-of-the-box local models demonstrated limited performance.