Benchmarking LLM decision support in inflammatory aneurysms: DeepSeek-R1, DeepSeek-V3 and ChatGPT-4o

Decision support for inflammatory aneurysms demands guideline-concordant specificity. We compared three large-language-model systems-DeepSeek-R1, ChatGPT-4o and DeepSeek-V3-on accuracy and time efficiency for core diagnostic, interventional and surveillance decisions. In a prospective, rater-blinded evaluation, each model completed a 50-item instrument aligned to contemporary aortic guidance in five independent clean-context sessions (15 answer sets). Five consultant vascular specialists scored all outputs using a prespecified 0/1/2 rubric. The primary outcome was total accuracy (0-100); secondary outcomes were completion time and domain subscores. Inter-rater reliability was summarised with Fleiss' κ and ICC(2,k). Mean total scores were 89.8 ± 1.8 (DeepSeek-R1), 88.4 ± 1.8 (ChatGPT-4o) and 77.8 ± 1.5 (DeepSeek-V3). One-way ANOVA showed a significant model effect (F(2,12) = 75.34, p = 1.61 × 10⁻⁷); Tukey contrasts indicated near-parity between R1 and 4o (Δ 1.44, p = 0.397) with both exceeding V3 (R1-V3 Δ 12.00, p = 2.80 × 10⁻⁷; 4o-V3 Δ 10.56, p = 1.13 × 10⁻⁶). Completion time separated strongly (R1 58.5 ± 0.95 s; 4o 33.4 ± 0.46 s; V3 19.8 ± 0.89 s; F(2,12) = 3053, p = 5.69 × 10⁻¹⁷), and accuracy rose with time (Pearson r = 0.802, 95% CI 0.49-0.93, p = 3.24 × 10⁻⁴; Theil-Sen 0.285 points·s⁻¹). Differences concentrated in guideline-derived items (R1 36.44 ± 1.49; 4o 36.32 ± 1.45; V3 29.16 ± 0.86). Inter-rater agreement was substantial at item level (κ = 0.662) and excellent for totals (ICC[2,k] = 0.973); intra-model variability was ≈ 2%. In rule-dense, safety-critical decisions for inflammatory aneurysms, slower, reasoning-oriented decoding yields the highest accuracy, while a low-latency multimodal system achieves near-parity at markedly shorter times; a throughput-optimised model trades speed for omission-prone answers. Model selection should align with task criticality, with human verification retained.