Evaluating reasoning models for therapy recommendations in gastrointestinal stromal tumors: expert and LLM-based evaluations of OpenAI o1 and DeepSeek-R1

Purpose This study aims to evaluate two advanced reasoning LLMs in generating treatment recommendations for real-world gastrointestinal stromal tumor (GIST) cases and assess their concordance with multidisciplinary team (MDT) decisions at a certified tertiary sarcoma center. Methods Sixty-five real-world GIST cases from a tertiary sarcoma center were used to compare two advanced reasoning models—OpenAI o1 and DeepSeek-R1. Recommendations were generated using a multi-expert prompting strategy with current clinical guidelines as context. Five sarcoma specialists and an independent LLM (Mistral AI) evaluated alignment with MDT decisions and guideline concordance. Results OpenAI o1 achieved higher concordance with MDT decisions than DeepSeek-R1 (76.9% vs. 53.8%, p < 0.001) and more recommendations aligned with either MDT or guidelines (80.0% vs. 63.1%, p = 0.005). Inter-rater reliability among human evaluators was excellent (ICC = 0.929). The LLM judge’s evaluations showed moderate agreement with human assessments (κ = 0.647). OpenAI o1 responses were significantly longer than those of DeepSeek-R1 and MDT records. Conclusions OpenAI o1 outperformed DeepSeek-R1 in generating clinically relevant GIST therapy recommendations. The study highlights the feasibility of using LLMs both as decision support tools and as evaluators (“LLM-as-a-judge”) in oncology, while emphasizing the need for expert oversight in clinical deployment.