Large language model processing capabilities of ChatGPT 4.0 to generate molecular tumor board recommendations—a critical evaluation on real world data

Background Large language models (LLMs) like ChatGPT 4.0 hold promise for enhancing clinical decision-making in precision oncology, particularly within molecular tumor boards (MTBs). This study assesses ChatGPT 4.0’s performance in generating therapy recommendations for complex real-world cancer cases compared to expert human MTB (hMTB) teams. Methods We retrospectively analyzed 20 anonymized MTB cases from the Comprehensive Cancer Center Augsburg (CCCA), covering breast cancer (n = 3), glioblastoma (n = 3), colorectal cancer (n = 2), and rare tumors. ChatGPT 4.0 recommendations were evaluated against hMTB outputs using metrics including recommendation type (therapeutic/diagnostic), information density (IDM), consistency, quality (level of evidence [LoE]), and efficiency. Each case was prompted thrice to evaluate variability (Fleiss’ Kappa). Results ChatGPT 4.0 generated more therapeutic recommendations per case than hMTB (median 3 vs. 1, p = 0.005), with comparable diagnostic suggestions (median 1 vs. 2, p = 0.501). Therapeutic scope from ChatGPT 4.0 included off-label and clinical trial options. IDM scores indicated similar content depth between ChatGPT 4.0 (median 0.67) and hMTB (median 0.75; p = 0.084). Moderate consistency was observed across replicate runs (median Fleiss’ Kappa=0.51). ChatGPT 4.0 occasionally utilized lower-level or preclinical evidence more frequently (p = 0.0019). Efficiency favored ChatGPT 4.0 significantly (median 15.2 vs. 34.7 minutes; p < 0.001). Conclusion Incorporating ChatGPT 4.0 into MTB workflows enhances efficiency and provides relevant recommendations, especially in guideline-supported cases. However, variability in evidence prioritization highlights the need for ongoing human oversight. A hybrid approach, integrating human expertise with LLM support, may optimize precision oncology decision-making.