Dsai-05 Fine-Tuning Large Language Models Using NCCN Guidelines to Streamline Physician Assessment and Plans for CNS Metastatic Cancer: A Trial-in-Progress

Management of central nervous system (CNS) metastatic cancer can be overwhelming to many oncologists who do not actively deal with it on a day-to-day basis. NCCN clinical guidelines are constantly evolving, and with personalized patient data being so different, it can be challenging to stay up to date with treatments while also being able to make decisions on vast clinical trials and potential second- and third-line options. Large language models (LLMs), such as GPT-4o and GPT4.5, have shown promise in supporting clinical decisions. Yet, they require rigorous validation to ensure safety and accuracy, particularly in complex conditions like CNS metastases. This ongoing trial assesses the performance of an NCCN-guideline-tuned LLM designed to streamline physician decision-making for CNS metastatic cancer treatment planning. Objectives include evaluating the AI model’s capability to generate accurate, NCCN-concordant first-line treatment recommendations, assessing its effectiveness in suggesting second-line and subsequent therapies incorporating patient-specific considerations, quality of life, and clinical trial data, and identifying potential inaccuracies or "hallucinations" in AI recommendations. The study employs a cross-sectional design using standardized prompts to generate AI treatment recommendations referencing an offline NCCN guideline database specific to CNS metastatic cancers. Treatment scenarios will ask the AI to cover first-line options, backup strategies post-initial treatment failure, and complex multi-line recommendations with associated survival data, recurrence rates, hazard ratios, and clinical trial information. Recommendations will be independently reviewed by Oncologists using a Generative AI Performance Score (G-PS) calculator to validate adherence to guidelines, clinical appropriateness, and frequency of inaccuracies. If successful, this AI-driven approach could significantly improve clinical efficiency, enhance treatment accuracy, and support personalized patient care. Enrollment and evaluations are ongoing; pre-specified analysis endpoints have not yet been reached, and data will be recorded in time.