Automated RECIST tumor response classification through prompt-guided large language models

Abstract This study investigates whether an entirely offline, general-purpose large language model (LLM) can reliably automate the classification of routine radiology reports according to RECIST (Response Evaluation Criteria in Solid Tumors) guidelines, focusing on how different prompting strategies support accurate, privacy-preserving tumor response assessment without additional model fine-tuning. An offline, in-house implementation of LLaMA-3.3 (70B) was used to classify real-world CT imaging reports from oncology patients. Reports were authored following RECIST-structured reporting but had outcome labels programmatically withheld prior to processing. Three prompting strategies—zero-shot, few-shot, and chain-of-thought prompting—were tested to guide the model in assigning RECIST categories: Baseline (BL), Complete Response (CR), Partial Response (PR), Stable Disease (SD), and Progressive Disease (PD). Model outputs were benchmarked against original expert labels using accuracy, precision, recall, and F1 scores. Across all tested prompting strategies, the LLaMA-3.3 model achieved strong classification performance. The best results were obtained with chain-of-thought prompting, reaching micro F1 scores of 0.81 across all RECIST categories. Overall model predictions aligned well with human expert assessments. Operating entirely offline within hospital infrastructure, the system preserved full compliance with stringent data privacy requirements. Prompt-driven large language models can accurately classify tumor response categories from real-world radiology reports in a scalable, reproducible, and privacy-preserving manner. Offline LLM deployment, combined with optimized prompting strategies, offers a promising approach for automating structured oncology report interpretation, potentially enhancing consistency and efficiency in clinical decision support workflows.