RadOnc-GPT: An Exploratory Feasibility Study of a Prototype LLM Agent for Radiation Oncology Outcomes Labeling and Registry Auditing

Purpose Manual outcomes labeling limits the scale and timeliness of radiation oncology research and registry maintenance. We explored the technical feasibility of RadOnc-GPT, a prototype GPT-4o-based agent that retrieves structured and longitudinal clinical evidence from institutional systems and returns schema-constrained outcome labels and supporting rationale. Material and methods In this single-institution retrospective feasibility study, we evaluated two tiers of increasing complexity: (i) reproduction of structured demographic and radiotherapy-course data and (ii) outcomes labeling using structured and unstructured data for mandibular osteoradionecrosis in head-and-neck cancer and cancer recurrence in separate prostate and head-and-neck cancer cohorts. Only cases in which the model and baseline registry disagreed underwent review by one unblinded domain-specific oncologist per dataset. Post-review values are therefore described as discrepancy-adjudicated and condition on concordant baseline labels being correct. Results Across 500 patients, the prototype matched all six demographic/identifier fields (3,000/3,000) and reproduced per-course treatment information in 497/500 patients. In 395 evaluable outcomes-labeling cases, initial accuracy/F 1 were 85.0%/63.2% for osteoradionecrosis, 92.5%/92.5% for prostate recurrence, and 92.7%/90.9% for head-and-neck cancer recurrence. Review of 47 discordant cases classified 24 as baseline-label errors, 18 as model errors, and 5 as indeterminate. Discrepancy-adjudicated accuracy/F 1 were 95.2%/89.7%, 95.0%/95.1%, and 96.3%/95.5%, respectively. In an assumption-based pooled stress test, treating 1%, 3%, or 5% of the 348 concordant cases as hidden model errors reduced pooled accuracy from 95.4% to 94.4%, 92.6%, and 90.8%, respectively. These scenario analyses do not estimate the true latent-error rate. Conclusion In this single-institution feasibility study, RadOnc-GPT reproduced selected structured radiotherapy data, generated candidate outcome labels from longitudinal clinical records, and flagged model–registry discrepancies for expert review. These results support further evaluation of LLM-assisted outcomes labeling and registry auditing, including blinded multi-reviewer adjudication, multi-institutional validation, and testing in natural-prevalence cohorts.