Ordinal uncertainty grading in chest radiograph reports using large language models: a REFINE-aligned feasibility and methodology study

PURPOSE: Radiologists differ in how strongly they hedge. We tested whether large language models can grade linguistic uncertainty in chest radiograph reports on a 0-3 scale and whether aggregate agreement conceals category-specific differences. METHODS: We used 3,927 public chest radiograph reports (Open-I, Indiana University). Three radiologists independently scored 100 reports from 0 (definitive wording) to 3 (extensive ambiguity). Majority vote, with prespecified attending adjudication for seven complete three-way disagreements, formed the reference standard. Claude Opus 4.5 and Gemini 2.5 Pro scored the same reports from a fixed prompt without task-specific training. We measured agreement using Cohen's quadratic weighted kappa (QWK). We also recorded per-score recall, ordinal error, prompt and reference-standard robustness, and comparison with a keyword-counting baseline. We then scored the remaining 3,827 reports with both models. RESULTS: Human pairwise QWK ranged from 0.421 to 0.659. Claude and Gemini had similar aggregate agreement: QWK: 0.807 [95% confidence interval (CI), 0.722-0.876] and 0.814 (95% CI, 0.735-0.877), respectively; paired difference: 0.007 (95% CI, -0.062 to 0.079). In this validation sample, Claude identified 5/8 reports exactly as Score 3 and Gemini 1/8; Score-3 precision was 5/11 and 1/3, respectively. Of Gemini's seven exact Score-3 misses, six were assigned the adjacent Score 2 category. Reference-standard sensitivities yielded QWK: 0.810-0.841; prompt-variant pairwise agreement ranged from 0.818 to 0.913; repeat runs were identical for 29/30 Claude and 28/30 Gemini reports. All 3,827 Gemini discovery requests completed successfully, with token accounting recorded for the discovery cohort. CONCLUSION: The approach is workable at the corpus scale, but similar aggregate agreement concealed differences in exact Score-3 classification, largely involving the Score-2/3 boundary. The score reflects wording, not clinical risk, and must not be used for risk triage. Clinical or large-scale use requires external validation enriched for ambiguous cases and prior radiologist calibration. CLINICAL SIGNIFICANCE: These scores may help radiologists audit reporting language and build balanced research evaluation sets, but downstream benefit must be demonstrated rather than assumed.