Answers are grounded in this paper and cite their sources.
Background Large language models can report numerical surgical risks, but discrimination, probability accuracy and stability under equivalent input representations may differ. We evaluated these properties across two institutions using locally deployed language models and common clinical predictors. Methods This retrospective study included 1,909 digestive resection operations in VitalDB and 1,791 scheduled-resection candidates in MOVER. The outcome was in-hospital death or discharge more than 14 days after anesthesia end. Qwen3-14B and Gemma3-12B, both quantized to Q4_K_M, generated probabilities from ten fields at anesthesia end. Logistic regression and gradient boosting used the same clinical fields. Source evaluation used patient-grouped folds; external evaluation transferred models and source-only logistic calibration maps without fitting to MOVER outcomes. Paired tests compared identical-text replay, reversed field order and equivalent hemoglobin/albumin unit conversion. Uncertainty was estimated with 1,000 patient-cluster bootstrap samples conditional on fitted models. Results The outcome occurred in 220/1,909 VitalDB operations (11.5%) and 140/1,791 MOVER candidates (7.8%). External areas under the receiver operating characteristic curve were 0.788 for logistic regression, 0.782 for gradient boosting, 0.656 for Qwen and 0.722 for Gemma. Raw Brier scores were 0.0760, 0.0723, 0.0699 and 0.0689, respectively. Source calibration increased external Brier scores by 0.01065 for Qwen (95% interval 0.00791–0.01320) and 0.00200 for Gemma (0.00015–0.00377). Among 429 MOVER operations with a convertible laboratory value, unit conversion changed raw risk by an average absolute 0.93 percentage points for Qwen (0.75–1.14) and 6.49 for Gemma (5.77–7.20). The corresponding changes after fixed source calibration were 1.20 and 2.66 percentage points. Some representation changes improved outcome error, whereas others worsened it. Conclusions In these two quantized models, lower aggregate probability error coexisted with weaker discrimination, unsuccessful calibration transfer and representation-dependent risks. External outcome evaluation and paired representation tests provide complementary evidence when assessing numerical language-model risk estimates.
No takes yet. Share an insight, caveat, or question.
Shi et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: