Key result
Adding surgeon input fails to improve surgical risk model discrimination.
Why the study?
Surgical risk models are typically evaluated independently of how surgeons interpret their predictions after deployment, leaving it unclear whether surgeon responses to deployed risk predictions add prognostic information.
Does incorporating surgeon risk assessments improve the discrimination of deployed surgical risk prediction models (NSQIP-SRC, POTTER, or MySurgeryRisk) for 30-day surgical complications?
Observational (n=1,702)
Yes
Does incorporating surgeon risk assessments improve the discrimination of deployed surgical risk prediction models (NSQIP-SRC, POTTER, or MySurgeryRisk) for 30-day surgical complications?
Answers are grounded in this paper and cite their sources.
Surgeons frequently disagree with deployed surgical risk prediction models (53% of cases), and incorporating their assessments does not significantly improve overall model discrimination for 30-day complications.
BACKGROUND: Surgical risk models are typically evaluated independently of how surgeons interpret their predictions after deployment. We evaluated surgeon-model disagreement across distinct clinical environments and whether surgeon responses added prognostic information. STUDY DESIGN: In a multi-institutional study, surgeons reviewed 30-day complication risks generated by locally deployed NSQIP-SRC, POTTER, or MySurgeryRisk models and assessed risk relative to model estimates. We evaluated disagreement, observed outcomes, and model discrimination overall, among discordant assessments, and across model-specific risk quartiles. Logistic regression models incorporating model predictions and surgeon responses underwent 5-fold cross-validation, with discrimination compared using DeLong's test. RESULTS: Thirty-one surgeons evaluated 1,702 predictions across 2 academic medical centers. Surgeons disagreed with 53% of predictions, assessing risk as higher in 37% and lower in 16%. Lower-than-model assessments were followed by no complication in 84% of cases, whereas higher-than-model assessments were followed by a complication in 23%. Model AUCs were 0.66, 0.82, and 0.83 for NSQIP-SRC, POTTER, and MySurgeryRisk, respectively. Among discordant assessments, AUCs were 0.61, 0.79, and 0.83, respectively. Incorporating surgeon responses increased AUCs to 0.68, 0.83, and 0.86, respectively; none of these differences were statistically significant. Effects varied across risk quartiles. CONCLUSIONS: Surgeons frequently disagreed with deployed surgical risk models, but disagreement was not uniformly beneficial or consistently associated with poorer model discrimination. Evaluating how clinicians interpret and modify model predictions may be important for understanding real-world performance beyond conventional model metrics.
No takes yet. Share an insight, caveat, or question.
Volpi et al. (2026) conducted an observational in Surgical patients undergoing 30-day complication risk prediction (n=1,702). Surgeon-model interaction (incorporating surgeon risk assessments into NSQIP-SRC, POTTER, or MySurgeryRisk predictions) vs. Risk prediction models alone (NSQIP-SRC, POTTER, or MySurgeryRisk) was evaluated on Surgeon-model disagreement and model discrimination (AUC) for 30-day surgical complications. Surgeons disagreed with 53% of model risk predictions, and incorporating surgeon responses nonsignificantly increased AUCs from 0.66, 0.82, and 0.83 to 0.68, 0.83, and 0.86.
Synapse has enriched 10 related papers. Consider them for comparative context: