Risk Prediction Accuracy Differs for Emergency Versus Elective Cases in the ACS-NSQIP

BACKGROUND: Accurate risk estimation is essential when benchmarking surgical outcomes for reimbursement and engaging in shared decision-making. The greater complexity of emergency surgery patients may bias outcome comparisons between elective and emergency cases. OBJECTIVE: To test whether an established risk modelling tool, the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) predicts mortality comparably for emergency and elective cases. METHODS: From the ACS-NSQIP 2011-2012 patient user files, we selected core emergency surgical cases also common to elective scenarios (gastrointestinal, vascular, and hepato-biliary-pancreatic). After matching strategy for Common Procedure Terminology (CPT) and year, we compared the accuracy of ACS-NSQIP predicted mortality probabilities using the observed-to-expected ratio (O:E), c-statistic, and Brier score. RESULTS: In all, 56,942 emergency and 136,311 elective patients were identified as having a common CPT and year. Using a 1:1 matched sample of 37,154 emergency and elective patients, the O:E ratios generated by ACS-NSQIP models differ significantly between the emergency O:E = 1.031; 95% confidence interval (CI) = 1.028-1.033 and elective populations (O:E = 0.79; 95% CI = 0.77-0.80, P < 0.0001) and the c-statistics differed significantly (emergency c-statistic = 0.927; 95% CI = 0.921-0.932 and elective c-statistic = 0.887; 95% CI = 0.861-0.912, P = 0.003). The Brier score, tested across a range of mortality rates, did not differ significantly for samples with mortality rates of 6.5% and 9% (eg, emergency Brier score = 0.058; 95% CI = 0.048-0.069 versus elective Brier score = 0.057; 95% CI = 0.044-0.07, P = 0.87, among 2217 patients with 6.5% mortality). When the mortality rate was low (1.7%), Brier scores differed significantly (emergency 0.034; 95% CI = 0.027-0.041 versus elective 0.016; 95% CI = 0.009-0.023, P value for difference 0.0005). CONCLUSION: ACS-NSQIP risk estimates used for benchmarking and shared decision-making appear to differ between emergency and elective populations.