Challenges of Modeling Outcomes for Surgical Infections: A Word of Caution

Background: We developed a novel analytic tool for colorectal deep organ/space surgical site infections (C-OSI) prediction utilizing both institutional and extra-institutional American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) data. Methods: Elective colorectal resections (2006–2014) were included. The primary end point was C-OSI rate. A Bayesian-Probit regression model with multiple imputation (BPMI) via Dirichlet process handled missing data. The baseline model for comparison was a multivariable logistic regression model (generalized linear model; GLM) with indicator parameters for missing data and stepwise variable selection. Out-of-sample performance was evaluated with receiver operating characteristic (ROC) analysis of 10-fold cross-validated samples. Results: Among 2,376 resections, C-OSI rate was 4.6% (n = 108). The BPMI model identified (n = 57; 56% sensitivity) of these patients, when set at a threshold leading to 80% specificity (approximately a 20% false alarm rate). The BPMI model produced an area under the curve (AUC) = 0.78 via 10-fold cross- validation demonstrating high predictive accuracy. In contrast, the traditional GLM approach produced an AUC = 0.71 and a corresponding sensitivity of 0.47 at 80% specificity, both of which were statstically significant differences. In addition, when the model was built utilizing extra-institutional data via inclusion of all (non-Mayo Clinic) patients in ACS-NSQIP, C-OSI prediction was less accurate with AUC = 0.74 and sensitivity of 0.47 (i.e., a 19% relative performance decrease) when applied to patients at our institution. Conclusions: Although the statistical methodology associated with the BPMI model provides advantages over conventional handling of missing data, the tool should be built with data specific to the individual institution to optimize performance.