How to Improve the Performance of Intraoperative Risk Models

BACKGROUND: Computerized reviews of patient data promise to improve patient care through early and accurate identification of at-risk and well patients. The significance of sampling strategy for patient vital signs data is not known. In the instance of the surgical Apgar score (SAS), we hypothesized that larger sampling intervals would improve the specificity and overall predictive ability of this tool. METHODS: We used electronic intraoperative data from general and vascular surgical patients in a single-institution registry of the American College of Surgeons National Surgical Quality Improvement Program. The SAS, consisting of lowest heart rate, lowest mean arterial blood pressure, and estimated blood loss between incision and skin closure, was calculated using 5 methods: instantaneously and using intervals of of 5 and 10 minutes with and without interval overlap. Major complications including death were assessed at 30 days postoperatively. RESULTS: Among 3000 patients, 272 (9.1%) experienced major complications or death. As the sampling interval increased from instantaneous (shortest) to 10 minutes without overlap (largest), the sensitivity, positive predictive value, and negative predictive value did not change significantly, but significant improvements were noted for specificity (79.5% to 82.9% across methods, P for trend <0.001) and accuracy (76.0% to 79.3% across methods, P for trend <0.01). In multivariate modeling, the predictive utility of the SAS as measured by the c-statistic nearly increased from Δc = +0.012 (P = 0.038) to Δc = +0.021 (P < 0.002) between the shortest and largest sampling intervals, respectively. Compared with a preoperative risk model, the net reclassification improvement and integrated discrimination improvement for the shortest versus largest sampling intervals of the SAS were net reclassification improvement 0.01 (P = 0.8) vs 0.06 (P = 0.02), and for integrated discrimination improvement, they were 0.008 (P < 0.01) vs 0.015 (P < 0.001). CONCLUSIONS: When vital signs data are recorded in compliance with American Society of Anesthesiologists' standards, the sampling strategy for vital signs significantly influences performance of the SAS. Computerized reviews of patient data are subject to the choice of sampling methods for vital signs and may have the potential to be optimized for safe, efficient patient care.