Making Sense of Surgical Risk When the Data Aren’t Perfect

Informed consent discussions, performance measurement, and studies of patient outcomes all require accurate information regarding the likelihood of a good or a bad outcome after surgery for an individual patient. Statistical risk models, typically based on multivariate analyses of retrospective data, aim to help clinicians quantify the probability of adverse outcomes for individual patients. Beginning with Goldman et al.’s 1977 landmark publication of a multivariate risk index for postoperative cardiac events,1 statistical risk models have been used for >3 decades in various surgical contexts prospectively to predict the likelihood of adverse outcomes for patients considering surgical interventions.2 Predictions derived from such models can then be used to help decide whether to have surgery (e.g., medical versus surgical management of hip fracture), what kind of surgery to have (e.g., conventional versus transcatheter aortic valve replacement), or where to have surgery (e.g., outpatient surgicenter versus hospital). In predicting risk before surgery, one size does not necessarily fit all; different risk factors might be more or less important for different groups of patients. Perhaps for this reason, different approaches to predicting risk have emerged that are specific to certain surgical specialties and specific procedures. For example, the Society of Thoracic Surgeons (STS) risk score is used to predict operative mortality for patients undergoing cardiothoracic surgery.3 In addition to being a helpful research tool for comparing patient populations, the STS score has emerged as a decision point in determining the management approach for symptomatic aortic stenosis. For instance, in the Placement of AoRTic TraNscathetER Valve (PARTNER) trial, an STS score of ≥10% was used to define a high-risk surgical group.4 Only those patients at or above this cutoff were considered for transcatheter aortic valve replacement. Other surgical specialties using procedure-specific risk models include orthopedics5 and transplant surgery.6 In this issue of Anesthesia & Analgesia, Reponen et al.7 review evidence on preoperative risk stratification for patients undergoing cranial neurosurgery. The authors compared commonly used risk scores by conducting a qualitative systematic review. From a starting sample of >2,000 papers, Reponen et al.7 analyzed 25 published reports describing the outcomes after craniotomy as a function of 5 different scales: American Society of Anesthesiologists physical status classification (ASA; 10 studies), the Karnofsky Performance Score (KPS; 16 studies), the Charlson Comorbidity Score (CCS; 3 studies), the modified Rankin Scale (mRS; 2 studies), and the Sex, KPS, ASA classification, meningioma Location, and peritumoral Edema score (SKALE; 2 studies). The authors found evidence of some association among 3 of these scores and postoperative outcomes: ASA physical status was associated with mortality and with infectious complications including surgical site infection and postoperative meningitis. The KPS was associated with mortality, functional impairment, and systemic complications, including pneumonia, heart failure, and deep venous thrombosis. The CCS was associated with mortality in patients who had elective treatment of intracranial aneurysms and in elderly patients with intracranial tumors.7 As Reponen et al.7 note, important differences in study design, patient populations, and outcome definitions across studies limit the conclusions that can be drawn from this review. The 25 studies they reviewed included retrospective and prospective designs, sample sizes ranging from 30 to almost 5000 patients, and varying definitions of operative and nonoperative complications. Indeed, the authors note that they were unable to perform a quantitative meta-analysis (as opposed to a narrative review) because of marked differences in the methods and outcomes used across studies. The work of Reponen et al.7 offers a useful starting place for further research and discussion regarding prediction of outcomes after surgery. Importantly, it highlights the importance of critically examining the individual variables contained within any risk model; ideally, a plausible physiologic mechanism should link the individual elements of a risk score to the outcome of interest. In terms of postoperative mortality, for example, increasing age and comorbidity burden might predict increased mortality. In other cases, the relationship may be less apparent. For example, the data suggest that patients with a higher ASA physical status classification may be more likely to experience postoperative infections. While this association may be supported by observations in the data, further research is likely to be required to tease out the specific mechanisms that link poor physical status to infectious outcomes and to develop effective interventions to decrease risk among future patients. The review by Reponen et al.7 also highlights challenges in incorporating variables into risk scores that rely on individual clinician judgment. Indeed, the Goldman Index, and statistical risk modeling as a whole, grew out of a concern that existing tools for patient assessment, such as the ASA physical status classification, were prone to subjectivity due to their reliance on individual clinician’s judgments. Yet, when it has been included in statistical risk models, the ASA physical status classification turns out to be a tremendously useful predictor of postoperative outcomes specifically because it takes advantage of physicians’ abilities to judge how sick an individual patient is, above and beyond the comorbidities that appear in a patient’s chart. For example, a recent review of heterogeneous patient populations found that, despite poor interrater reliability, ASA physical status was an accurate predictor of postsurgical mortality.8 Nonetheless, reliance on human judgment in scoring patient risk raises important concerns regarding the potential for risk assessments, such as the ASA physical status classification, to vary across raters and to be overly influenced by individual biases. Of the 5 scores that Reponen et al.7 reviewed, 4 had components that require some degree of judgment on the part of the assessing clinician. As such, the paper highlights opportunities for further research on opportunities to increase the reliability of such assessments across clinicians and settings and how such clinical assessments can potentially be incorporated into large-scale prospective data collection efforts. Such efforts are ongoing outside of anesthesia through the Veterans Affairs and American College of Surgeons’ National Surgical Quality Improvement Programs (NSQIP9 and ACS-NSQIP10) and within anesthesiology through the Anesthesia Quality Institute’s National Anesthesia Clinical Outcomes Registry11 and the University of Michigan’s Multicenter Perioperative Outcomes Group.12 Ultimately, data from these and other registries may carry potential not only for the development of additional risk prediction models for neurosurgery and other specialties but also to shed light on how the collection and data on specific risk predictors can be more effectively standardized going forward. The work of Reponen et al.7 makes it clear that further work is needed to develop reliable and robust approaches to predicting the risk for neurosurgical patients. In the meantime, however, there are patients and clinicians who need guidance as they consider therapeutic options. For patients and clinicians, the work of Reponen et al.7 confirms that worse functional status and greater degrees of comorbidity are likely to be associated with perioperative morbidity and mortality in craniotomy patients7 and that further work is still needed to identify the specific combinations of factors that most accurately predict poor outcomes. In the meantime, then, perhaps we should consider the risk scores reviewed by Reponen et al.7 to be places to start in our conversations about operative risk among neurosurgical patients, but not the ultimate arbiters of therapeutic choices. DISCLOSURES Name: Meghan B. Lane-Fall, MD, MSc. Contribution: This author helped write the manuscript. Attestation: Meghan B. Lane-Fall approved the final manuscript. Name: Mark D. Neuman, MD, MSc. Contribution: This author helped write the manuscript. Attestation: Mark D. Neuman approved the final manuscript. This manuscript was handled by: Gregory J. Crosby, MD.